RabbitMQ Receiver fails with exception

[HardikkumarSoni]

We are running spark application, which receives data form rabbitmq as stream. We are observing below issue after 8 hours (duration is not constant, some time after 24 hours also).

Below exception causes application to be restarted.

ERROR 2017-05-16 11:24:54,524 RabbitMQInputDStream.scala:99 - org.apache.spark.streaming.rabbitmq.receiver.RabbitMQReceiver: Got this unknown exception: com.rabbitmq.client.ShutdownSignalException: clean connection shutdown; protocol method: #method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
com.rabbitmq.client.ShutdownSignalException: clean connection shutdown; protocol method: #method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
at com.rabbitmq.client.QueueingConsumer.handle(QueueingConsumer.java:203) ~[datapond-spark-summary-1.0.4-SNAPSHOT.jar:na]
at com.rabbitmq.client.QueueingConsumer.nextDelivery(QueueingConsumer.java:220) ~[datapond-spark-summary-1.0.4-SNAPSHOT.jar:na]
at org.apache.spark.streaming.rabbitmq.receiver.RabbitMQReceiver.org$apache$spark$streaming$rabbitmq$receiver$RabbitMQReceiver$$receive(RabbitMQInputDStream.scala:90) ~[datapond-spark-summary-1.0.4-SNAPSHOT.jar:na]
at org.apache.spark.streaming.rabbitmq.receiver.RabbitMQReceiver$$anon$1.run(RabbitMQInputDStream.scala:72) [datapond-spark-summary-1.0.4-SNAPSHOT.jar:na]
WARN 2017-05-16 11:24:54,526 Logging.scala:91 - org.apache.spark.streaming.receiver.ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Trying to connect again
WARN 2017-05-16 11:24:54,530 Logging.scala:70 - org.apache.spark.streaming.receiver.ReceiverSupervisorImpl: Receiver has been stopped
ERROR 2017-05-16 11:24:54,806 Logging.scala:95 - org.apache.spark.util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[receiver-supervisor-future-0,5,main]
java.lang.Error: java.lang.InterruptedException: sleep interrupted
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148) ~[na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method) ~[na:1.8.0_121]
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply$mcV$sp(ReceiverSupervisor.scala:195) ~[spark-streaming_2.10-1.6.3.1.jar:1.6.3.1]
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply(ReceiverSupervisor.scala:188) ~[spark-streaming_2.10-1.6.3.1.jar:1.6.3.1]
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply(ReceiverSupervisor.scala:188) ~[spark-streaming_2.10-1.6.3.1.jar:1.6.3.1]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[scala-library-2.10.6.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[scala-library-2.10.6.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
... 2 common frames omitted
ERROR 2017-05-16 11:24:54,925 SignalLogger.scala:57 - org.apache.spark.executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM
receiver-supervisor-future-0,5,main]
java.lang.Error: java.lang.InterruptedException: sleep interrupted
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148) ~[na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method) ~[na:1.8.0_121]
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply$mcV$sp(ReceiverSupervisor.scala:195) ~[spark-streaming_2.10-1.6.3.1.jar:1.6.3.1]
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply(ReceiverSupervisor.scala:188) ~[spark-streaming_2.10-1.6.3.1.jar:1.6.3.1]
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply(ReceiverSupervisor.scala:188) ~[spark-streaming_2.10-1.6.3.1.jar:1.6.3.1]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[scala-library-2.10.6.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[scala-library-2.10.6.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
... 2 common frames omitted

Any help would be appreciated.

[compae]

Sorry for the delay.

This error was corrected?

It's possible that other application or human send one kill to the Spark job?

Regards

[HardikkumarSoni]

It was an error coming in spark driver ..Spark nodes was not having NTP time sync .. Issue is resolved ...

Thanks for your reply ...

Regards
Hardik