Benchmark client failed to reconnect to segmentstore after restart segmentstore

[Ranjan-Padhi]

While running IO through benchmark client, If delete one or all segmentstore pods. Pods are coming up in < 30 secs but benchmark IO client failed to reconnect to segmentstore and due to this IO unable to recover

2020-08-04 14:44:12:550 +0000 [clientInternal-2-1] WARN io.pravega.client.segment.impl.SegmentOutputStreamImpl - 2259b1fb-135e-4223-9c64-aa48e75ecd83 Failed to connect:
java.util.concurrent.CompletionException: java.lang.IllegalStateException
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
        at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:975)
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at io.pravega.common.concurrent.Futures.lambda$delayedFuture$23(Futures.java:588)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
        at io.pravega.client.segment.impl.SegmentOutputStreamImpl.lambda$null$4(SegmentOutputStreamImpl.java:584)
        at io.pravega.common.concurrent.Futures.lambda$delayedFuture$25(Futures.java:608)
        at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)

in segmentstore there is no error or exception except below warning message

2020-08-04 14:41:01,928 305448 [core-2] INFO  i.p.s.server.CacheManager - CacheManager Unregistered ReadIndex[13-125] (Testscope/TestStream1/342.#epoch.0).
2020-08-04 14:41:01,928 305448 [core-2] INFO  i.p.s.s.h.h.PravegaRequestProcessor - [requestId=773094113280] Sending empty response on connection [id: 0xf4c0d744, L:/172.28.58.23:12345 - R:/172.28.58.10:48266] while reading segment Testscope/TestStream1/342.#epoch.0 due to CancellationException.

Note :- If terminate the benchmark and rerun the benchmark again then it is running successfully

Pravega Release details :
pravega:0.8.0-2598.6acd31734
pravega/bookkeeper:0.7.1
pravega/bookkeeper-operator:0.1.2
pravega/pravega-operator:0.5.1
pravega/zookeeper:0.2.8
pravega/zookeeper-operator:0.2.8