Add broker throw IllegalStateException

[qq359130530]

I deployed a Kafka cluster, then deployed CC, managed the Kafka cluster, then incrementally deployed a Kafka node, then added the new Broker to the cluster using the CC add_broker interface, which throws an IllegalStateException. The stack information is as follows:

[2024-11-06 15:01:45,959] INFO UserTask 66dd8585-4ffd-4395-8c1b-482f411afc52 is completed and removed from active tasks list (com.linkedin.kafka.cruisecontrol.servlet.UserTaskManager)
[2024-11-06 15:01:45,961] INFO Task [66dd8585-4ffd-4395-8c1b-482f411afc52] calculation finishes, result:
{"KafkaPartitionState":{"offline":[],"urp":[],"with-offline-replicas":[],"under-min-isr":[]},"KafkaBrokerState":{"OfflineReplicaCountByBrokerId":{"2":0},"IsController":{"1":false,"2":true},"Summary":{"StdLeadersPerBroker":91.0,"Leaders":182,"MaxLeadersPerBroker":182,"Topics":18,"MaxReplicasPerBroker":182,"StdReplicasPerBroker":91.0,"Brokers":2,"AvgReplicationFactor":1.0,"AvgLeadersPerBroker":91.0,"Replicas":182,"AvgReplicasPerBroker":91.0},"OutOfSyncCountByBrokerId":{"2":0},"LeaderCountByBrokerId":{"1":182,"2":0},"OnlineLogDirsByBrokerId":{"1":["/home/sdb/kafka/kafka_data_dir","/home/sdc/kafka/kafka_data_dir","/home/sdd/kafka/kafka_data_dir"],"2":["/home/sdb/kafka/kafka_data_dir","/home/sdc/kafka/kafka_data_dir","/home/sdd/kafka/kafka_data_dir"]},"BrokerSetByBrokerId":{},"OfflineLogDirsByBrokerId":{"1":[],"2":[]},"ReplicaCountByBrokerId":{"1":182,"2":0}},"version":1} (operationLogger)
[2024-11-06 15:01:50,842] INFO Processing async request AddBrokerRequest. (com.linkedin.kafka.cruisecontrol.servlet.handler.async.AbstractAsyncRequest)
[2024-11-06 15:01:50,845] INFO Create a new UserTask 50387b5f-d82a-41a5-9bd0-f659bdaaa3b6 with SessionKey SessionKey{_session=com.linkedin.kafka.cruisecontrol.servlet.ServletSession@532bb10c,_requestUrl=POST /kafkacruisecontrol/add_broker,_queryParams={brokerid=[2], dryrun=[false], json=[true]}} (com.linkedin.kafka.cruisecontrol.servlet.UserTaskManager)
[2024-11-06 15:01:50,855] WARN Received exception when trying to execute runnable for "Add brokers" (com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable)
java.lang.IllegalStateException: Cannot execute new proposals due to failure to retrieve whether the Kafka cluster has an already ongoing partition reassignment.
at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.sanityCheckDryRun(KafkaCruiseControl.java:288) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.init(GoalBasedOperationRunnable.java:132) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:157) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.AddBrokersRunnable.getResult(AddBrokersRunnable.java:91) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.AddBrokersRunnable.getResult(AddBrokersRunnable.java:35) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable.run(OperationRunnable.java:45) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.run(GoalBasedOperationRunnable.java:38) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support LIST_PARTITION_REASSIGNMENTS
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180) ~[kafka-clients-3.5.1.jar:?]
at com.linkedin.kafka.cruisecontrol.executor.ExecutionUtils.ongoingPartitionReassignments(ExecutionUtils.java:372) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.executor.ExecutionUtils.partitionsBeingReassigned(ExecutionUtils.java:350) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.executor.Executor.listPartitionsBeingReassigned(Executor.java:1245) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.sanityCheckDryRun(KafkaCruiseControl.java:285) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
... 9 more
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support LIST_PARTITION_REASSIGNMENTS
[2024-11-06 15:01:50,861] ERROR Error processing POST request '/add_broker' due to: 'java.lang.IllegalStateException: Cannot execute new proposals due to failure to retrieve whether the Kafka cluster has an already ongoing partition reassignment.'. (com.linkedin.kafka.cruisecontrol.KafkaCruiseControlRequestHandler)
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot execute new proposals due to failure to retrieve whether the Kafka cluster has an already ongoing partition reassignment.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.AbstractAsyncRequest.getResponse(AbstractAsyncRequest.java:56) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.AbstractRequest.handle(AbstractRequest.java:37) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlRequestHandler.handlePost(KafkaCruiseControlRequestHandler.java:156) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlRequestHandler.doGetOrPost(KafkaCruiseControlRequestHandler.java:74) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.ServletRequestHandler.doPost(ServletRequestHandler.java:46) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.lang.IllegalStateException: Cannot execute new proposals due to failure to retrieve whether the Kafka cluster has an already ongoing partition reassignment.
at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.sanityCheckDryRun(KafkaCruiseControl.java:288) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.init(GoalBasedOperationRunnable.java:132) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:157) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.AddBrokersRunnable.getResult(AddBrokersRunnable.java:91) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.AddBrokersRunnable.getResult(AddBrokersRunnable.java:35) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable.run(OperationRunnable.java:45) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.run(GoalBasedOperationRunnable.java:38) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
... 1 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support LIST_PARTITION_REASSIGNMENTS
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180) ~[kafka-clients-3.5.1.jar:?]
at com.linkedin.kafka.cruisecontrol.executor.ExecutionUtils.ongoingPartitionReassignments(ExecutionUtils.java:372) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.executor.ExecutionUtils.partitionsBeingReassigned(ExecutionUtils.java:350) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.executor.Executor.listPartitionsBeingReassigned(Executor.java:1245) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.sanityCheckDryRun(KafkaCruiseControl.java:285) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.init(GoalBasedOperationRunnable.java:132) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:157) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.AddBrokersRunnable.getResult(AddBrokersRunnable.java:91) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.AddBrokersRunnable.getResult(AddBrokersRunnable.java:35) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable.run(OperationRunnable.java:45) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.run(GoalBasedOperationRunnable.java:38) ~[cruise-control-2.5.142-SNAPSHOT.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
... 1 more
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support LIST_PARTITION_REASSIGNMENTS
[2024-11-06 15:01:50,863] INFO 10.67.0.97 - - [06/Nov/2024:07:01:50 +0000] "POST /kafkacruisecontrol/add_broker?dryrun=false&brokerid=2&json=true HTTP/1.1" 500 6139 (CruiseControlPublicAccessLogger)
[2024-11-06 15:01:50,959] WARN UserTask 50387b5f-d82a-41a5-9bd0-f659bdaaa3b6 is completed with Exception and removed from active tasks list (com.linkedin.kafka.cruisecontrol.servlet.UserTaskManager)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationFuture (file:/home/master/cruise-control/cruise-control/build/libs/cruise-control-2.5.142-SNAPSHOT.jar) to field java.lang.Throwable.detailMessage
WARNING: Please consider reporting this to the maintainers of com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationFuture
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2024-11-06 15:01:50,963] INFO Task [50387b5f-d82a-41a5-9bd0-f659bdaaa3b6] calculation fails, exception:
java.util.concurrent.ExecutionException: Operation 'Add brokers' received exception. java.lang.IllegalStateException: Cannot execute new proposals due to failure to retrieve whether the Kafka cluster has an already ongoing partition reassignment. (operationLogger)
[2024-11-06 15:02:45,959] INFO Expiring the session associated with SessionKey{_session=com.linkedin.kafka.cruisecontrol.servlet.ServletSession@532bb10c,_requestUrl=POST /kafkacruisecontrol/add_broker,_queryParams={brokerid=[2], dryrun=[false], json=[true]}}. (com.linkedin.kafka.cruisecontrol.servlet.UserTaskManager)
[2024-11-06 15:03:05,976] INFO Kicking off metric sampling for time range [1730876345976, 1730876585976], duration 240000 ms with timeout 120000 ms. (com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcherManager)
[2024-11-06 15:03:06,001] INFO [Consumer clientId=CruiseControlMetricsReporterSampler-consumer--914373556761640071, groupId=null] Seeking to offset 0 for partition __CruiseControlMetrics-11 (org.apache.kafka.clients.consumer.KafkaConsumer)
[2024-11-06 15:03:06,002] INFO [Consumer clientId=CruiseControlMetricsReporterSampler-consumer--914373556761640071, groupId=null] Seeking to offset 0 for partition __CruiseControlMetrics-9 (org.apache.kafka.clients.consumer.KafkaConsumer)
[2024-11-06 15:03:06,002] INFO [Consumer clientId=CruiseControlMetricsReporterSampler-consumer--914373556761640071, groupId=null] Seeking to offset 0 for partition __CruiseControlMetrics-7 (org.apache.kafka.clients.consumer.KafkaConsumer)
[2024-11-06 15:03:06,002] INFO [Consumer clientId=CruiseControlMetricsReporterSampler-consumer--914373556761640071, groupId=null] Seeking to offset 0 for partition __CruiseControlMetrics-5 (org.apache.kafka.clients.consumer.KafkaConsumer)
[2024-11-06 15:03:06,002] INFO [Consumer clientId=CruiseControlMetricsReporterSampler-consumer--914373556761640071, groupId=null] Seeking to offset 0 for partition __CruiseControlMetrics-3 (org.apache.kafka.clients.consumer.KafkaConsumer)
[2024-11-06 15:03:06,002] INFO [Consumer clientId=CruiseControlMetricsReporterSampler-consumer--914373556761640071, groupId=null] Seeking to offset 0 for partition __CruiseControlMetrics-1 (org.apache.kafka.clients.consumer.KafkaConsumer)
[2024-11-06 15:03:06,003] INFO [Consumer clientId=CruiseControlMetricsReporterSampler-consumer--914373556761640071, groupId=null] Seeking to offset 0 for partition __CruiseControlMetrics-30 (org.apache.kafka.clients.consumer.KafkaConsumer)