v3.2.0

PIPs

• PIP-323: Complete Backlog Quota Telemetry #21709
• PIP-322: Pulsar Rate Limiting Refactoring #21680
• PIP-320: OpenTelemetry Scaffolding #21635
• PIP-318: Don't retain null-key messages during topic compaction #21541
• PIP-315: Configurable max delay limit for delayed delivery #21490
• PIP-313: Support force unsubscribe using consumer api #21452
• PIP-312:
  • Use StateStoreProvider to manage state in Pulsar Functions endpoints #21597
  • Use StateStoreProvider to manage state in Pulsar Functions endpoints #21438
• PIP-307:
  • Support subscribing multi topics for WebSocket #21390
  • Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager #20748
• PIP-305: Customize DNS servers to use for Pulsar Client #21352
• PIP-303: Add optional parameters for getPartitionedStats #21228
• PIP-302: Introduce refreshAsync API for TableView #21271
• PIP-301: Introduce LoadBalanceResources to unify the load-date CRUD #21129
• PIP-300: Add custom dynamic configuration for plugins #21127
• PIP-299: Stop dispatch messages if the individual acks will be lost in the persistent storage #21118
• PIP-297: Support terminating Function & Connector with the fatal exception #21079
• PIP-293: Delete config disableBrokerInterceptors #20993
• PIP-290: Provide a way to implement WSS E2E encryption and not need to expose the private key to the WebSocket Proxy #20923
• PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index #20867
• PIP-277: Add current option in the Clusters list cmd #20614
• PIP-264: Enhanced OTel-based metric system #21080

Broker

• Replication stopped due to unload topic failed #21947
• Fix getMessageById throws 500 #21919
• Fix write all compacted out entry into compacted topic #21917
• Fix deadlock while skip non-recoverable ledgers #21915
• Fix leader broker cannot be determined when the advertised address and advertised listeners are configured #21894
• Fix DLQ producer name conflicts when multiples consumers send messages to DLQ #21890
• Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 #21885
• Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException #21856
• Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21842
• Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher #21765
• Fix the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764
• Fix closing of AbstractDispatcherSingleActiveConsumer and reduce flakiness of test #21736
• Skip topic auto-creation for ExtensibleLoadManager internal topics #21729
• Fix ServiceUnitStateChannel monitor to tombstone only inactive bundle states #21721
• Avoid compaction task stuck when the last message to compact is a marker #21718
• Fix the issue of topics possibly being deleted #21704
• CheckTopicExists supports checking partitioned topic without index #21701
• Remove internalGetListAsync and validatePartitionTopicUpdateAsync #21699
• Cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission #21696
• Fix typo in the config key #21690
• network package lost if enable haProxyProtocolEnabled #21684
• Pulsar Rate Limiting Refactoring changes (PIP-322) #21681
• Record GeoPersistentReplicator.msgOut before producersendAsync #21673
• Print recoverBucketSnapshot log if cursorProperties are empty #21651
• Fix memory leak during topic compaction #21647
• Fix lookupRequestSemaphore leak when topic not found #21646
• Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21633
• Fix getting incorrect KeyValue schema version #21632
• Add optional parameters for getPartitionedStats #21611
• PIP-301 Part-3: Add QuotaResources #21596
• Delete topic timeout due to NPE #21595
• Fix incorrect unack count when using shared subscription on non-persistent topic #21592
• Duplicate LedgerOffloader creation when namespace/topic… #21591
• Support not retaining null-key message during topic compaction #21578
• Correct schema deletion for partitioned topic #21574
• Fix create topic with different auto creation strategies causes race condition #21545
• Fix thousands orphan PersistentTopic caused OOM #21540
• Fix setReplicatedSubscriptionStatus incorrect behavior #21510
• Fix failure while creating non-durable cursor with inactive managed-ledger #21508
• Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 #21496
• Do not write replicated snapshot marker when the topic which is not enable replication #21495
• Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
• Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
• Fix issue with consumer read uncommitted messages from compacted topic #21465
• Delete methods marked Lombok @Getter #21462
• Fix resource_quota_zpath #21461
• Fix namespace bundle stuck in unloading status #21445
• Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered #21434
• Fix MultiRoles token provider NPE when using anonymous clients #21429
• Allow broker deployment in heterogeneous hw config cluster without restricting nic speed detection #21409
• Fix avoid creating new topic after migration is started #21368
• Fix race condition of replication cluster connection during migration topic #21364
• Optimize and clean up aggregation of topic stats #21361
• Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
• PIP-188 Fix cluster migration state store into local metadatastore #21359
• PIP-188 Add support to auto create topic resources into green cluster before migration [part-3] #21354
• Fix heartbeat namespace create transaction internal topic #21348
• Fix unload operation stuck when use ExtensibleLoadManager #21332
• Fix typos in Subscription interface #21321
• Use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error #21282
• Cleanup deprecated method of metadata store etcd #21253
• Miss headersAndPayload and messageIdData in MessagePublishContext #21245
• Improve error messages when updating partition count #21243
• Fixed produce and consume when anonymousUserRole enabled #21237
• Fix inconsistent topic policy #21231
• Fixed reset for AggregatedNamespaceStats #21225
• fix flaky test PatternTopicsConsumerImplTest #21222
• Let the producer request success at the first time if the previous one is inactive #21220
• Fix lookup heartbeat and sla namespace bundle when using extensible load manager #21213
• Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211
• Make the TopicCompactionService to support find entry based on publishTime or index #21208
• Replicator leak when removeReplicator in NonPersistentTopic #21205
• Print warn log if ssl handshake error & print ledger id when switch ledger #21201
• Avoid record inactiveproducers when deduplication is disable. #21193
• Make the new exclusive consumer instead the inactive one faster #21183
• improve read entry error log for troubleshooting #21169
• Disable balancing based on DirectMemory. #21168
• Make specified producer could override the previous one #21155
• Improve logs for troubleshooting #21141
• Fix web tls url null cause NPE #21137
• Fix unack count when mixing non batch index and batch index acks #21126
• Fix unsubscribe non-durable subscription error #21099
• Fix isolated group not work problem. #21096
• Fix deleting topic not delete the related topic policy and schema. #21093
• revert remove duplicate topics name when deleteNamespace #21087
• Cleanup correctly heartbeat bundle ownership when handling broker deletion event #21083
• Fix write duplicate entries into the compacted ledger after RawReader reconnects #21081
• Remove bundle-data in local metadata store. #21078
• Fix typos: lister -> listener #21068
• Make sure all inflight writes have finished before completion of compaction #21067
• Make read compacted entries support maxReadSizeBytes limitation #21065
• Improve cache handling for partitioned topic metadata when doing lookup #21063
• Use MessageDigest.isEqual when comparing digests #21061
• Removing webURL check null #21043
• Fix potential case cause retention policy not working on topic level #21041
• Fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21025
• Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled #21015
• PIP-293: Delete config disableBrokerInterceptors #20995
• Fix incorrect unack msk count when dup ack a message #20990
• Fix can't stop phase-two of compaction even though messageId read reaches lastReadId #20988
• Fix get outdated compactedTopicContext after compactionHorizon has been updated #20984
• Fix compaction subscription delete by inactive subscription check. #20983
• Fix message loss during topic compaction #20980
• Fix incorrect number of read compacted entries #20978
• Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there #20951
• Fix http error message of Bookie rack placement config API when bookie-rack is not configured. #20949
• Fix chunked messages will be filtered by duplicating #20948
• Rackaware policy is ineffective when delete zk rack info after bkclient initialize #20944
• Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin #20939
• Fix consumers are not redirected to migrated cluster #20928
• PIP-300: Add custom dynamic configuration for plugins #20884
• Delete unused param #20866
• Improve performance of StatsOutputStream by using ArrayDeque instead of Stack #20808
• Fix missing generate some metrics in BrokerOperabilityMetrics #20720
• Fix ModularLoadManagerImpl always delete active bundle-data. sec ver. #20620
• Consider iowait as idle. #19110
• Fix bug caused by optimistic locking #18390

Pulsar IO and Pulsar Functions

• [IO] Make connectors load sensitive fields from secrets #21675
• [IO] PIP-297: Support terminating Function & Connector with the fatal exception #21143
• [IO] Fix --retain[-key]-ordering not working error for sink #21060
• [IO] Improve kinesis connector config. #21004
• [IO] Update test certs for Elasticsearch #21001
• [IO] Support configuration secret interpolation #20901
• [FN] Add missing version field back to querystate API #21966
• [FN] Throw 404 RestException when state key not found #21921
• [FN] Fix Deadlock in Functions Worker LeaderService #21711
• [FN] Fix functions_log4j2.xml delete strategy config #21215
• [FN] Fix ProducerConfig cannot update error #21037
• [FN] Fix the --batch-builder not working error for functions #21023
• [FN] Make python install dependencies from requirements.txt #20174g function removal to avoid label errors #19584

CLI

• [Admin] Set ns level backlog quota does not take effect if retention exists #20690
• [Admin] Add clusters check when set replication clusters #21650
• [Admin] Fix KeyValue schema compatibility check caused OOM #21645
• [Admin] Do not print full stacktrace when get partitioned metadata not found #20979
• [Admin] Fix get topic stats fail if a subscription catch up concurrently #20971
• [CLI] Fix the variable producerName of BatchMsgContainer is null #20819
• [CLI] Introduce Regex and File Input Parameters for Enhanced Topic Deletion Command #21664
• [CLI] Add get-cluster-migration cmd #21473
• [CLI] Add current option in the Clusters list cmd #21139
• [CLI] Add some checks for topic-level setOffloadPolicies #20943
• [CLI] Create new pulsar-cli-utils module #20782
• [CLI] Refactor pulsar-client-tools module #20764
• [CLI] Add command line option for configuring the memory limit #20663

Others

• [TXN] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly #21505
• [TXN] Fix getting last message ID when there are ongoing transactions #21466
• [TXN] Fix the consumer stuck due to deduplicated messages in pending ack state #21177
• [ML] Fix orphan scheduled task for ledger create timeout check #21542
• [ML] Fix unfinished callback when deleting managed ledger #21530
• [ML] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads #21333
• [ML] Fix thread safe issue with RangeCache.put and RangeCache.clear #21302
• [ML] Fix wrong msg backlog of non-durable cursor after trim ledgers #21250
• [ML] Reader can set read-pos to a deleted ledger #21248
• [ML] Persist mark deleted ops to ZK if create cursor ledger was failed #20935
• [ML] Fix discontinuous ledger deletion #20898
• [META] Improve fault tolerance of blocking calls by supporting timeout #21028
• [META] Fix deadlock in AutoRecovery. #21010
• [WS] Support subscribing multi/pattern topic for Websocket #21379
• [WS] PIP-290 Make WSS support E2E encryption #20958
• [MISC] Add Pulsar BOM (Bill of Materials) #21871
• [MISC] Pin Netty version in pulsar-io/alluxio #21728
• [MISC] Disable rebase and merge to prevent unsquashed PRs #21517
• [Build] Increase apt's initial timeout from 250ms to 2000ms and increase retries #21727
• [Build] Configure https timeout for apt, copy config to java-test-image #21724
• [Build] Add a default username in the image #21695
• [Build] Fix Stage Docker images fail on M1 Mac #21659
• [Build] Change UBUNTU_MIRROR default value #21520
• [Build] Unified the version of the library org.checkerframework #21503
• [Build] Fix apt download issue in building the docker image #21489
• [Build] Rename schema_example.conf to schema_example.json #21447
• [Build] Use git-archive to generate source release #20981

Library updates

• Upgrade spotbugs maven plugin version for Java 21 compatibility #21389
• Upgrade bookkeeper to 4.16.3 #21146
• Upgrade Netty to 4.1.104 and io_uring to 0.0.24 #21723
• Upgrade Jacoco version to 0.8.11 to support Java 21 #21388
• Bump broker okio version to 3.4.0 #21064
• Bump GRPC version to 1.55.3 to fix CVE #21057
• Upgrade alluxio version to 2.9.3 to fix CVE-2023-38889 #21715
• Bumped version to 3.2.0-SNAPSHOT #21147
• Upgrade Lombok to 1.18.30 to support compiling with JDK21 #21278
• Upgrade Guava to 32.1.2-jre #21090
• Upgrade Apache ZooKeeper to 3.9.1 #20933

A special thanks to the following contributors (alphabetic Github IDs) who contributed to Pulsar 3.2.0:
lhotari, jiangpengcheng, merlimat, poorbarcode, nodece, thetumbled, hrzzzz, BewareMyPower, RobertIndie, onobc, Technoboy-, wenbingshen, aloyszhang, heesung-sn, massakam, liangyepianzhou, coderzc, asafm, crossoverJie, TakaHiR07, zymap, yaalsn, rdhabalia, dragosvictor, pandalee99, AnonHxy, hangc0276, pengxiangrui127, mattisonchao, Denovo1998, 1Jack2, shibd, horizonzy, liudezhi2098, KevinLiLu, hanmz, erobot, Demogorgon314, tisonkun, diegosalvi, liangyuanpeng, kecona, alpreu, tuteng, frankjkelly, lifepuzzlefun, Shawyeok, Apurva007, Crispy-fried-chicken, michaeljmarshall, chenhongSZ, eolivelli, vineeth1995, codelipenghui, zhanghaou, JooHyukKim, klevy-toast.