v2.8.0
PIPs
- [PIP 45] Pluggable metadata interface #9148 #9221 #9240 #9273 #9274 #9338 #9346 #9351 #9412 #9485 #9586 #9967 #9973 #10391 #10457 #10532 #10545 #10647
- [PIP 50] Package management service #8637 #8680 #8744 #8764 #8816 #8817 #8858 #8861 #8893 #8907
- [PIP 68] Exclusive producer #8685 #8992 #9554 #9600
- [PIP 70] Lightweight broker entry metadata #8618 #9067 #9088 #9091
- [PIP 71] Pulsar SQL migrate SchemaHandle to Presto decoder #8422
- [PIP 74] Client memory limits #8965 #9676
- [PIP 75] Perform serialization/deserialization with LightProto #9046
- [PIP 76] Streaming offloader #9096
- [PIP 82] Tenant and namespace level rate limiting #10008 #10201 #10204 #10218
- [PIP 83] Message consumption with pooled buffer #10184
- [PIP 85] Support get reader schema for a message in Java Client #10476
Transactions
- Support pending ack state persistence #8881
- Stable position and low watermark for the transaction buffer #9195
- Transaction timeout support #9229
- Transaction coordinator retry to complete the transaction #9236
- Fix race condition when appending transaction log #9238
- Transaction buffer snapshot #9490
- Add metrics for transaction coordinator #9706
- Clean the useless transaction individual acknowledgements based on low watermark #9722
- Fix memory leak when deleting transaction marker #9751
- Check the transaction state at the client side #9776
- Clean aborted transactions for the transaction buffer #9974
- Fix transaction coordinator retry to end transaction #10131
- Fix NPE when opening a new transaction #10139
- Fix transaction log failed to recover #10146
- Fix transaction coordinator recover timeout #10162
- Handling committing and aborting state when recovering transaction coordinator #10179
- Fix NPE in case of enableTransactionCoordinator=false #10182
- Fix transaction buffer client timeout #10206
- Fix recover max local id issue for the transaction coordinator #10224
- Support using transactions on standalone #10238
- Fix transaction buffer lookup issue #10257
- Fix transaction timeout issue at the client side #10323
- Fix transaction client reconnect issue after transaction coordinator unloaded #10327
- Fix transaction timeout not canceled after expired #10366
- Transaction log low watermark optimization #10422
- Fix the issue of transaction buffer client channel is inactive #10407
- Add Admin API for getting transaction coordinator stats #10639
- Add Admin API for getting transaction in transaction buffer stats #10642
- Add Admin API for getting transaction in pending ack stats #10648
- Add Admin API for getting transaction buffer stats and pending ack stats #10650
- Add Admin API for getting transaction coordinator internal stats #10653
- Setup transaction metadata with metadata store #10677
- Fix issue with acknowledge messages from multiple subscriptions of a topic #10689
- Admin API for getting transaction metadata #10690
- Admin API for getting slow transactions #10701
- Fix transaction log handle managed ledger WriteFail state #10711
- Admin API for getting pending ack internal stats #10725
- Fix transaction ack delete marker position when don't have transaction ack #10741
- Fix transaction pending ack generate managedLedgerStats fail exception #10797
- Use Zookeeper Prometheus metric provider to export Zookeeper metrics #10803
- Always allow system topic for transaction buffer snapshot auto create #10876
Security
- Optional auth method name header in HTTP authentication #6799
- Topics level authorization support #7523
- Authorization for setting topic/subscription auto-creation and subscription expire time on namespace #7900
- Allow serializable stream-provider field into AuthenticationTls #10020
- Add configuration for running OWASP Dependency Check for all modules #10288
- Update default TLS protocols to TLSv1.3 and TLSv1.2 for the broker and proxy #10598
Broker
- Share EventLoopGroup between broker and BookKeeper client #2603
- Dispatch batch messages according to consumer permits 7266
- Improve the max pending publish buffer mechanism 7406
- Allow disabling HTTP TRACE/TRACK verbs #7907
- Fix DispatchRateLimiter does not take effect #8611
- Validate namespace isolation policy regex before updating #8804
- Perform automatically cert refresh for pulsar-admin #8831
- Fix updating rack info dynamically #8844
- Fix deadlock when checking message expiration #8877
- Support limit max topics per namespace #8942
- Make ledger rollover check task internally #8946
- Clean up topic that failed to unload from the cache #8968
- Support get broker response for the message acknowledgement #8996
- Support message index for a topic #9039
- Supporting limit max topics per namespace by namespace policy #9042
- Streaming dipsatcher support #9056
- Make Netty acceptor threadPool size configurable #9061
- Fix deadlock when unloading namespace bundles #9116
- Fixed checking for maxTopicsPerNamespace #9121
- Change the service URL to a not required param when creating a cluster #9127
- Support setting replicator dispatch rate policy at the topic level #9175
- Fix max topic for namespace does not work #9193
- Fix memory leak of the managed ledger interceptor #9194
- Fix max consumers per topic cannot be disabled at the namespace level #9214
- Support schema REST API for V1 topics #9218
- Fix peek message metadata while enabling broker entry metadata #9255
- Support fetching metadata from entry data in publish callback #9257
- Wait for the async broker port listener close operations to complete at shutdown #9308
- Support script based mapping network topology #9363
- Make managed ledger storage configurable #9397
- Support setting enabled subscription types #9401
- Fixed NPE and cache invalidation in leader election #9460
- Fix exception when get an optional field for Protobuf message #9468
- Ignore replicated subscription configurations from the client when disabled by broker #9523
- Support expose producer metrics through Prometheus endpoint #9541
- Fix NPE that occurs in PersistentStickyKeyDispatcherMultipleConsumers when debug log enabled #9587
- Make LocalPolicies immutable to avoid concurrent modify inconsistent #9598
- Fix writing/encoding issue of GenericJsonRecord #9608
- Expose the native record for struct schema #9614
- Add metrics for producer throttling #9649
- Fix MaxUnackedMessagesPerConsumer cannot be changed dynamically and cannot be disabled #9657
- Added support for force deleting tenant #9677
- Fix managed ledger not found exception when force delete namespace #9691
- Reduce CPU consumption of metrics creation #9735
- Ensure read-lock is not continuously held on a section while iterating over concurrent maps #9787
- Add pending read subscription metrics to stats-internal #9788
- Allow broker to discover and unblock stuck subscription #9789
- Disabled the tenants/namespaces force deletion by default #9819
- Add metrics for the connections of the broker #9876
- Make readModifyUpdate in MetadataCacheImpl thread-safe #9900
- Optimize NamespaceBundle.toString() which is on the hot path #9976
- Support set compaction threshold in broker.conf #9989
- Support set properties for a namespace #10015
- Fix cannot cleanup expired data after managed-ledger restart #10087
- [Default configuration] Enable sticky read by default #10090
- Add time based backlog quota #10093
- Fix IllegalStateException in PersistentReplicator #10098
- Support set max consumers per subscription for non-persistent dispatcher #10121
- Limit the number of producers/consumers that can connect per topic for each IP address #10188
- Close namespace clients when PulsarService is closed #10196
- Shutdown Broker gracefully, but forcefully after brokerShutdownTimeoutMs #10199
- Update the authentication data when an authentication refresh happens #10203
- Add preciseTopicPublishRateLimiterEnable to broker.conf #10216
- Fix the typo in the chunkedMessageRate stats and metrics #10223
- Fix entry cache size to be a negative value #10225
- Fix replicated subscriptions related LightProto issues #10247
- Pause the replicated subscriptions snapshots when there is no real traffic #10292
- Fix the inconsistency of advertised address #10312
- Support listenerName for HttpLookupService #10319
- Support truncate topic #10326
- Fix authorization error if partition number of partitioned topic is updated 10333
- Fix managed ledger name that transaction log used #10334
- Catch topic policy not hit exception in handleSubscribe #10341
- Fix ConcurrentModificationException when attempting to update local broker data #10347
- Support seek to separate messageId/time for multiple topic reader #10348
- Resource locks should automatically revalidate after a metadata session is re-established #10351
- Add authentication data for the remote cluster #10357
- Support array type claims in JWT #10375
- Optimize locks in AuthenticationAthenz #10381
- Prevent carrying state of PositionImplRecyclable when recycled #10404
- Dispatch messages to consumer with permits #10417
- Fix NPE in unblock stuck subscription task when dispatcher has not created #10430
- Fix topic loading fails without any error when replicator init fails #10432
- Set timeout to unblock web-threads on update partition API #10447
- Fix CPU 100% when deleting namespace #10454
- Remove residual info after forcibly deleting the namespace #10465
- Fix potential memory leak of TopicPolicies #10466
- Fix publish callback's entry data is null during ledger rollover #10467
- Fix readModifyUpdate should return the new value in the future for metadata cache #10474
- Fix partitioned system topic check issue #10529
- Removed AspectJ based metrics for ZooKeeper #10533
- Allow creating MetadataCache with custom serde #10543
- Fix ack receipt version check issue #10551
- Expose average message size metrics for a topic #10553
- Fixed missed ZK caching when fetching list of namespaces for a tenant #10594
- Setup pulsar cluster with MetadataStore #10600
- Setup initial namespaces with MetadataStore #10612
- Convert bundle split into an HTTP async operation #10619
- Add metrics for non-contiguous deleted messages range #10638
- Fix consumer related topic stats only available while consumer or reader are connected #10644
- Allow configuring the number of BK client worker threads #10649
- Fix ConcurrentOpenLongPairRangeSet remove all ranges #10656
- Ensure all the ReadHandle gets properly closed on cache invalidation #10659
- Avoid the context switch when managedLedgerNewEntriesCheckDelayInMillis=0 #10660
- Allow configuring busy-wait in broker and client #10661
- Use a single JWT parser instance #10664
- Fix issues in advanceNonDurableCursors #10667
- LockManager use a concurrent hash map to handle locks notifications #10680
- Avoid making copies of internal maps when iterating #10691
- Fix issue where StackOverflowError occurs when trying to redeliver a large number of already acked messages #10696
- Fix NPE when filtering read entries #10704
- Fix deadlock when enabling managed ledger interceptor #10706
- Fix inconsistent behavior in LongPairRangeSet #10713
- Fix repeated iterator generation of LongPairRangeSet #10722
- Cancel scheduled tasks as the first step in closing for the managed ledger #10739
- Prevent race conditions between timeout and completion of managed ledger #10740
- Add advertised listener name for geo-replicator #10751
- Add a read lock when traversing batchDeletedIndexes #10763
- Replace use of Prometheus client CollectorRegistry.getSampleValue #10764
- Release OpAddEntry.data when entry is copied and discarded for the managed ledger #10773
- Avoid warning logs on topic policies not present #10785
- Remove pulsar-zookeeper module and call zookeeper quorumPeerMain to start zookeeper #10823
- Fix consumer stuck issue due to reuse entry wrapper #10824
- Fix possible data race in getFirstAvailableConsumerPermits #10831
Functions
- Update default function log location in function_worker.yaml #8470
- Optimize batch source discovery and task ack #8498
- Provide an interface for functions worker service #8560
- Improve naming of threads used in batch source #8608
- Fix the reader used to read the metadata topic during worker service initialization not been closed #8637
- Add timeout to hasMessageAvailable to leader election process #8687
- Support key based batch builder for Go Functions #8761
- Fix panic when discarding message for Go Functions #8776
- Move initialize dlog namespace metadata to bin/pulsar #8781
- Support URL fetching for Go/Python Functions #8808
- Add cmd flag for retaining key ordering #8886
- Supports to use package command #8973
- Support create and update sink with package name #8987
- Support create and update source with package name #8988
- Allow stats operations not to be blocked in functions #9005
- Adding timeout to open table call for function state #9006
- Fix get function info error of REST API #9115
- Improve the error message when creating sinks #9131
- Expose Pulsar Admin through Function context #9246
- Enable Function Workers to use exclusive producer to write to internal topics #9275
- Add additional logging when setting up state table for function instance #9304
- Allow memory limit to be set for the pulsar client used in the ThreadRuntime in Pulsar Function #9320
- Make admin operations on Statestore non blocking #9348
- Fix maxPendingAsyncRequests not affect Kubernetes Runtime #9349
- Add download directory support to function Kubernetes Runtime #9377
- Optimize built-in source/sink startup by eliminating redundant NAR unpacking and checksum calculation #9413 #9500
- Enhance Kubernetes manifest customizer with default options #9445
- Fix possible deadlock on broker-function service startup #9499
- Close InputStreams properly #9568
- Add maximum allowed amount of resources setting for functions #9584
- Support writing general records to Pulsar sink #9590
- Move metrics port configuration to InstanceConfig #9610
- Add resource granularity settings for functions #9736
- Prevent invalid broker or proxy configuration for authorization #9746
- Log stacktraces of threads that failed to terminate on shutdown within timeout in ExecutorProvider #9840
- Support get partition index for a Record #9947
- Deleting a Pulsar Function with a name that includes a colon character crashes the pulsar broker #9946
- Exposing Prometheus metrics for Pulsar function local run mode #10156
- Expose local run config metrics-port-start to CLI #10185
- Fix Pulsar Function localrun with multiple instances and metrics server is enabled #10208
- Improve localrun performance by using JVM options #10273
- Fix function API can not use authdata to check superuser #10364
- Fix potential bug getting stats and remove unnecessary error log #10500
- Fix deadlock on Monitoring thread blocked by LeaderService.isLeader() #10502
- Allow user to set custom configs to plugin worker service #10504
- K8s Function Name Length Check Allows Invalid StatefulSet #10531
- Get function cluster from broker config when start function worker with broker #10552
- Process async results in the same Java runnable thread #10618
- Support using AutoProduceBytesSchema as the function output schema #10716
- ReflectionUtils use Class.forName in order to properly discover classes in Functions Runtime while using DefaultImplementation #10827
- Fix the out of index issue when dispatch messages based on the avgBatchSizePerMsg #10828
IO Connectors
- [Kafka Source Connector] Fix invalid topic name generation #9035
- [Kafka Source Connector] Allow managing Avro encoded messages #9448
- Configure Pulsar IO connector YAML file generator for all connectors #9629
- [Kafka Source Connector] KeyValue schema support for KafkaBytesSource #10002
- Allow using GenericObject for developing a Sink connector #10034
- Sink unwrap internal AutoConsumeSchema and allow to handle topics with KeyValue schema #10211
- Using ObjectMapper instead of Gson to parse Source/SInk configuration #10441
- Expose subscription type in the SinkContext #10446
- Add the ability to seek/pause/resume for a topic to the SinkContext #10498
- Use Message.getReaderSchema() in Pulsar IO Sinks when possible #10557
- [Kinesis Sink Connector] Fix backoff class not found #10744
- [Kinesis Sink Connector] Fix does not acknowledge messages #10769
- [Kafka Sink Adaptor] Support non-primitive schemas #10410
Pulsar SQL
- Fix the misleading setting in presto configuration #8549
- Fix injection factory cast error #9472
- Add max split entry queue size bytes limitation #9628
- Fix Pulsar SQL query bytes schema data error #9631
- Fix Pulsar SQL query compression data #9663
- Support native Protobuf decoder #9841
- Support query uppercase topic name #9980
- Only return persistent topic when list tables #10368
- Fix Presto startup on JDK11 #10559
- Fix pulsar sql issues when run select count(*) for the table with primary schema #10840
Tiered Storage
Proxy
- Fix the proxy does not support partitioned topic auto-creation type #7903
- Enable Conscrypt for Jetty in the Broker and in the Proxy #10541
Java Client
- Fix warn log on the producer side when duplicated messages have been dropped #8729
- Allow setting subscription name for Reader #8801
- Fix authParams showing in log with secret string(*****) #8910
- Avoid enabling DLQ on Key_Shared subscription #9163
- Add support for the JSON format token #9313
- Expose reached end of topic API for Reader/Consumer #9381
- Fix returned a completed future when acknowledging a batch message before complete the individual acknowledgments #9383
- Fix NPE when strip checksum for the producer #9400
- Fix inconsistent equals and hashCode for MessageIds #9440
- Allow disabling producer max queue size #9650
- Include pulsar-client-admin-api in the shaded version of pulsar-client-admin #9689
- Fix NPE in the ClientCnx #9761
- Fix DLQ can't work with AUTO_CONSUME schema #9935
- Fix NPE caused by null value of SchemaInfo's properties #9985
- Support multi-topic reader #9995
- Process messages from different partitions on different listener threads #10017
- Ensure close resource to avoid memory leak #10028
- Support set start message ID for each topic/partition on Reader #10033
- Add a JSON RecordBuilder to the GenericJsonSchema #10052
- Allow GenericRecord to wrap any Java Object #10057
- Fix NPE while configuring consumer builder #10063
- Support get native schema #10076
- Support KeyValue on Message.getValue() when using AutoConsumeSchema #10107
- Change the default retry topic name and dead letter topic name #10129
- Fix KeyValue with SEPARATED encoding for the GenericObject #10186
- Ensure download schema before decoding the payload for the AUTO_CONSUME schema #10248
- Fix typo of the maxPendingChunkedMessage method #10223
- Trait NONE schema as BYTE schema for AUTO_CONSUME schema #10277
- Fix pause consume issue with MultiTopicsConsumerImpl #10305
- Make message consumption thread safe and lock-free #10352
- Reset state before recycling OpSendMsg instance #10405
- Fix hasMessageAvailable return true but can't read message #10414
- Fix NPE in GenericJsonRecord #10482
- Fix behaviour of Schema.AUTO_CONSUME() with KeyValueSchema and multi versions #10492
- Avoid sending flow requests with zero permits #10507
- Make failPendingMessages called from within the ProducerImpl object mutex #10528
- Add schemaType field in SchemaHash #10573
- Fix NPE when ACK grouping tracker checks duplicated message id #10586
- Support consume multiple schema types messages by AutoConsumeSchema #10604
- Fixed issues in pulsar-client shading configuration #10614
- MessageCrypto interface should not expose Netty ByteBuf class in the API #10616
- Added org.apache.bookkeeper:cpu-affinity to shaded profile #10681
- Skip the periodic re-check of the partitions count on non-partitioned topic #10708
- Unlock the write lock of the UnAckedMessageTracker before call redeliverUnacknowledgedMessages #10768
- Fix AutoConsumeSchema decode data without schema version #10811
C++ Client
- Fix dangling reference bug in getRandomName #8596
- Optimize batch message buffer allocation #8749
- Make pool connections configurable in perf tools #8913
- Support setting listener name #9119
- Fix batch message handling of the UnAckedMessageTracker #9170
- Fix ServerError is not converted to string in log #9277
- Remove Boost::System runtime dependency #9498
- Removed usages of boost::regex #9533
- Account for different variables names on different CMake versions #9559
- Allow to disable static or dynamic lib at build time #9570
- Avoid multiple compilations of same source files #9675
- Support configure debug level logs simply #10031
- Add /opt/homebrew/ as a possible path for OpenSSL on Mac #10141
- Fix race condition in MemoryLimitController #10142
- Fix releasing semaphore and memory quota after send timeout #10144
- Allow configuring memory limit from C API #10145
- Fix use-after-free undefined behavior due to object lifetime problem #10220
- Support enable replicate subscription #10243
- Fix C++ client cannot be built with Boost <=1.53 #10307
- Support check connect state #10349
- Avoid sending flow requests with zero permits #10506
- Add single file logger factory #10712
- Reduce redeliverMessages when message listener is enabled #10726
Python Client
- Replace Exceptions with PulsarExceptions #7600
- Added logger wrapper support to make logging configurable #7713
- Initial python 3.9 client wheel build support #9389
- Support enable replicate subscription #10243
WebSocket
- Negative acknowledge support #8249
- Support deliverAt and deliverAfter attribute #8945
- Fix returned status code does not depend on the exception type #9031
- Allow to consume and pass message to client without decryption #10026
- Support pong command #10035
Pulsar Admin
- Support get applied message TTL policy for a topic #9225
- Support get applied inactive topic policy for a topic #9230
- Support get applied delayed delivery policy for a topic #9245
- Support get applied max unacked message per subscription policy for a topic #9290
- Support get applied max producer for a topic #9293
- Support get applied max consumer for a topic #9296
- Support get applied deduplication status policy for a topic #9339
- Support get applied retention policy for a topic #9362
- Support get applied offloader policy for a topic #9505
- Support get applied max unacked messages per consumer for a topic #9694
- Support get applied dispatch rate policy for a topic #9824
- Support get applied subscription dispatch rate policy for a topic #9827
- Support get applied backlog quota policy for a topic #9828
- Support get applied persistence policy for a topic #9831
- Support get applied cluster subscribe rate for a topic #9832
- Support get applied replicator dispatch rate for a topic #9833
- Support get applied compaction threshold #10038
- Lazily init PulsarAdmin in PulsarAdminTool #9312
- Fix create partition of existing topic does not throw RestException #9342
- Support get service URL of the leader broker #9799
- Support get persistent topics or non-persistent topics for pulsar admin client #9877
- Fix can not disable and remove max consumer per subscription #10070
- Add support for setting time based limit on backlog quota #10401
- Make client-admin-api to use interfaces with builders instead of POJOs #10818
Tools
- [Pulsar Perf] Support load WebSocket service URL from config file #9000)
- [BookKeeper Shell] Make bookkeeper shell more user friendly #9281
- [Client Tools] Supports end-to-end encryption #9615
- [Pulsar Perf] Support specify topics and subscriptions #9716
- [Client Tools] Allow printing GenericRecord contents #9785
- [Pulsar Perf] Fix compatibility issues with previous version #9838
- [Pulsar Perf] Add ability to create partitioned topics #9859
- [Client Tools] handle data with different schemas when consuming messages #10301
- [Client Tools] Fix NoClassDefFoundError when running pulsar cli tools in dev directory #10807
Dependencies
- Upgrade Kubernetes client and remove sundr-codegen #8576
- Upgrade ZooKeeper to 3.6.2 and Curator to 5.1.0 #8549
- [CVE-2020-26238] Upgrade cron-utils to 9.1.3 and sprint-context to 5.3.1 #8822
- Upgrade Swagger to 1.6.2 and Maven Swagger Plugin to 3.1.7 #8845
- Upgrade to Apache Avro 1.10.2 #9898
- Add jersey-client as dependency of pulsar-client-auth-sasl #10055
- Upgrade Athenz version to 1.10.9 and remove yahoo.bintray.com repository #10079
- [CVE-2020-15250] Upgrade junit version to 4.13.1 #10147
- [CVE-2020-8908,CVE-2018-10237] Upgrade jclouds to 2.3.0 #10149
- Remove jackson-mapper-asl dependency to resolve multiple CVEs #10262
- [CVE-2021-21409] Upgrade Netty to 4.1.63.Final #10266
- [CVE-2018-12541] Upgrade vertx to 3.9.7 #10261
- Upgrade BookKeeper to 4.14.1 #10686
- [CVE-2021-28169] Upgrade Jetty to 9.4.42.v20210604 #10907