v2.3.2

2.3.2 — 2019-05-30

Fixes

• Validate admin operation on topic with authoritative parameter #4270
• fix bug with source local run #4278
• fix cannot use size (type _Ctype_int) as type _Ctype_ulong #4212
• Fix segfault in c++ producer #4219
• AlwaysCompatible doesn't use AlwaysSchemaValidator in 2.3.1 component/schemaregistry #4181
• Avoid potentially blocking method during topic ownership check #4190
• [pulsar-broker]Fix: client-producer can't reconnect due to failed producer-future on cnx cache #4138
• Removing # TYPE comment from topic metrics in Prometheus #4136
• For functions metrics in prometheus also remove TYPE #4081
• Fix: set receive queue size for sinks #4091
• Fix: Exception when switch cluster from auth enabled to auth disabled #4069
• Fix update cli source sink #4061
• Fix connectors nested configs #4067
• For functions metrics, avoid having HELP #4029
• Fix Python functions state which is completely broken #4027
• [issue #3975] Bugfix NPE on non durable consumer #3988
• Fix: Function auth should ignore exception because it might be anonymous user #4185
• [pulsar-function] fix worker-stats broken admin-api #4068
• fix errors in sql doc #4030
• Fix the swagger files generated by removing troublesome class #4024
• [pulsar-function] fix broken backward compatibility with v1-namespace while registering function #4224
• Revert dup consumer and related code #4142
• [issue 4274][pulsar-io]Add double quotation marks for metrics with remote_cluster #4295

Enhancements

• By default, auto configure the size of Bookie read/write cache #4297
• Upgrade to BookKeeper 4.9.2 #4288
• [pulsar-function] support bookie authentication from function-worker #4088
• Optimizing performance for Pulsar function archive download #4082
• allow users to update output topics for functions and sources #4092
• improve data-generator source performance #4058
• [client] Set actual topic name to partitioned consumer #4064
• ack records in datagenerator print sink #4052
• [security] Upgrade athenz libraries #4056
• [python client] Handle subrecords in JsonSchema encoding #4023
• [Issue 3987][pulsar-broker]Handle config is null when create tenant #4019
• Add bookkeeper client version constraint #4013
• Improve error handling for triggering function when there is a schema mismatch #3995
• [pulsar-broker] add producer/consumer id in error-logging #3961

2.3.1

Fixes

• Fixed C++ batch acks tracker to evict message from sendList array. This was causing a slowdown in
  C++ consumers #3618
• Allow publishing messages >5MB with batching (when they compress to <5MB) #3673 and #3718
• Use at least 8 threads in Jetty thread pool. This fixed deadlocks in Jetty requests handling #3776
• Fixed Reader.HasNext() in Go client #3764
• Fixed increasing consumer permits after ack dedup operation. #3787
• Set the dedup cursor as "inactive" after recovery #3612
• Fix read batching message by pulsar reader #3830
• Fix submit function with code specified via URL #3934
• Fixed reader reading from a partition #3960
• Fixed issue with Authorization header missing after client gets redirected #3869

Enhancements

• Added producer.flush() on Python #3685
• Introduced schema builder to define schema #3682
• Support passing schema definition for JSON and AVRO schemas #3766
• Exposing InitialPosition setting in Python consumer #3714

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/21?closed=1

2.3.0

General

• Support for schema definitions in the Pulsar Python client library
• PIP-25: Token based authentication #2888, #3067 and #3089
• Updated to Apache BookKeeper 4.9.0
• ZStandard compression codec #3159. Note that when a producer choose Zstd compression, a consumer will need to be at least at version 2.3.0
  to be able to correctly receive the messages.
• Support for Java 11 #3006
• Added Schema.AUTO_PRODUCE type to allow to publish serialized data and validate it against the
  topic schema #2685
• Added Schema.KeyValue to allow for schema to be validated on message keys as well as payloads. #2885
• Support TLS authentication and authorization in standalone mode #3360
• When creating namespace, use local cluster by default #3571
• Tag BookKeeper ledgers created by Pulsar with topic/subscription names for info/debug purposes
  #3525
• Enabled sticky reads in BooKeeper reads to increase IO efficiency with read-ahead #3569
• Several optimization in Pulsar SQL Presto connector (#3128, #3135, #3139, #3144, #3143)
• Configure Pulsar broker data cache automatically from JVM settings #3573
• Reuse the SSL context objects #3550
• Automatic schema update can be disabled through admin interface #2691
• Support Dead-Letter-Queue from WebSocket proxy #2968
• Pull-mode for WebSocket proxy #3058
• Export Jetty stats to Prometheus #2804
• Added stats for Pulsar proxy #2740
• Allow subscribers to access subscription admin-api #2981
• Make brokers read on closest Bookie in a multi-region deployment #3171

Fixes

• Fixed deadlock in reusing ZookKeeper event thread #3591
• In functions log topic appender, don't set producer name #3544
• When cursor recovery encounters empty cursor ledger, fallback to latest snapshot #3487
• Fixed C++ regex-consumer when using HTTP service URL #3407
• Fix race condition: broker not scheduling read for active consumer #3411

Pulsar IO

• Added Debezium connector for Change-Data-Capture into Pulsar #2791
• Added MongoDB connector #3561
• Added Elastic Search connector #2546
• Added HBase sink #3368
• Added Local files connector #2869
• Report source/sink stats in Prometheus #3261
• Allow filtering in Twitter Firehose connector #3298
• Sources/Sinks can be launched using fat jars as well #3166

Pulsar Functions

• Added Kubernetes runtime #1950
• Secrets interface #2826
• Cleanup subscriptions when deleting functions #3299
• Add Windowfunction interface to functions api #3324
• Support for accessing state in Python #2714
• Support submitting Python functions as wheel file
• Support submitting Python functions as Zip file with dependencies included #3321
• Add minimum amount of resources to run setting for functions #3536
• Fixed the behavior of Function start/stop #3477

Java client

• Moved Pulsar v1 client API into separate artifact #3228.
  
  Applications that are using the Pulsar v1 API, deprecated since 2.0 release, need to update the Maven dependency to use the pulsar-client-1x artifact instead of pulsar-client. Eg.

  <dependency>
      <groupId>org.apache.pulsar</groupId>
      <artifactId>pulsar-client-1x</artifactId>
      <version>2.3.0</version>
  </dependency>

• Fixed shading issues with Javadoc bundled in client jars by separating the API in a different Maven module #3309
• Improve Javadocs #3592
• Support specifying multiple hosts in pulsar service url and web url #3249
• Automatically discover when partitions on a topic are increased #3513
• Added Client.getPartitionsForTopic() #2972 (Javadoc)
• Added Consumer.pauseMessageListener() and Consumer.resumeMessageListener() #2961
• Removed shading relocations for Circe-checksum and lz4 libraries, to ensure native libraries
  are correctly loaded when using shaded client lib. #2191

Python client

• Fixed Message.properties() #3595

Go client

• Added Producer.flush() to flush all outstanding messages #3469
• Support Consumer.Seek() #3478
• Added Message.Topic() #3346
• Allow to specify SubscriptionInitPos option in ConsumerOptions #3588
• Added TLS hostname verification #3580
• Allow to link statically against libpulsar.a#3488
• Expose Producer.LastSequenceID() and Message.SequenceID() #3416

C++ client

• Enable batching by default when using sendAsync() #2949
• Allow to specify schema info in Avro format #3354
• Added Producer.flush() to flush all outstanding messages #3020
• Added TLS hostname verification #2475
• Allow to specify SubscriptionInitialPosition #3567
• Added Message.getTopicName() #3326
• Added Cosnsumer.receiveAsync() #3389
• Build libpulsar.a with all required dependencies #3488
• Removed Boost from Pulsar API headers #3374

Adaptors

• Kafka client wrapper, added support for explicit partitioning and custom partitioner #3462
• Support config auto.offset.reset to Pulsar KafkaConsumer #3273
• In Apache Flink connector, added support for Batch Sink API (2979#, #3039 and #3046)
• Added Java batch examples for Flink adaptor
• Added Java streaming examples for Flink adaptor
• Added Scala examples for Flink adaptor

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/18?closed=1

Downloads at https://pulsar.apache.org/download

2.2.1

This release includes fixes for 2.2.0 release. In particular:

• Fixed issue when proxy HTTP admin API requests Pulsar proxy #3022
• Fixed Consumer.unsubscribe() in Python client library #3093
• Fixed ZLib decompression in C++ client #2854
• Fixed Pulsar functions context publish in Python #2844

For a complete list of issues fixed, see https://github.com/apache/pulsar/milestone/19?closed=1

2.2.0

This is the first release of Pulsar as an Apache Top Level Project

This is a feature release, including several new features, improvements and fixes for issues reported for 2.1.1-incubating.

• Pulsar SQL
• Pulsar Java Client Interceptors
• Integration of functions and io with schema registry
• Dead Letter Topic
• Flink Source connector
• JDBC Sink Connector
• HDFS Sink Connector
• Google Cloud Storage Offloader

2.1.1-incubating

This release fixes issues reported for 2.1.0-incubating.

• #2473 - Downgrading ZK to stable version 3.4.13
• #2219 - Cpp client: add PatternMultiTopicsConsumerImpl to support regex subscribe
• #2387 - Fixed race condition during expansion of concurrent open hash maps
• #2348 - Fix NPE when splitting and unloading bundle
• #2223 - fix bug in FunctionRuntimeManager involving not cleaning up old invalid assignments
• #2367 - [compaction] make topic compaction works with partitioned topic
• #2203 - Make sure schema is initialized before the topic is loaded

The complete list of changes can be found at:

https://github.com/apache/incubator-pulsar/milestone/17?closed=1

2.1.0-incubating

This is the seventh release of Apache Pulsar since entering the ASF incubator.

There are several new features and major improvements:

• Pulsar IO: A connector framework for moving data in and out of Apache Pulsar leveraging Pulsar Functions runtime.
• Tiered Storage: An extension in Pulsar segment store to offload older segments into long term storage (e.g. HDFS, S3). S3 support is built in 2.1 release.
• Stateful function (preview) in Pulsar Functions
• Pulsar Go Client
• Avro and Protobuf Schema support

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/13?closed=1

2.0.1-incubating

This release fixes issues reported for 2.0.0-rc1-incubating.

• #1893 - Fixed issues with Python packages on PyPI
• #1797 - Proxy doesn't strip the request
  URL for admin requests correctly
• #1862 - Fix REST APIs provided by Pulsar proxy

The complete list of changes can be found at:
https://github.com/apache/incubator-pulsar/milestone/14?closed=1

Release 1.22.1-incubating

This is the sixth release of Apache Pulsar since entering the ASF incubator.

Major changes in this release include:
Fixes
[#1660] Deadlock while closing non persistent topic
[#1591] Deadlock while closing non shared consumer
[#1554] Handle invalid mark delete position at managed cursor
[#1262] Broker should not start replicator for root partitioned topic
[#1662] NPE when cursor failed to close empty subscription
[#1370] Relocate service files for shading pulsar-client-admin module
[#1265] Fixed lookup redirect logic on Proxyside
[#1428] Handle Race condition in concurrent bundle split
[#1817] Fixed mem leak when acknowledging while disconnected from broke
[#1851] Fixing resource leak due to open file descriptors in SecurityUtility.

The complete list of changes can be found at:
https://github.com/apache/incubator-pulsar/milestone/15?closed=1

v2.0.0-rc1-incubating

This is the fifth release of Apache Pulsar since entering the ASF incubator and the first time we increase major release number.

There are several new features and major improvements:

• Pulsar functions: Lightweight
  compute framework
• New type-safe Java API for producer/consumers
• Schema registry — Enforce schema on topics
• Topic compaction — Out of band compaction of messages to allow consumer to fetch a
  snapshot with last published message for each message key.
• Upgraded to Apache BookKeeper 4.7.0
• Performance improvements — Up to 3x throughput improvements compared to Pulsar-1.22 and
  99.9 Pct publish latencies <10ms
• Simplified terminology and admin tools
  • Renamed "property" into "tenants"
  • Short topic names: my-topic
  • Topics independent of cluster names: my-tenant/my-namespace/my-topic

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/12?closed=1