v1.2.5 (includes v2.1.4)

Now includes a new implementation of a Spark connector that while initially based on the v1 (balanced) connector is now different enough to warrant a separate directory in the repository named 'v2' to distinguish it from the existing 'balanced' connector. The initial version of that connector being released is version 2.1.4. Both v1 and v2 are being maintained (unlike the connector located in the 'archived' directory) and the fact the new connector is dubbed 'v2' doesn't mean that you need to upgrade from the latest version of 'v1' if it is working for you (for example 'v2' is currently slower than 'v1').

v1.2.5

• Fixes a slow memory leak, is it highly recommended that you upgrade to this version ASAP, especially if you are seeing memory usage growth for the connector over time.

v2.1.4 (initial public release)

• This 'v2' connector takes a different approach at mapping JetStream streams (which are partition-less) to partitions (that Spark expects), and as such presents different throughput and latency profiles.

v1.2.4

The time stamp in the second column is now the time at which the adapter received the message from NATS rather than the time at which the message was recorded in the stream (as that time stamp is available in the JS Metadata for the message).

You can adjust the format of the time stamp through the "nats.datetime.format" configuration attribute.

You can now pass the name of your own class implementing the SSL context factory through the "nats.ssl.context.factory.class" configuration attribute.

v1.2.3

Adds any message headers and the complete JS metadata info for the message in two additional columns

Update spark version to 3.3.4

Retire the partitioned flavor of the connector as it's not being actively developed anymore

v1.2.2

Adds support for compressed payload to the source:

• The source configuration optional parameter "nats.storage.payload-compression" specifies the compression type to be used to decompress the payload of the message. Currently supported values are: "zlib" (plus "uncompressed" and "none"). If the payload of a particular message cannot be decompressed, it is then passed on as-is.

Improve handling of possible underlying connectivity issues.

Use fast binding when subscribing to the consumer to remove the requirement on the $JS API Stream info subject being exported when sourcing and using an existing durable consumer imported from another account.

v1.2.0 Use a single consumer with multiple subject filters

NOTE: Now requires NATS servers version 2.10 or above.

Balanced:

• Using the new multi-subject filters in consumers feature of NATS 2.10. Now only creates and uses a single consumer for all the subjects being listed in the config, rather than one consumer per subject.

Partitioned:

• no change (as it already assumes the streams/consumers are defined administratively)

v1.1.5 Config improvements: can control whether the balanced connector tries to define the stream/consumer (enabled by default, can be adjusted)

Balanced:

• Adds new "nats.js.define-consumer" config parameter (defaults to true) which means the connector will not try to create the stream consumer
• "nats.js.define-stream" config parameter now defaults to true

Partitioned:

• no change (as it already assumes the streams/consumers are defined administratively)

v1.1.4 Config improvements, doesn't try to define the stream to consume from (by default, can be adjusted)

1.1.4 TLS connection, stream creation is configurable and off by default

• Balanced doesn't try to create the stream unless the property nats.js.define-stream is set to "true"
• Update NATS Java library to 2.17.1-SNAPSHOT
• Uses the new functionality of the Java NATS library to construct the SSL context
• You can specify only the truststore and not the keystore if you don't want to do mTLS but still check the server's cert using the truststore
• You can specify the log4j properties file that is used when isLocal is set to true using the LOG_PROP_PATH environment variable. This means that the file which was previously hard-coded doesn't even need to exist for the connector to run when isLocal is false.
• Assembly SBT plugin and task added if you want to create an uber-jar

v1.1.3 [ADDED] JS API prefix and user name/password config options.

[ADDED] JS API prefix and user name/password config options.

v1.1.2: 1.1.2 [IMPROVED] Defines the stream only if not already defined

Release v1.1.2

Balanced connector: only defines the stream if it is not already defined, adds the ability to specify the replica count.

Release v1.1.1

Can now use both NATS creds and TLS client certificates at the same time

Default stream storage type for balanced is now "file"