v2.6.0

Noticeable Changes

• Automatically merge commits by @erikvanoosten in #1073
  In this PR zio-kafka automatically merges all commits that were generated in the course of a single run of the runloop. This frees users from having to merge streams and do the commit merging themselves.
• Add processing timeout at the stream level by @erikvanoosten in #1009
  Say good bye to the runloop-timeout! From this version on we respect Kafka configuration max.poll.interval.ms. When a stream does not consume any records for max.poll.interval.ms or longer (while records are available), the stream will be interrupted and the consumer closed. This allows another member of the consumer group to take over the work.
• Add tuning options and docs by @erikvanoosten in #1081
  A new tuning guide on https://zio.dev/zio-kafka/.
• Enable the consumer to commit an offset with metadata by @flavienbert in #1067

See the PR descriptions for more information.

All Changes

• Update zio-kafka, zio-kafka-testkit to 2.5.0 by @zio-scala-steward in #1070
• Update README.md by @zio-assistant in #1071
• Gitignore cleanup by @erikvanoosten in #1072
• Small cleanup by @erikvanoosten in #1065
• Do not run CI and benches for gh-pages branch by @erikvanoosten in #1075
• Add instructions for pruning the benchmark history by @erikvanoosten in #1077
• Automatically merge commits by @erikvanoosten in #1073
• Update zio-sbt to 0.4.0-alpha.22 by @erikvanoosten in #1078
• Update jackson-databind to 2.15.3 by @zio-scala-steward in #1080
• Add processing timeout at the stream level by @erikvanoosten in #1009
• Fix flaky test by @erikvanoosten in #1082
• Fix flaky test again by @erikvanoosten in #1083
• Add tuning options and docs by @erikvanoosten in #1081
• Add unit test for QueueSizeBasedFetchStrategy by @erikvanoosten in #1017
• Update sbt to 1.9.7 by @zio-scala-steward in #1084
• Update scalafmt-core to 3.7.15 by @zio-scala-steward in #1085
• Enable the consumer to commit an offset with metadata by @flavienbert in #1067
• Keep track of last pulled offset by @erikvanoosten in #1086

New Contributors

• @flavienbert made their first contribution in #1067

Full Changelog: v2.5.0...v2.6.0

v2.5.0

Noticeable Changes

• Update kafka-clients to 3.6.0 @zio-scala-steward (#1068)
• Drop support of Java 8, 9 and 10. Minimal supported version is Java 11 (#1059)

All Changes

• Update kafka-clients to 3.6.0 @zio-scala-steward (#1068)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.21 @zio-scala-steward (#1059)
• Update README.md @zio-assistant (#1062)
• Update README.md @zio-assistant (#1061)
• Update zio-streams, zio-test-sbt to 2.0.18 @zio-scala-steward (#1060)
• Update zio-kafka, zio-kafka-testkit to 2.4.3 @zio-scala-steward (#1058)

v2.4.3

Noticeable Changes

• [Breaking] Bootstrap servers is regular config @lukestephenson (#1038)

  • AdminClientSettings::bootstrapServers field has been removed and replaced with the AdminClientSettings::withBootstrapServers method.
  • ConsumerSettings::bootstrapServers field has been removed. Use ConsumerSettings::withBootstrapServers method instead.
  • ProducerSettings::bootstrapServers field has been removed. Use ProducerSettings::withBootstrapServers method instead.

• Fix Consumer.fromJavaConsumer and Producer.fromJavaProducer type signatures @devsprint (#1033)

All Changes

• Put back the ProducerSettings::driverSettings and AdminClientSettings::driverSettings methods to avoid a useless breaking change and to provide a more unified interface through all the settings case classes @guizmaii (#1057)
• Update zio, zio-streams, zio-test, ... to 2.0.18 @zio-scala-steward (#1056)
• Update zio-streams, zio-test-sbt to 2.0.17 @zio-scala-steward (#1053)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.19 @zio-scala-steward (#1054)
• Fix Scala Steward configuration @guizmaii (#1052)
• Use ZIO Scala Steward Github app with the scala-steward CI @guizmaii (#1051)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.18 @github-actions (#1044)
• Update zio, zio-streams, zio-test, ... to 2.0.17 @github-actions (#1049)
• Update sbt to 1.9.6 @github-actions (#1050)
• Update scala-library to 2.13.12 @github-actions (#1047)
• Update sbt-scalafix to 0.11.1 @github-actions (#1046)
• Bootstrap servers is regular config @lukestephenson (#1038)
• [Internal] Micro-optimisation: Only compute the requestedPartitions Set if necessary @guizmaii (#1021)
• Update scala3-library to 3.3.1 @github-actions (#1037)
• Update scalafmt-core to 3.7.14 @github-actions (#1034)
• Fix Consumer.fromJavaConsumer and Producer.fromJavaProducer type signatures @devsprint (#1033)
• [Internals] Fix build @guizmaii (#1032)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.17 @github-actions (#1031)
• Update scalafmt-core to 3.7.13 @github-actions (#1030)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.14 @github-actions (#1029)
• Update sbt to 1.9.4 @github-actions (#1027)
• Update zio-streams, zio-test-sbt to 2.0.16 @github-actions (#1026)
• Update zio, zio-streams, zio-test, ... to 2.0.16 @github-actions (#1025)
• Update logback-classic to 1.4.11 @github-actions (#1020)
• Update logback-classic to 1.3.11 @github-actions (#1019)
• Update README.md @zio-assistant (#1016)
• Update README.md @zio-assistant (#1015)
• Update zio-kafka, zio-kafka-testkit to 2.4.2 @github-actions (#1013)
• Update zio-logging-slf4j, ... to 2.1.14 @github-actions (#1014)

v2.4.2

Noticeable Changes

• Update kafka-clients to 3.5.1 @github-actions (#994)

• Add note about partition reassignment with multiple subscriptions @svroonland (#1010)

• Document pollTimeout setting @erikvanoosten (#995)

• new ConsumerSettings::commitTimeout configuration. Default value: 15 seconds
  This timeout is a workaround for a deadlock that can occur when committing. Details #852

• Consumer:

  • new Consumer.fromJavaConsumer function to make a zio-kafka Consumer

• Producer:

  • new Producer.fromJavaProducer function to make a zio-kafka Producer
  • Fix signature of Producer#produceAll @vladimirkl (#993)
  • Enriched interface with the following additional new methods:

  trait Producer {
  
    def produce(
      record: ProducerRecord[Array[Byte], Array[Byte]]
    ): Task[RecordMetadata]
  
    def produceAsync(
      record: ProducerRecord[Array[Byte], Array[Byte]]
    ): Task[Task[RecordMetadata]]
  
    def produceChunk(
      records: Chunk[ProducerRecord[Array[Byte], Array[Byte]]]
    ): Task[Chunk[RecordMetadata]]
    
    def produceChunkAsync(
      records: Chunk[ProducerRecord[Array[Byte], Array[Byte]]]
    ): Task[Task[Chunk[RecordMetadata]]]
  
    def produceChunkAsyncWithFailures(
      records: Chunk[ProducerRecord[Array[Byte], Array[Byte]]]
    ): UIO[UIO[Chunk[Either[Throwable, RecordMetadata]]]]
  
  }

• Improvement of the SslHelper.validateEndpoint mechanism to make it as resilient as Kafka: If some Kafka bootstrap servers are down, that shouldn't affect the result of this function. Details #950

All Changes

• [Internal] Don't run benchmarks on forks @guizmaii (#989)
• Producer.produceChunkAsyncWithFailures @lukestephenson (#879)
• Update embedded-kafka to 3.5.1 @github-actions (#1012)
• Add note about partition reassignment with multiple subscriptions @svroonland (#1010)
• Create Consumer and Producer from existing apache kafka Consumer/Producer @svroonland (#1011)
• Document setting pollTimeout @erikvanoosten (#995)
• Update logback-classic to 1.4.9 @github-actions (#1008)
• Update logback-classic to 1.3.9 @github-actions (#1007)
• [Internal] Reorganize Consumer code @guizmaii (#999)
• Update scalafmt-core to 3.7.12 @github-actions (#1006)
• Update scalafmt-core to 3.7.11 @github-actions (#1002)
• Update sbt-tpolecat to 0.5.0 @github-actions (#1001)
• Update sbt to 1.9.3 @github-actions (#1000)
• [Internal] Re-organise Producer code @guizmaii (#998)
• Enrich Producer interface @guizmaii (#990)
• Fix signature of Producer#produceAll @vladimirkl (#993)
• Update kafka-clients to 3.5.1 @github-actions (#994)
• Fix: Runloop might be suspended when assigned with many partitions @yaarix (#986)
• Fix: Make the SslHelper.validateEndpoint as resilient as Kafka. If some Kafka bootstraps servers are down, that shouldn't affect the result of the validation @guizmaii (#950)
• Update scalafmt-core to 3.7.10 @github-actions (#985)
• Improve Scalafix configuration @guizmaii (#984)
• Implement default commit timeout @vladimirkl (#982)
• Prevent asInstanceOf @erikvanoosten (#963)
• Fix CI @guizmaii (#981)
• Update scalafmt-core to 3.7.9 @github-actions (#979)
• Update sbt-tpolecat to 0.4.4 @github-actions (#978)
• Update scalafmt-core to 3.7.8 @github-actions (#976)
• Update sbt to 1.9.2 @github-actions (#973)
• Update sbt-tpolecat to 0.4.3 @github-actions (#972)
• Update scalafmt-core to 3.7.7 @github-actions (#971)
• Fix scala 2.12 scaladoc @erikvanoosten (#969)
• Introduce FetchStrategy to allow custom fetch strategies @erikvanoosten (#953)
• Update scalafmt-core to 3.7.6 @github-actions (#968)
• Update zio-kafka, zio-kafka-testkit to 2.4.1 @github-actions (#967)
• Update README.md @zio-assistant (#965)
• Update README.md @zio-assistant (#964)
• Only calculate initial state once @erikvanoosten (#962)
• Document ConsumerSettings better @erikvanoosten (#961)

🌱 Dependency Updates

• Bump actions/setup-java from 3.11.0 to 3.12.0 @dependabot (#1005)
• Bump scala-steward-org/scala-steward-action from 2.59.0 to 2.61.0 @dependabot (#997)
• Bump benchmark-action/github-action-benchmark from 1.17.0 to 1.18.0 @dependabot (#974)
• Bump scala-steward-org/scala-steward-action from 2.56.0 to 2.59.0 @dependabot (#975)

v2.4.1

Noticeable Changes

• [Fix] Issue #856 is now fixed
• [Fix] After a call to consumer.stopConsumption, starting a new consumption with the same Consumer instance was not doing anything (fixed in #857)
• [Breaking] The Producer.withProducerService(f) function has been removed. You need to replace it with ZIO.serviceWithZIO[Producer](f)

All Changes

• Micro-optimisation for manual consumption mode: Use only 1 IO to execute all the c.seek calls in doSeekForNewPartitions @guizmaii (#900)
• Using a Consumer without consuming shouldn't throw a RunloopTimeout exception @guizmaii (#857)
• Update scalafmt-core to 3.7.5 @github-actions (#952)
• Update sbt to 1.9.1 @github-actions (#948)
• Clean some code of LiveTransactionalProducer and Producer @guizmaii (#947)
• Update zio-kafka, zio-kafka-testkit to 2.4.0 @github-actions (#946)
• Update README.md @zio-assistant (#945)

v2.4.0

Noticeable Changes

1. Update kafka-clients from 3.4.1 to 3.5.0

2. Update Scala 2.12 to 2.12.8 and Scala 2.13 to 2.13.11

3. The optimistic prefetch mechanism, which we called "optimistic resume" in previous releases, introduced in v2.3.0 has been removed because of several performance issues.
   Instead, the prefetch mechanism, as present in v2.2 and earlier, has been put back and optimized by fixing a race condition in the enqueueing of internal messages to fetch more data and the next poll cycle starting.

   The prefetch amount is now controlled with a ConsumerSettings::maxPartitionQueueSize setting instead of the old ConsumerSettings::perPartitionChunkPrefetch setting.
   See ConsumerSettings::maxPartitionQueueSize documentation: #943

   The ConsumerSettings::enableOptimisticResume setting introduced in v2.3.2 has therefore been removed.

All Changes

• Document maxPartitionQueueSize setting @svroonland (#943)
• Review #803: Optimize partitionsToFetch computation @guizmaii (#942)
• Improve prefetch @svroonland (#803)
• Use less heap by calling offer directly @erikvanoosten (#940)
• Fix Consumer.consumeWith interface @guizmaii (#939)
• updating kafka-clients to version 3.5.0 @Casper12345 (#936)
• Revert "Optimistically resume partitions early" @svroonland (#932)
• [Internal] Only evaluate if (hasGroupId) once in getConsumerGroupMetadataIfAny @guizmaii (#931)
• Update scala-collection-compat to 2.11.0 @github-actions (#928)
• Revert "Update kafka-clients to 3.5.0 (#925)" @guizmaii (#927)
• Update embedded-kafka to 3.4.1 @github-actions (#926)
• Update kafka-clients to 3.5.0 @github-actions (#925)
• Update logback-classic to 1.3.8 @github-actions (#922)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.12 @github-actions (#924)
• Avoid using ZIO.succeed when wrapping Kafka code @guizmaii (#921)
• [DX] Fix ConsumerSettings interface @guizmaii (#919)
• Fix project compilation @guizmaii (#916)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.11 @github-actions (#912)
• Update README.md @zio-assistant (#915)
• Update scala-library to 2.12.18 @github-actions (#913)
• Update scala-library to 2.13.11 @github-actions (#914)
• Update zio-kafka, zio-kafka-testkit to 2.3.2 @github-actions (#911)

🌱 Dependency Updates

• Bump scala-steward-org/scala-steward-action from 2.54.0 to 2.56.0 @dependabot (#934)

v2.3.2

Noticeable changes

1. Update Scala3 from 3.2.2 to 3.3.0
2. Update kafka-clients from 3.4.0 to 3.4.1
3. New helper functions in the testkit. See #893 for details.
4. We now have an Adopters list. Don't hesitate to put the name of your company there if you use zio-kafka! 🔥
5. New setting in ConsumerSettings: ConsumerSettings::enableOptimisticResume, true by default.

This new setting allows you to disable an internal optimisation we call "optimistic resume".
This optimisation has been introduced in v2.3.0.
Some users have reported very high latencies when this optimisation is enabled.
Disabling it seems to resolve this latency issue. (More details, see #903)

Our benchmarks also show that this optimisation can negatively impact performance if you use manual subscriptions (Subscription.manual(...)).
Therefore, If you use manual subscriptions, you might want to try to disable the "optimistic resume" optimisation and see if it improves the throughput and/or latency you observe in production.

Please, report any improvement or issue observed with and without this optimisation activated in our ZIO Discord channel so that we better learn about the positive and negative impacts this optimisation can have on real-world workloads.

Thanks! ❤️

All Changes

• Fix: Runloop thread name is always zio-kafka-runloop-thread-0 @guizmaii (#909)
• Update Scala3 to v3.3.0 @guizmaii (#873)
• Update kafka-clients to 3.4.1 @github-actions (#906)
• Update zio-streams, zio-test-sbt to 2.0.15 @github-actions (#905)
• Update kafka-clients to 3.4.1 @scala-steward (#908)
• Update zio, zio-streams, zio-test, ... to 2.0.15 @github-actions (#904)
• Extract all code refactoring from #587 @guizmaii (#901)
• [Testkit] Allow to customise broker properties @V-Lamp (#893)
• Document optimistic resume @erikvanoosten (#894)
• Update CI definition after merge of #896 @guizmaii (#897)
• Update README.md @zio-assistant (#896)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.10 @github-actions (#895)
• Add new ConsumerSettings::enableOptimisticResume setting (true by default) to allow users to disable the "optimistic resume" optimisation && Write benchmarks where this optimisation is disabled @guizmaii (#889)
• Update sbt to 1.9.0 @github-actions (#891)
• Update zio-streams, zio-test-sbt to 2.0.14 @scala-steward (#888)
• Add Rocker as adopter @loathwine (#885)
• Update zio, zio-streams, zio-test, ... to 2.0.14 @github-actions (#886)
• Update scalafmt-core to 3.7.4 @github-actions (#887)
• Update jackson-databind to 2.15.2 @github-actions (#882)
• Update README.md @zio-assistant (#878)
• Update README.md @zio-assistant (#877)
• Update README @guizmaii (#876)
• docs: add KelkooGroup as adopter @gaeljw (#875)
• Initiate an "Adopters" list @guizmaii (#872)
• [Internal] Clean some code @guizmaii (#871)
• [Internal] Simplify code: Remove useless doPoll method @guizmaii (#870)
• [Internal] Extract Command from the Runloop file and rename it to RunloopCommand @guizmaii (#867)
• [Internal] Improve function name @guizmaii (#869)
• [Internal] Reduce timeouts in tests from 5 to 2 minutes @guizmaii (#866)
• [Internal] Fix tests logging configuration @guizmaii (#865)
• [Internal] Use SLF4J/Logback to configure logging in tests @guizmaii (#858)
• [Internal] Simplify Diagnostics interface @guizmaii (#861)
• Update zio-kafka version in README && Update zio-kafka-example project @guizmaii (#855)

🌱 Dependency Updates

• Bump scala-steward-org/scala-steward-action from 2.52.0 to 2.54.0 @dependabot (#874)

v2.3.1

zio-kafka-testkit library

In this version, we renamed the zio-kafka-test-utils library to zio-kafka-testkit, we changed its base package from zio.kafka to zio.kafka.testkit, and we re-wrote the documentation on how to write tests when using zio-kafka and zio-kafka-testkit.

To migrate, you will have to:

1. Change in your build.sbt:

// Before
libraryDependencies += "dev.zio"  %% "zio-kafka-test-utils" % "2.3.0" % Test

// After
libraryDependencies += "dev.zio"  %% "zio-kafka-testkit" % "2.3.1" % Test

2. Fix your imports

// Before
import zio.kafka.ZIOSpecWithKafka
import zio.kafka.KafkaTestUtils
import zio.kafka.KafkaRandom
import zio.kafka.embedded.Kafka

// After
import zio.kafka.testkit.ZIOSpecWithKafka
import zio.kafka.testkit.KafkaTestUtils
import zio.kafka.testkit.KafkaRandom
import zio.kafka.testkit.Kafka

You can find the new documentation here: https://zio.dev/zio-kafka/writing-tests

Other changes

• We un-deprecated the ZIOSpecWithKafka trait, and we added some documentation about its purpose and usage in the documentation.
  See https://zio.dev/zio-kafka/writing-tests#ziospecwithkafka-trait

All Changes

• Write zio-kafka-testkit documentation @guizmaii (#846)
• Rename the zio-kafka-test-utils module to zio-kafka-testkit && change its base package from zio.kafka to zio.kafka.testkit @guizmaii (#845)
• Update zio-logging-slf4j2 to 2.1.13 @github-actions (#854)
• Un-deprecate ZIOSpecWithKafka && Add doc explaining when to use it @guizmaii (#851)
• Update jackson-databind to 2.15.1 @github-actions (#850)
• Add minimal "manual consumer" benchmarks @guizmaii (#849)
• Update sbt to 1.8.3 @github-actions (#847)
• Remove all rights reserved @erikvanoosten (#843)
• Update zio-kafka to 2.3.0 @github-actions (#842)
• Update README.md @zio-assistant (#841)
• Bump version, remove all rights reserved @erikvanoosten (#840)

v2.3.0

Consumer’s performances

This release brings many performance improvements to zio-kafka’s Consumer, getting closer to the performances of the native Java kafka-clients’ Consumer.

Because of these changes, the broker will be polled more often which can cause higher CPU and memory consumption. You may need to re-tune the consumer settings. For example, you can lower max.poll.records when there is high memory pressure, and you can increase pollTimeout when you observe high CPU usage and latency is not a concern.

Our benchmarks are showing that with this new version of zio-kafka, to consume a million messages, zio-kafka’s Consumer now takes ~737.614 ± 19.178 ms while the native Java kafka-clients one takes ~640.530 ± 1.855 ms

For comparison, the Consumer of zio-kafka v2.2 was taking ~1950.181 ± 590.420 ms with very high variability in the benchmark results:

// v2.2
Result "zio.kafka.bench.ConsumersComparisonBenchmark.zioKafka":
1950.181 ±(99.9%) 590.420 ms/op [Average]
(min, avg, max) = (1015.872, 1950.181, 4967.793), stdev = 1192.678
CI (99.9%): [1359.761, 2540.602] (assumes normal distribution)

while now, the results are much more stable:

// v2.3.0
Result "zio.kafka.bench.ConsumersComparisonBenchmark.zioKafka":
737.614 ±(99.9%) 19.178 ms/op [Average]
(min, avg, max) = (677.381, 737.614, 820.314), stdev = 38.740
CI (99.9%): [718.437, 756.792] (assumes normal distribution)

See details of the current performance results here: #809 (comment)

Note that these benchmarks were run on my local machine, a MacBook Pro 2019 Intel 2.4Ghz 8 cores 64GB 2667 MHz DDR4.
To reproduce these results, you can run this sbt command on any version of zio-kafka containing these benchmarks:

sbt "clean;Test/compile;zioKafkaBench/Jmh/run -wi 10 -i 10 -r 1 -w 1 -t 1 -f 5 -foe true .*ConsumersComparisonBenchmark*"

If you monitor your Consumers' performances in production, don't hesitate to come to the zio-kafka channel of the ZIO Discord to share the changes you could have observed with this new version! ❤️

Breaking changes

• The AdminClient::listConsumerGroupOffsets(groupSpecs: Map[String, ListConsumerGroupOffsetsSpec]) and AdminClient::listConsumerGroupOffsets(groupSpecs: Map[String, ListConsumerGroupOffsetsSpec], options: ListConsumerGroupOffsetsOptions) methods were returning a Task[Map[TopicPartition, OffsetAndMetadata]] and are now returning a Task[Map[String, Map[TopicPartition, OffsetAndMetadata]]]

The String in the key of the Map being the Consumer Group IDs (same ones as the ones you passed in parameters)

• The AdminClient.fromManagedJavaClient has been renamed AdminClient.fromScopedJavaClient

• The ConsumerSettings::perPartitionChunkPrefetch Consumer configuration has been removed

Deprecations

• The ZIOSpecWithKafka trait in zio-kafka-test-utils is deprecated

See https://github.com/zio/zio-kafka/blob/master/zio-kafka-test-utils/src/main/scala/zio/kafka/ZIOSpecWithKafka.scala#L7-L23 to see how to remove it from your code.

Other noticeable changes

• When instantiating a client (AdminClient, Producer or Consumer) zio-kafka is now checking that your client SSL configuration is valid for your server SSL configuration, failing if the client configuration is incorrect.

This check prevents your app from crashing with an OOM a few minutes after you start the client.
This check protects you against a long-standing bug present in kafka-clients, see https://issues.apache.org/jira/browse/KAFKA-4090

This check is provided to you by Conduktor, backported from their internal fork of zio-kafka.

For more details about this check, you can have a look at its code here: https://github.com/zio/zio-kafka/blob/master/zio-kafka/src/main/scala/zio/kafka/utils/SslHelper.scala

All changes

• Optimistically resume partitions early @erikvanoosten (#809)
• Performances: Remove Scala-- code @guizmaii (#834)
• Update scala-collection-compat to 2.10.0 @github-actions (#833)
• Backport Conduktor's improvements @guizmaii (#781)
• Update jackson-databind to 2.15.0 @scala-steward (#831)
• Small improvements taken from #788 @erikvanoosten (#829)
• Update logback-classic to 1.3.7 @scala-steward (#827)
• Fix Scala3 build @guizmaii (#824)
• Update embedded-kafka to 3.4.0.1 @github-actions (#823)
• Update zio-sbt-ci and CI definition @guizmaii (#822)
• Add a minimal example app, useful to tests things manually if necessary @guizmaii (#807)
• Fix CI @guizmaii (#820)
• Update README.md @guizmaii (#819)
• Update zio, zio-streams, zio-test, ... to 2.0.13 @github-actions (#818)
• New code owners part 2 @erikvanoosten (#812)
• Use finalizer to ensure seeing de-register in unit test @erikvanoosten (#813)
• Remove unnecessary nowarn @erikvanoosten (#810)
• New code owners @erikvanoosten (#811)
• ConsumerAccess::runloopAccess should be private @guizmaii (#808)
• Backport improvement made in zio/zio#7996 @guizmaii (#805)
• Run the Runloop on a dedicated single-threaded thread pool @guizmaii (#802)
• Optimize offerRecordsToStreams code @guizmaii (#791)
• Update zio, zio-streams, zio-test, ... to 2.0.12 @github-actions (#793)
• Fix CI @guizmaii (#792)
• Update ZIO @guizmaii (#787)
• Simplify CI definition @guizmaii (#786)
• Improve stream restart in restartStreamsOnRebalancing mode @erikvanoosten (#785)
• Update README.md @github-actions (#782)
• Simpler shutdown handling @svroonland (#778)

🌱 Dependency Updates

• Bump peter-evans/create-pull-request from 5.0.0 to 5.0.1 @dependabot (#837)
• Bump benchmark-action/github-action-benchmark from 1.16.2 to 1.17.0 @dependabot (#835)
• Bump scala-steward-org/scala-steward-action from 2.51.0 to 2.52.0 @dependabot (#814)
• Bump peter-evans/create-pull-request from 4.2.3 to 5.0.0 @dependabot (#816)
• Bump peter-evans/create-pull-request from 4.2.3 to 5.0.0 @dependabot (#789)
• Bump actions/setup-java from 3.10.0 to 3.11.0 @dependabot (#790)
• Bump actions/setup-java from 3.10.0 to 3.11.0 @dependabot (#779)

v2.2

This release contains many performance enhancements and internal code optimizations to the consumer side of zio-kafka. The Runloop, the heart of the Consumer, has been optimized to reduce the polling and streaming latency and be able to provide higher throughput, as well as less GC pressure. This results in a throughput that is 2 to 5 times higher.

The Runloop polls the underlying Kafka client for new data. It also applies backpressure for streams that do not require data by pausing partitions. When the stream needs more data, the partition is resumed. Previously the Runloop would poll every pollInterval. Now it polls continuously. When all streams stop processing, data polling stops so that the broker can detect that this Kafka client is stalled.

Several other changes to be aware of:

• OffsetBatch.merge was deprecated in favour of add.
• Consumer setting pollInterval was removed because polling is now continuous. We recommend you use ConsumerSettings(bootstrapServers) and the with* methods to build consumer settings to prevent future source incompatibilities.
• Consumer setting runloopTimeout was added. When the runloop has nothing to do for this period, it will fail.

Changes

• Increase default runloop timeout @erikvanoosten (#775)
• Give ConsumerSettings.runloopTimeout a default @erikvanoosten (#774)
• Fix restartStreamsOnRebalancing @erikvanoosten (#772)
• Improve readability @erikvanoosten (#773)
• Update scalafmt-core to 3.7.3 @github-actions (#769)
• Add better-monadic-for scalac plugin @guizmaii (#765)
• Improve readability @guizmaii (#763)
• Remove unused offsetRetrieval field from ChangeSubscription command @guizmaii (#764)
• Fix getConsumerGroupMetadataIfAny: It shouldn't access the low-level consumer directly @guizmaii (#762)
• Optimize offerRecordsToStreams code @guizmaii (#761)
• Optimise handlePoll code: Ensure that both runningStreams and newAssignedStreams are Chunks to take advantage of the optimised version of ++ when computing the updatedStreams @guizmaii (#760)
• Divide allocations: Instantiate the OffsetImpl as late as possible to avoid allocatiing one OffsetImpl and one TopicPartition for each consumed Record @guizmaii (#759)
• add test case for the runloop polling behavior @tnielens (#684)
• Optimise offerRecordsToStreams code @guizmaii (#756)
• Tune benchmark parameters @svroonland (#729)
• Review of PR #727: Remove unused Request::dataQueue @guizmaii (#747)
• Improve performances: Remove the aggregateAsync call @guizmaii (#751)
• Improve Logback configuration in benchmarks @guizmaii (#750)
• Review of PR #727 @guizmaii (#745)
• Fix ConsumersComparisonBenchmark: Avoid booting the Consumer instance in the bench, which could add noise in the bench results @guizmaii (#748)
• Remove buffered records @erikvanoosten (#744)
• ZStream based implementation of poll loop @svroonland (#736)
• Feed streams through queue instead of promise @erikvanoosten (#727)
• First seek then use for manual assignments @erikvanoosten (#726)
• Stabilize benchmark @svroonland (#714)
• Fix CI @guizmaii (#723)
• Optimise Runloop::run code @guizmaii (#721)
• Add profiling in the CI @guizmaii (#696)
• Init comparison benchmarks: compare kafka-clients vs zio-kafka consumers @guizmaii (#695)
• Rollbacks @guizmaii (#722)
• Update logback-classic to 1.4.6 @github-actions (#717)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.3.10+94-cbe49d51-SNAPSHOT @github-actions (#718)
• Make as much computation as possible in fulfillRequests code @guizmaii (#720)
• Rename OffsetBatch::merge(Offset) to OffsetBatch::add(Offset) to improve readability @guizmaii (#700)
• Apply ZIO SBT Ecosystem and CI Plugins @khajavi (#702)
• Improve polling behavior @svroonland (#683)
• Update logback-classic to 1.3.6 @github-actions (#709)
• Update README.md @github-actions (#693)
• Consumer benchmark @svroonland (#689)

🌱 Dependency Updates

• Bump hmarr/auto-approve-action from 3.2.0 to 3.2.1 @dependabot (#734)
• Bump hmarr/auto-approve-action from 3.1.0 to 3.2.0 @dependabot (#705)
• Bump benchmark-action/github-action-benchmark from 1.16.0 to 1.16.2 @dependabot (#704)