v2.9.0

This release brings important fixes in the rebalancing mechanism, as well as other minor improvements.

A couple of methods are now deprecated:

• Serde#inmapM, renamed to inmapZIO
• Deserializer#contramapM, renamed to contramapZIO
• Serializer#mapM, renamed to mapZIO
• Consumer#fromJavaConsumer(javaConsumer), replaced by fromJavaConsumerWithPermit

If you are using scala-steward, the renamed methods are automatically changed.

🚀 Changes

• Add example dashboard to docs by @erikvanoosten in #1357
• Fix slow rebalanceSafeCommits behavior by @svroonland in #1358
• Remove attemptBlocking from AdminClient by @svroonland in #1359
• Fix concurrent modification exception in Consumer release by @svroonland in #1365
• Fix stream interrupt when no data, Add tests for PartitionStreamControl by @svroonland in #1371
• More logging around rebalancing when rebalanceSafeCommits is true by @svroonland in #1360
• Require a semaphore in Consumer.fromJavaConsumer by @svroonland in #1368
• Rename Serde methods for consistency with ZIO 2 by @svroonland in #1378
• Fix deprecation notice version on Consumer.fromJavaConsumer by @svroonland in #1379
• Decouple stream halt detection timeout from max poll interval by @svroonland in #1376

🌱 Dependency Updates

• Update kafka-clients and embedded-kafka to 3.8.1 by @zio-scala-steward in #1352
• Update zio-kafka, zio-kafka-testkit to 2.8.3 by @zio-scala-steward in #1353
• Update zio, zio-streams, zio-test, ... to 2.1.12 by @zio-scala-steward in #1364

🔧 Maintenance

• Update README.md by @zio-assistant in #1355
• Update sbt to 1.10.5 by @zio-scala-steward in #1362
• Automate pruning of benchmark history by @erikvanoosten in #1370
• Fix automatic pruning of benchmark history by @erikvanoosten in #1372
• Fix automatic pruning of benchmark history (attempt 2) by @erikvanoosten in #1373
• Bump scala-steward-org/scala-steward-action from 2.70.0 to 2.71.0 by @dependabot in #1377
• Use max.poll.records i.s.o. config name by @erikvanoosten in #1381
• Formatting by @svroonland in #1383
• Disable mima check for new minor releases by @erikvanoosten in #1384

Full Changelog: v2.8.3...v2.9.0

v2.8.2

This release reverts the producer changes of v2.8.1 because they has a huge impact on performance. We will reconsider the changes and make sure the performance tests accurately cover producing events before applying this change again.

What's Changed

• Revert back to 2.8.0 sendFromQueue implementation in producer by @ytalashko in #1304

Full Changelog: v2.8.1...v2.8.2

v2.8.1

Version 2.8.1 has an important bugfix that prevents partitions from being consumed twice in some circumstances (#1294). Many thanks to @ytalashko for finding the root cause, providing a fix, and tirelessly discussing the problem until it was fully understood and the fix applied.

🚀 Noteworthy changes

• Fix assign followed by fast revoke during rebalance @ytalashko (#1294)
• Add alternative fetch strategy for many partitions @erikvanoosten (#1281)
• Document metrics and consumer tuning based on metrics @erikvanoosten (#1280)

🚅 More changes

• Prevent users from enabling auto commit @erikvanoosten (#1290)
• Alternative producer implementation @erikvanoosten (#1285)
• Small improvements to the Producer @erikvanoosten (#1272)
• Improve example app @erikvanoosten (#1279)

🌱 Maintenance

• Update sbt-tpolecat to 0.5.2 @zio-scala-steward (#1301)
• Update embedded-kafka to 3.7.1.1 @zio-scala-steward (#1300)
• Update logback-classic to 1.5.7 @zio-scala-steward (#1299)
• Update zio-streams, zio-test-sbt to 2.1.7 @zio-scala-steward (#1297)
• Update zio, zio-streams, zio-test, ... to 2.1.7 @zio-scala-steward (#1295)
• Update scalafmt-core to 3.8.3 @zio-scala-steward (#1291)
• Update README.md @zio-assistant (#1278)
• Update README.md @zio-assistant (#1277)
• Update zio-kafka, zio-kafka-testkit to 2.8.0 @zio-scala-steward (#1274)
• Update zio-streams, zio-test-sbt to 2.1.6 @zio-scala-steward (#1275)

v2.8.0

Version 2.7.5 was not fully compatible with 2.7.4 which we missed during the release. Therefore, with some delay, we bump the minor version.

Moving back to pre-2.7.0 behavior: in this release, upon a lost partition the stream ends without error. To detect lost partitions, you can configure a rebalance listener.

🚀 Changes

• Allow for some auth exceptions during poll @erikvanoosten (#1270)
• A lost partition is no longer fatal @erikvanoosten (#1252)
• Update consumer docs @petern-sc (#1269)

🌱 Maintenance

• Update zio, zio-streams, zio-test, ... to 2.1.6 @zio-scala-steward (#1273)
• Update sbt to 1.10.1 @zio-scala-steward (#1271)
• Update kafka-clients to 3.7.1 @zio-scala-steward (#1268)
• Update zio-streams, zio-test-sbt to 2.1.5 @zio-scala-steward (#1267)
• Update zio, zio-streams, zio-test, ... to 2.1.5 @zio-scala-steward (#1266)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.28 @zio-scala-steward (#1263)
• Update zio-streams, zio-test-sbt to 2.1.4 @zio-scala-steward (#1261)
• Update zio, zio-streams, zio-test, ... to 2.1.4 @zio-scala-steward (#1260)
• Update scalafmt-core to 3.8.2 @zio-scala-steward (#1259)
• Update zio-streams, zio-test-sbt to 2.1.3 @zio-scala-steward (#1257)
• Update zio, zio-streams, zio-test, ... to 2.1.3 @zio-scala-steward (#1256)
• Aborts builds when a newer build is ready @erikvanoosten (#1255)
• Update README.md @zio-assistant (#1254)
• Update zio-kafka, zio-kafka-testkit to 2.7.5 @zio-scala-steward (#1253)

v2.7.5

Minor breaking changes

• breaking: Do not run user rebalance listener on same thread runtime @svroonland (#1205)
  In case you register a rebalance listener, it now runs on the default zio runtime and no longer on the special same-thread-runtime. Access to the underlying java client is removed. If this affects you, please let us know your use case.
• Remove no longer needed collection-compat @erikvanoosten (#1221)
  The library collection-compat is no longer a dependency. If you are using it, please add it to your own dependencies.

🚀 Changes

• Allow stream to be interrupted when there is no traffic @erikvanoosten (#1251)
  Lost partitions during no traffic situations are now correctly detected. That is, the stream for the partition that is lost is interrupted, the other streams are closed gracefully, the consumer aborts with an error. Note that in the next release we will change this behavior, then the stream for the partition that is lost is still ended, but the other streams will be unaffected.
• Initialize Java Producer in blocking threadpool @kyri-petrou (#1230)
• Call removeMembersFromConsumerGroup without members @sderosiaux (#1227)
• Document snapshots and Update resources @erikvanoosten (#1223)
• Drop experimental label for rebalanceSafeCommits @erikvanoosten (#1222)
• Correct prune benchmark history instructions @erikvanoosten (#1217)
• Add MIMA checks to ensure binary compatibility @erikvanoosten (#1214)

🌱 Maintenance

• Update zio-streams, zio-test-sbt to 2.1.2 @zio-scala-steward (#1249)
• Update zio, zio-streams, zio-test, ... to 2.1.2 @zio-scala-steward (#1248)
• Update zio-logging-slf4j, ... to 2.3.0 @zio-scala-steward (#1246)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.27 @zio-scala-steward (#1247)
• Update zio-logging-slf4j, ... to 2.2.4 @zio-scala-steward (#1244)
• Update zio-streams, zio-test-sbt to 2.1.1 @zio-scala-steward (#1243)
• Update zio, zio-streams, zio-test, ... to 2.1.1 @zio-scala-steward (#1242)
• Update zio-streams, zio-test-sbt to 2.1.0 @zio-scala-steward (#1241)
• Update zio, zio-streams, zio-test, ... to 2.1.0 @zio-scala-steward (#1240)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.26 @zio-scala-steward (#1237)
• Update sbt-scalafix to 0.12.1 @zio-scala-steward (#1234)
• Update sbt to 1.10.0 @zio-scala-steward (#1236)
• Update scala-library to 2.13.14 @zio-scala-steward (#1235)
• Update zio-logging-slf4j, ... to 2.2.3 @zio-scala-steward (#1226)
• Regenerate readme @erikvanoosten (#1225)
• Update logback-classic to 1.5.6 @zio-scala-steward (#1224)
• Update scala-collection-compat to 2.12.0 @zio-scala-steward (#1220)
• Revert "Bump coursier/cache-action from 6.4.5 to 6.4.6" @erikvanoosten (#1216)
• Update zio-streams, zio-test-sbt to 2.0.22 @zio-scala-steward (#1215)
• Update logback-classic to 1.5.5 @zio-scala-steward (#1211)
• Update sbt-native-packager to 1.10.0 @zio-scala-steward (#1212)
• Update zio, zio-streams, zio-test, ... to 2.0.22 @zio-scala-steward (#1213)
• Update logback-classic to 1.5.4 @zio-scala-steward (#1210)
• Update sbt-tpolecat to 0.5.1 @zio-scala-steward (#1206)
• Update scalafmt-core to 3.8.1 @zio-scala-steward (#1204)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.25 @zio-scala-steward (#1200)
• Update zio-kafka, zio-kafka-testkit to 2.7.4 @zio-scala-steward (#1197)
• Update README.md @zio-assistant (#1195)
• Bump benchmark-action/github-action-benchmark from 1.20.1 to 1.20.3 @dependabot (#1245)
• Bump coursier/cache-action from 6.4.5 to 6.4.6 @dependabot (#1209)
• Bump benchmark-action/github-action-benchmark from 1.18.0 to 1.20.1 @dependabot (#1208)

v2.7.4

Noteworthy Changes

• Update kafka-clients to 3.7.0 @zio-scala-steward (#1191)

Changes

• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.24 @zio-scala-steward (#1194)
• Update kafka-clients to 3.7.0 @zio-scala-steward (#1191)
• Update scala3-library to 3.3.3 @zio-scala-steward (#1181)
• Update sbt-scalafix to 0.12.0 @zio-scala-steward (#1184)
• Update logback-classic to 1.5.3 @zio-scala-steward (#1190)
• Update logback-classic to 1.5.2 @zio-scala-steward (#1185)
• Update logback-classic to 1.5.1 @zio-scala-steward (#1178)
• Fix a merge error and regenerate actions @erikvanoosten (#1177)
• Update README.md @zio-assistant (#1176)
• Update README.md @zio-assistant (#1175)
• Update README.md @zio-assistant (#1174)
• Update zio-streams, zio-test-sbt to 2.0.21 @zio-scala-steward (#1167)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.23 @zio-scala-steward (#1166)
• Update README.md @zio-assistant (#1173)
• Update scalafmt-core to 3.8.0 @zio-scala-steward (#1171)
• Update README.md @zio-assistant (#1172)
• Update sbt to 1.9.9 @zio-scala-steward (#1170)
• Update scala-library to 2.13.13 @zio-scala-steward (#1169)
• Update zio-logging-slf4j, ... to 2.2.2 @zio-scala-steward (#1165)
• Update logback-classic to 1.5.0 @zio-scala-steward (#1163)
• Update zio-kafka, zio-kafka-testkit to 2.7.3 @zio-scala-steward (#1164)

🌱 Dependency Updates

• Bump scala-steward-org/scala-steward-action from 2.64.0 to 2.65.0 @dependabot (#1192)
• Bump scala-steward-org/scala-steward-action from 2.63.0 to 2.64.0 @dependabot (#1186)
• Bump release-drafter/release-drafter from 5 to 6 @dependabot (#1189)
• Bump scala-steward-org/scala-steward-action from 2.61.0 to 2.63.0 @dependabot (#1133)

v2.7.3

Noteworthy Changes

• Collect consumer metrics @erikvanoosten (#1143)

Zio-kafka now passes a lot of metrics to zio. Use any zio metrics backend to expose them! See the issue #1143 for more information on which metrics are collected.

Changes

• Update zio, zio-streams, zio-test, ... to 2.0.21 @zio-scala-steward (#1144)
• Add scala steward badge (2nd attempt) @erikvanoosten (#1157)
• Update README.md @zio-assistant (#1155)
• Ensure that fibers forked in Runloop.make are interruptible @svroonland (#1154)
• Add scala steward badge @erikvanoosten (#1145)
• Collect consumer metrics @erikvanoosten (#1143)
• Update zio-logging-slf4j, ... to 2.2.0 @zio-scala-steward (#1152)
• Update zio-logging-slf4j, ... to 2.1.17 @zio-scala-steward (#1151)
• Update README.md @zio-assistant (#1150)
• Update zio-kafka, zio-kafka-testkit to 2.7.2 @zio-scala-steward (#1149)

v2.7.2

Note worthy changes

• Add partitionsFor method to producer @kyri-petrou (#1147)
• retry Consumer.CommitTimeout exception @myazinn (#1140)

Other changes

• Use Semaphore @adamgfraser (#1142)
• Make tests faster @erikvanoosten (#1139)
• Delete RunloopTimeout and small improvements @erikvanoosten (#1141)
• Update kafka-clients to 3.6.1 @zio-scala-steward (#1131)
• Improve support and documentation for manual offsets @erikvanoosten (#1135)
• Update sbt to 1.9.8 @zio-scala-steward (#1138)
• Update zio-concurrent, zio-streams, ... to 2.0.20 @zio-scala-steward (#1137)
• Update zio, zio-concurrent, zio-streams, ... to 2.0.20 @zio-scala-steward (#1136)
• Update logback-classic to 1.4.14 @zio-scala-steward (#1129)
• Update logback-classic to 1.3.14 @zio-scala-steward (#1128)
• Update zio-kafka, zio-kafka-testkit to 2.7.1 @zio-scala-steward (#1127)
• Remove unused config for Mergify @erikvanoosten (#1124)
• Adds test for stopping consumption while doing async commits @erikvanoosten (#1123)
• Update README.md @zio-assistant (#1125)
• Update logback-classic to 1.4.13 @zio-scala-steward (#1122)

v2.7.1

Introducing Rebalance Safe Commits

This release introduces a new experimental mode that prevent duplicate message processing during a rebalance. This mode has been in the making for 8 months and has been implemented over 9 different PRs.

We have marked Rebalance Safe Commits as an experimental feature because zio-kafka is used in many different and sometimes surprising ways. Some of those are not compatible with the new mode.

RebalanceSafeCommits can be enabled via ConsumerSettings.withRebalanceSafeCommits. Please read the scaladocs to see if this mode is applicable to how you use zio-kafka.

Much more information on the new mode (motivation, why it works, the challenges, etc.), are available in the descriptions of PR Await commits during a rebalance. A manual is forthcoming.

Changes

• Make maxRebalanceDuration configurable @erikvanoosten (#1118)
• Update zio-logging-slf4j, ... to 2.1.16 @zio-scala-steward (#1114)
• Automatically add new flame graphs to gh_pages, prune old @erikvanoosten (#1105)
• Await commit during a rebalance @erikvanoosten (#1098)
• Update README.md @zio-assistant (#1113)
• Update zio-kafka, zio-kafka-testkit to 2.7.0 @zio-scala-steward (#1112)

v2.7.0

Improvements

Zio-kafka became a little bit faster by doing less work when there is no rebalance.

Really large and tiny messages are better described in the tuning documentation, and ConsumerSettings.withoutPartitionPreFetching was introduced to make it easier to disable pre-fetching.

Previously, in some use cases, you could end up with a live-lock trying to get access to the underlying java consumer. This should no longer happen.

Breaking changes

Zio-kafka no longer supports scala 2.12. Scala 2.13 has been available for 4½ year so everyone has had plenty of time to move to a more modern scala version. (#1104)

This release has some some minor breaking changes:

• The diagnostics events for rebalances have been improved in an incompatible way (#1102)
• Jackson was removed as a dependency (#1096)
• partitionPreFetchBufferLimit is now 0-based instead of 1-based (#1091)

Changes

• Update scalafmt-core to 3.7.17 @zio-scala-steward (#1111)
• Make access to the java consumer fair @flavienbert (#1109)
• Extend tuning docs @erikvanoosten (#1110)
• Protect agains user diagnostics, better rebalance events @erikvanoosten (#1102)
• Drop support for scala 2.12 @erikvanoosten (#1104)
• Update zio-streams, zio-test-sbt to 2.0.19 @zio-scala-steward (#1101)
• Update zio, zio-streams, zio-test, ... to 2.0.19 @zio-scala-steward (#1100)
• Update zio-logging-slf4j, ... to 2.1.15 @zio-scala-steward (#1099)
• Track latest completed commit offset per partition @erikvanoosten (#1097)
• Always end streams in rebalance listener, support lost partitions @erikvanoosten (#1089)
• Remove jackson dependency @erikvanoosten (#1096)
• Make partitionPreFetchBufferLimit value 0 disable prefetching @erikvanoosten (#1091)
• Update zio-kafka, zio-kafka-testkit to 2.6.0 @zio-scala-steward (#1092)
• Update README.md @zio-assistant (#1090)
• Prevent double logging in tests @erikvanoosten (#1088)