v0.3.0

"Lux et veritas"

Changes since v0.3.0-beta3

Features

• Add Pub/Sub attribute access #535 #538
• Add idLabel and timestampLabel to saveAsPubsub #547 #561
• Improve skewJoin API and error upper bound computation #566 #567
• Add sparseFullOuterJoin #545 #548
• Include TensorFlow Example protobuf #536 #537
• Add shapeless datastore examples #521
• Scala 2.11.11 and SBT 0.13.15 #533 #557
• Use HDFS IO from Beam #516
• Add pretty name to anonymous functions #531 #532
• Make sure run is invoked on JobTest #515
• Make sure close is called in JobTest #513
• Improve test wiring error messages #539 #541
• Better error messages in case of BQ failure #569

Bug fixes

• Escape Scala reserved words #522 #523
• Make JobTest thread-safe #524
• Make Tap serializable #544 #560
• Fix serialization of LocalDateTime and LocalDate #568
• Human friendly job/app names for REPL jobs #550 #552
• Use GCP temp location or throw #553 #554
• Make ScioResult compatible with DataflowTemplateJob #527

List of breaking changes:

• https://github.com/spotify/scio/wiki/Apache-Beam

Known issues:

• Scala 2.12 build is not available pending one remaining compiler bug scala/bug#10233
• NPE in DirectRunner (including local scio-repl) in streaming mode BEAM-1835

v0.3.0-beta3

Changes since v0.3.0-beta2

• Fix BigtableIO #497
• Support BigQuery table description #491
• JIterable serialization improvement #507
• Bump Avro to 1.8.1 #510
• Package object clean up #508
• Add logging for BigtableUtil#updateNumberOfBigtableNodes #498

List of breaking changes:

• https://github.com/spotify/scio/wiki/Apache-Beam

Known issues:

• Scala 2.12 build is not available pending some compiler bugs (SI-10232, SI-10233)
• NPE in DirectRunner (including local scio-repl) in streaming mode BEAM-1835

v0.3.0-beta2

Changes since v0.3.0-beta1

• Features and bug fixes from v0.2.13
• Bump Beam to 0.6.0 (release notes)
• Support Array[Byte] as Sparkey key-value. (#483)
• Fix Sparkey upload performance issue
• Fix missing coder issue in SCollection#applyTransform

List of breaking changes:

• https://github.com/spotify/scio/wiki/Apache-Beam

Known issues:

• Scala 2.12 build is not available yet but almost there

v0.2.13

"Ex luna scientia"

Notable changes:

• Fix groupBy performance issue with large groups
• Improve Avro serialization performance
• Fix closure serialization in SCollection#partition
• Fix a Taps bug when directory doesn't exist

v0.3.0-beta1

Changes since v0.3.0-alpha2

• Features and bug fixes from v0.2.12
• Graduate type safe BigQuery API from experimental #336
• Consolidate temp location handling, --temp is the recommended flag now
• Add Sparkey side input support #402
• Add convenience implicits to scio-bigtable
• Support Coder[T] in PubsubIO
• Support Algebird aggregator in fan out cases
• Many streaming API tweaks
• Add ScioResult#waitUntilDone method
• Bugfix: lots of documentation updates

List of breaking changes:

• https://github.com/spotify/scio/wiki/Apache-Beam

Known issues:

• Scala 2.12 build is not available yet but almost there

v0.2.12

"In extremo"

Notable changes:

• Side input performance fixes #453
• Make Args serializable and add toString
• Fix #466 MultiJoin transform names
• Fix #449. Warn on scala.App
• Add setup/teardown to JobTest to be more extensible

v0.3.0-alpha2

Changes since v0.3.0-alpha1

• Bump Beam to 0.5.0 (release notes)
• Features and bug fixes from v0.2.11
• Rolled back from Java 8 java.time to joda-time
• Use --gcpTempLocation instead of --stagingLocation when running on Dataflow service

List of breaking changes:

• https://github.com/spotify/scio/wiki/Apache-Beam

Known issues:

• Scala 2.12 build is not available yet
• You might need "junit" % "junit" % "4.12" in your project dependencies

v0.2.11

"Saltatio mortis"

Notable changes:

• Feature: add ProtoIO to TestDataManager
• Feature: allow to switch on/off BigQuery cache (#427)
• Feature: allow to explicitly pick legacy/standard syntax inside BigQuery query (#427) (doc)
• Feature: bump Algebird to version 0.12.4 (0.12.4 release notes)
• Bugfix: allow to change job name via scio context (#440)
• Bugfix: fix compilation on encrypted linux partitions (#439)
• Bugfix: support compressed text files in TextTap
• Bugfix: improve kantan.csv integration (#426)

v0.3.0-alpha1

List of breaking changes:

• https://github.com/spotify/scio/wiki/Apache-Beam

Known issues:

• BigQuery writes may not work BEAM-1258
• Scala 2.12 build is not available yet

v0.2.10

"De Mysteriis Dom Sathanas"

Notable changes:

• Support for named transforms (#357), see this FAQ for more
• Expose ScioContext#isTest publicly
• Fix TFRecordFile handling
• Fix ProcessContext reuse for side input/output
• Fix #394. Add doc for hotKeyThreshold scaling (#404)
• Fix #400 Start repl from sbt (#403)
• Add integration test for DistCache (#398)