v0.12.0

This release builds on v0.12.0-RC1 and v0.12.0-RC2 and uses Beam 2.41.0.

See our Migration Guide for info on breaking changes.

🚀 Enhancements

v0.12.0

• Concurrently support legacy Parquet reads for 0.12.x (#4555, #4578) @clairemcginty
• Provide java.util.UUID Coder (#4546) @RustedBones
• setup scalafix (#4504) @farzad-sedghi

v0.12.0-RC2

• Update version util w/ migration guide, add developer info by @kellen in #4450
• Add dependency graph workflow by @kellen in #4467
• Migrate to magnolia1 by @RustedBones in #4422
• make it more clear how to run mdoc by @farzad-sedghi in #4487
• optional project and dataset for mock bq by @farzad-sedghi in #4474
• Cross compile IT tests by @RustedBones in #4491
• Improve KV batch API by @RustedBones in #4458
• support 1-to-n mapping function for transformOverride by @farzad-sedghi in #4484
• Update beam-runners-core-construction-java, ... to 2.41.0 by @scala-steward in #4506
• BatchDoFn and sio batch API on SCollection by @RustedBones in #4489
• Enforce javafmt and headers by @RustedBones in #4520
• Do not reference scio coder after materialization by @RustedBones in #4509
• Changed processElement to newer annotated style in scio-avro, scio-core by @shnapz in #4511
• Support Neo4jIO by @turb in #4488
• Implement gRPC lookup API by @RustedBones in #4492
• IO Windowing by @kellen in #4461
• Final refactoring of @processElement to bring new style by @shnapz in #4529
• Migrate from sbt-protobuf to sbt-protoc by @RustedBones in #4483

v0.12.0-RC1

• SMB secondary sort key by @kellen in #4288
• Remove unused jna dependency by @RustedBones in #4432
• Cherry-pick Parquet SplittableDoFn implementation. by @clairemcginty in #4436
• Add granularity option for Parquet-SDF reads by @clairemcginty in #4438
• Reduce coder memory footprint by @RustedBones in #4397
• Reduce warnings by @RustedBones in #4381
• Update beam to 2.40 by @RustedBones in #4433
• Remove deprecated APIs by @kellen in #4446
• Add 0.12.0 version information, deprecations, pubsub IO information by @kellen in #4447
• IT + FAQ cleanup by @kellen in #4448

🐛 Bug Fixes

v0.12.0

• Fix NullPointerException with extendedDebugInfo=true (#4550) @RustedBones
• Fix record coder structural value equality (#4547) @RustedBones
• Detect and retrieve GCP temp location (fixes #4535) (#4536) @turb
• Fix example page (#4534) @RustedBones
• Fix NullPointerException in DataflowResult (#4538) @bergman

v0.12.0-RC2

• Add newline for list by @kellen in #4460
• Fix smb secondary key test by @kellen in #4471
• update run instructions for examples by @farzad-sedghi in #4468
• Fix: Capturing ProcessContext in AsyncLookupDoFn leads to NPE in Runner V2 by @Illedran in #4495
• Propagate internal coder in tap() by @RustedBones in #4500
• Fix documentation generation by @RustedBones in #4517
• Fix example page using showdown by @RustedBones in #4533
• Fix google-api-client evicted by mysql-socket-factory 1.7.0 by @RustedBones in #4531

v0.12.0-RC1

• (bugfix) typed-Parquet sink wasn't using user-supplied Configuration by @clairemcginty in #4434
• Patch startRegex for es 8.3+ by @RustedBones in #4444
• Remove org.apache documentation exclusion by @kellen in #4330
• staging: allow .sbt/boot by @danielnorberg in #4262

🌱 Dependency Updates

v0.12.0

• Update Magnolify to 0.6.2 (#4577) @clairemcginty
• Update sbt-bloop to 1.5.4 (#4551) @scala-steward
• Update elasticsearch 7 to 7.17.6 (#4554) @RustedBones
• Update elasticsearch-java to 8.4.3 (#4553) @scala-steward
• Update sbt-scalafix to 0.10.4 (#4552) @scala-steward
• Bump JamesIves/github-pages-deploy-action from 4.4.0 to 4.4.1 (#4539) @dependabot
• Update joda-time to 2.11.1 (#4508) @scala-steward

v0.12.0-RC2

• Update scala-collection-compat to 2.8.0 by @scala-steward in #4456
• Update sbt, sbt-dependency-tree to 1.7.1 by @scala-steward in #4457
• Update testcontainers-scala-elasticsearch, ... to 0.40.9 by @scala-steward in #4465
• Update mysql-connector-java to 8.0.30 by @scala-steward in #4473
• Update sbt-mdoc to 2.3.3 by @scala-steward in #4475
• Update scala-collection-compat to 2.8.1 by @scala-steward in #4479
• Update sbt-tpolecat to 0.4.1 by @scala-steward in #4472
• Update testcontainers-scala-elasticsearch, ... to 0.40.10 by @scala-steward in #4480
• Update sbt-bloop to 1.5.3 by @scala-steward in #4477
• Update scalatest to 3.2.13 by @scala-steward in #4482
• Update sparkey to 3.2.4 by @spotify-steward in #4408
• Update scalactic to 3.2.13 by @scala-steward in #4481
• Update sbt-avro to 3.4.2 by @scala-steward in #4485
• Update sbt-scoverage to 2.0.2 by @scala-steward in #4493
• Update scalafmt-core to 3.5.9 by @scala-steward in #4499
• Bump JamesIves/github-pages-deploy-action from 4.3.4 to 4.4.0 by @dependabot in #4469
• Update elasticsearch-java to 8.4.1 by @scala-steward in #4510
• Update sbt-protoc to 1.0.6 by @scala-steward in #4507
• Update kantan.csv to 0.7.0 by @scala-steward in #4516
• Update shapeless to 2.3.10 by @scala-steward in #4518
• Update sbt-scalafix to 0.10.2 by @scala-steward in #4521
• Update sbt-mima-plugin to 1.1.1 by @scala-steward in #4523
• Update scalacheck to 1.17.0 by @scala-steward in #4526
• Update scala-compiler, scala-library, ... to 2.12.17 by @scala-steward in #4525
• Sync jackson version with beam's by @RustedBones in #4530
• Update mysql-socket-factory to 1.7.0 by @scala-steward in #4519

v0.12.0-RC1

• Bump JamesIves/github-pages-deploy-action from 4.3.3 to 4.3.4 by @dependabot in #4431
• Update sbt-scoverage to 2.0.0 by @spotify-steward in #4430
• Update elasticsearch-java to 8.3.1 by @spotify-steward in #4441
• Update sbt-bloop to 1.5.2 by @spotify-steward in #4440
• Update to elasticsearch 7.17.5 by @RustedBones in #4443

Contributors to this release

@RustedBones, @bergman, @clairemcginty, @dependabot, @dependabot[bot], @farzad-sedghi, @scala-steward and @turb

v0.11.12

⚠️ This release is binary incompatible. If your project have other dependencies on scio 0.11, skip this release and update to 0.11.13

🐛 Bug Fixes

• Fix NullPointerException with extendedDebugInfo=true (#4550) @RustedBones

Contributors to this release

@RustedBones

v0.11.11

⚠️ This release is binary incompatible. If your project have other dependencies on scio 0.11, skip this release and update to 0.11.13

This release contains mostly bugfixes to unblock use of Scio with Dataflow Runner V2.

🚀 Enhancements

• Unblock DataflowRunner V2 by replacing parameterized AtomicCoder refs (#4399) @clairemcginty
• Update CustomCoders usages to implement verifyDeterministic (#4428) @clairemcginty
• Do not capture ProcessContext in AsyncLookup callbacks (#4495) @Illedran
• Changed processElement to newer annotated style in scio-avro, scio-core (#4511) @shnapz
• Reduce coder memory footprint (#4397) @RustedBones
• Do not reference scio coder after materialization (#4509) @RustedBones

🐛 Bug Fixes

• Fix record coder structural value equality (#4547) @RustedBones

🌱 Dependency Updates

• Migrate to magnolia1 (#4422) @RustedBones
• Update Magnolify to 0.6.1 (#4542) @clairemcginty

Contributors to this release

@RustedBones, @clairemcginty, @Illedran, @shnapz

v0.12.0-RC2

🚀 Enhancements

• Update version util w/ migration guide, add developer info by @kellen in #4450
• Add dependency graph workflow by @kellen in #4467
• Migrate to magnolia1 by @RustedBones in #4422
• make it more clear how to run mdoc by @farzad-sedghi in #4487
• optional project and dataset for mock bq by @farzad-sedghi in #4474
• Cross compile IT tests by @RustedBones in #4491
• Improve KV batch API by @RustedBones in #4458
• support 1-to-n mapping function for transformOverride by @farzad-sedghi in #4484
• Update beam-runners-core-construction-java, ... to 2.41.0 by @scala-steward in #4506
• BatchDoFn and sio batch API on SCollection by @RustedBones in #4489
• Enforce javafmt and headers by @RustedBones in #4520
• Do not reference scio coder after materialization by @RustedBones in #4509
• Changed processElement to newer annotated style in scio-avro, scio-core by @shnapz in #4511
• Support Neo4jIO by @turb in #4488
• Implement gRPC lookup API by @RustedBones in #4492
• IO Windowing by @kellen in #4461
• Final refactoring of @processElement to bring new style by @shnapz in #4529
• Migrate from sbt-protobuf to sbt-protoc by @RustedBones in #4483

🐛 Bug Fixes

• Add newline for list by @kellen in #4460
• Fix smb secondary key test by @kellen in #4471
• update run instructions for examples by @farzad-sedghi in #4468
• Fix: Capturing ProcessContext in AsyncLookupDoFn leads to NPE in Runner V2 by @Illedran in #4495
• Propagate internal coder in tap() by @RustedBones in #4500
• Fix documentation generation by @RustedBones in #4517
• Fix example page using showdown by @RustedBones in #4533
• Fix google-api-client evicted by mysql-socket-factory 1.7.0 by @RustedBones in #4531

🌱 Dependency Updates

• Update scala-collection-compat to 2.8.0 by @scala-steward in #4456
• Update sbt, sbt-dependency-tree to 1.7.1 by @scala-steward in #4457
• Update testcontainers-scala-elasticsearch, ... to 0.40.9 by @scala-steward in #4465
• Update mysql-connector-java to 8.0.30 by @scala-steward in #4473
• Update sbt-mdoc to 2.3.3 by @scala-steward in #4475
• Update scala-collection-compat to 2.8.1 by @scala-steward in #4479
• Update sbt-tpolecat to 0.4.1 by @scala-steward in #4472
• Update testcontainers-scala-elasticsearch, ... to 0.40.10 by @scala-steward in #4480
• Update sbt-bloop to 1.5.3 by @scala-steward in #4477
• Update scalatest to 3.2.13 by @scala-steward in #4482
• Update sparkey to 3.2.4 by @spotify-steward in #4408
• Update scalactic to 3.2.13 by @scala-steward in #4481
• Update sbt-avro to 3.4.2 by @scala-steward in #4485
• Update sbt-scoverage to 2.0.2 by @scala-steward in #4493
• Update scalafmt-core to 3.5.9 by @scala-steward in #4499
• Bump JamesIves/github-pages-deploy-action from 4.3.4 to 4.4.0 by @dependabot in #4469
• Update elasticsearch-java to 8.4.1 by @scala-steward in #4510
• Update sbt-protoc to 1.0.6 by @scala-steward in #4507
• Update kantan.csv to 0.7.0 by @scala-steward in #4516
• Update shapeless to 2.3.10 by @scala-steward in #4518
• Update sbt-scalafix to 0.10.2 by @scala-steward in #4521
• Update sbt-mima-plugin to 1.1.1 by @scala-steward in #4523
• Update scalacheck to 1.17.0 by @scala-steward in #4526
• Update scala-compiler, scala-library, ... to 2.12.17 by @scala-steward in #4525
• Sync jackson version with beam's by @RustedBones in #4530
• Update mysql-socket-factory to 1.7.0 by @scala-steward in #4519

New Contributors

• @Illedran made their first contribution in #4495
• @shnapz made their first contribution in #4511

Full Changelog: v0.12.0-RC1...v0.12.0-RC2

v0.11.10

This is a bugfix release for a Beam issue in Scio 0.11.9/Beam 2.39.0 affecting side inputs.

🌱 Dependency Updates

• Update beam-runners-core-construction-java, ... to 2.41.0 (#4506) @spotify-steward-bot

v0.12.0-RC1

🚀 Enhancements

• SMB secondary sort key by @kellen in #4288
• Remove unused jna dependency by @RustedBones in #4432
• Cherry-pick Parquet SplittableDoFn implementation. by @clairemcginty in #4436
• Add granularity option for Parquet-SDF reads by @clairemcginty in #4438
• Reduce coder memory footprint by @RustedBones in #4397
• Reduce warnings by @RustedBones in #4381
• Update beam to 2.40 by @RustedBones in #4433
• Remove deprecated APIs by @kellen in #4446
• Add 0.12.0 version information, deprecations, pubsub IO information by @kellen in #4447
• IT + FAQ cleanup by @kellen in #4448

🐛 Bug Fixes

• (bugfix) typed-Parquet sink wasn't using user-supplied Configuration by @clairemcginty in #4434
• Patch startRegex for es 8.3+ by @RustedBones in #4444
• Remove org.apache documentation exclusion by @kellen in #4330
• staging: allow .sbt/boot by @danielnorberg in #4262

🌱 Dependency Updates

• Bump JamesIves/github-pages-deploy-action from 4.3.3 to 4.3.4 by @dependabot in #4431
• Update sbt-scoverage to 2.0.0 by @spotify-steward in #4430
• Update elasticsearch-java to 8.3.1 by @spotify-steward in #4441
• Update sbt-bloop to 1.5.2 by @spotify-steward in #4440
• Update to elasticsearch 7.17.5 by @RustedBones in #4443

Full Changelog: v0.11.10...v0.12.0-RC1

v0.11.9

This release contains mostly dependency upgrades and a bug fix. It includes Beam 2.39.0.

🐛 Bug Fixes

• Update CustomCoders usages to implement verifyDeterministic (#4428) @clairemcginty
• Fix typo in WordCount example (#4419) @RustedBones

🌱 Dependency Updates

• Update beam-runners-core-construction-java, ... to 2.39.0 (#4410) @spotify-steward
• Update algebra to 2.8.0 (#4427) @spotify-steward
• Update elasticsearch-java to 8.2.3 (#4426) @spotify-steward
• Update parquet-avro, parquet-column, ... to 1.12.3 (#4411) @spotify-steward
• Update sbt-assembly to 1.2.0 (#4413) @spotify-steward
• Update sbt-scalafix to 0.10.1 (#4417) @spotify-steward
• Update scala-compiler, scala-library, ... to 2.12.16 (#4418) @spotify-steward

Contributors to this release

@RustedBones, @clairemcginty, and @spotify-steward

v0.11.8

This release contains mostly small bugfixes and dependency upgrades.

🚀 Enhancements

• Add es8 integration test (#4313, #4416) @RustedBones

🐛 Bug Fixes

• Unblock DataflowRunner V2 by replacing parameterized AtomicCoder refs (#4399) @clairemcginty
• Drop empty SMB key groups when predicate is provided (#4389, #4395) @kellen
• Fix run pre-release gha by stating jobs synchonously (#4401) @RustedBones
• Fix UUID shard codec for jdbc (#4379) @RustedBones

🌱 Dependency Updates

• Update sbt-paradox to 0.10.2 (#4414) @spotify-steward-bot
• Update circe-core, circe-generic, ... to 0.14.2 (#4409) @spotify-steward-bot
• Update jackson-datatype-jsr310 to 2.13.3 (#4406) @spotify-steward-bot
• Update mysql-socket-factory to 1.6.1 (#4407) @spotify-steward-bot
• Update scalafmt-core to 3.5.8 (#4412) @spotify-steward-bot
• Update sbt-assembly to 0.15.0 (#4405) @spotify-steward-bot
• Update testcontainers-scala-elasticsearch, ... to 0.40.8 (#4404) @spotify-steward-bot
• Update elasticsearch-java to 8.2.2 (#4403) @spotify-steward-bot
• Update cassandra-all to 3.11.13 (#4387) @scala-steward
• Replace deprecated upload-release-asset by action-gh-release (#4415) @RustedBones
• Replace deprecated sbt-dependency-graph plugin (#4371) @RustedBones

Contributors to this release

@RustedBones, @clairemcginty, @kellen, @scala-steward, @spotify-steward-bot

Release v0.11.7

🐛 Bug Fixes

• Fix BQ table pagination in tableReferences (#4372) @RustedBones
• Fix BQ IT test (#4375) @RustedBones

📗 Documentation

• Prepare documentation for release v0.11.7 (#4374) @RustedBones
• Fix bullet points (#4373) @wscheep

🌱 Dependency Updates

• Enable MiMa for scio-parquet and scio-elasticsearch8 (#4370) @RustedBones
• Update scalactic to 3.2.12 (#4324) @scala-steward
• Update elasticsearch-java to 8.1.3 (#4347) @scala-steward
• Fix malformed scala-steward conf file (#4369) @RustedBones

Contributors to this release

@RustedBones, @scala-steward and @wscheep

Release v0.11.6

🚀 Enhancements

• Add scio-elasticsearch8 module (#4311) @RustedBones
• Mock BQ wildcard table (#4314) @RustedBones
• Include which user is authenticated in BQ 403 exceptions (#4191) @clairemcginty
• Move PopulateTestData script to GHA (#4294) @clairemcginty

🐛 Bug Fixes

• (fix #4216) extend BQ CoderInstances in main package (#4328) @clairemcginty
• Support build metadata to identify proper previous version (#4332) @RustedBones
• Fix GHA publish due to non-deterministic implicit (#4323) @RustedBones
• Fix projection for parquet.tensorflow JobTest + add example job (#4305) @clairemcginty
• (fix #4285) Set expiration time explicitly for temp BQ view tables (#4292) @clairemcginty

📗 Documentation

• Prepare documentation for release v0.11.6 (#4346) @RustedBones
• Add spotify OSS maintainer metadata (#4326) @kellen

🌱 Dependency Updates

• Update beam to 2.38.0 (#4318) @scala-steward
• Pin caffeine dependency (#4344) @regadas
• Ignore beam dependent version updates (#4345) @RustedBones
• Update pprint to 0.7.3 (#4298) @scala-steward
• Update cassandra-driver-core to 3.11.2 (#4342) @scala-steward
• Update scalatestplus scalacheck artifact (#4343) @RustedBones
• Update elasticsearch, ... to 7.17.3 (#4337) @scala-steward
• Update mysql-connector-java to 8.0.29 (#4329) @scala-steward
• Bump JamesIves/github-pages-deploy-action from 4.3.2 to 4.3.3 (#4331) @dependabot
• Update scalatest to 3.2.12 (#4325) @scala-steward
• Update tensorflow-core-api, ... to 0.4.1 (#4301) @scala-steward
• Update scalacheck to 1.16.0 (#4309) @scala-steward
• Update scalafmt-core to 3.5.2 (#4317) @scala-steward
• Update sbt-header to 5.7.0 (#4319) @scala-steward
• Update sbt-bloop to 1.5.0 (#4321) @scala-steward
• Update zoltar-api, zoltar-core, ... to 0.6.0 (#4320) @scala-steward
• Update magnolify-avro, magnolify-bigtable, ... to 0.4.8 (#4316) @scala-steward
• Update sbt-scalafix to 0.10.0 (#4308) @scala-steward
• Update sparkey to 3.2.2 (#4312) @scala-steward
• Bump actions/setup-java from 2 to 3 (#4310) @dependabot
• Update sbt-mima-plugin to 1.1.0 (#4307) @scala-steward
• Update jackson-module-scala to 2.13.2 (#4283) @scala-steward
• Bump actions/checkout from 2.4.0 to 3 (#4275) @dependabot
• Update jna to 5.11.0 (#4296) @scala-steward
• Update mysql-socket-factory to 1.5.0 (#4304) @scala-steward
• Update shapeless to 2.3.9 (#4297) @scala-steward
• Update sbt-api-mappings to 3.0.2 (#4290) @scala-steward
• Update sbt-mdoc to 2.3.2 (#4299) @scala-steward
• Update joda-time to 2.10.14 (#4291) @scala-steward
• Update scala-collection-compat to 2.7.0 (#4293) @scala-steward

Contributors to this release

@RustedBones, @clairemcginty, @dependabot, @dependabot[bot], @kellen, @regadas, @scala-steward and @stormy-ua