Skip to content

Akka.Streams.Kafka v1.5.39

Latest
Latest
Compare
Choose a tag to compare
@Aaronontheweb Aaronontheweb released this 19 Mar 16:54
1.5.39
1aeacf5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

1.5.39 March 19th 2025

Akka.Streams.Kafka 1.5.39 represents a major improvement in stability and performance for Kafka stream processing, particularly for applications using manual partition assignment and rebalancing scenarios. This release includes critical fixes for partition management and introduces new performance tuning capabilities that give users more control over their Kafka consumer behavior.

Key Improvements

  • Improved stability during partition rebalancing operations
  • Enhanced manual partition assignment behavior
  • Significantly more reliable and improved committing performance
  • New performance tuning capabilities for consumer polling
  • Better handling of partition revocation scenarios
  • Improved memory efficiency through C# record types

Breaking Changes

  • Manual partition assignment now uses IncrementalAssign instead of Assign - this prevents offset resets for users running ManualSubscriptions. If you're using manual partition assignment, you'll need to verify your offset management logic is compatible with incremental assignment behavior.
  • Several internal types have been converted from classes to C# records for better performance and nullability support. This change should be transparent for most users as records are fully compatible with standard class usage patterns. The only potential impact would be if you're using inheritance on these types (which is not a recommended pattern for Akka.Streams.Kafka types).
  • We removed some extension methods that should have never been made public in the first place.
  • We made some changes to ICommittable interface and others.

Major Bug Fixes and Improvements

Performance Data

For the PlainSource:

                                                                                     
BenchmarkDotNet v0.14.0, Pop!_OS 22.04 LTS                                           
13th Gen Intel Core i7-1360P, 1 CPU, 16 logical and 12 physical cores                
.NET SDK 9.0.100                                                                     
  [Host]  : .NET 9.0.0 (9.0.24.52809), X64 RyuJIT AVX2                               
  LongRun : .NET 9.0.0 (9.0.24.52809), X64 RyuJIT AVX2                               
                                                                                     
Job=LongRun  EvaluateOverhead=False  Concurrent=True                                 
Server=True  InvocationCount=1  IterationCount=10                                    
LaunchCount=3  RunStrategy=Monitoring  UnrollFactor=1                                
WarmupCount=3  Categories=MacroBenchmark,Consumer,Plain
Method PollBatchSize Mean Error StdDev msg/sec
ConsumeMessageAsync 500 41.29 μs 1.960 μs 2.933 μs 24,217.30

This is a ~2.5x improvement over what v1.5.38 was able to achieve.

Dependencies

1.5.39-beta2 March 14th 2025

1.5.39-beta1 March 13th 2025

v1.5.39 is a major update for Akka.Streams.Kafka

Changes:

  • 1aeacf5 Updated README.md (#480)
  • 7e8eeb4 Added v1.5.39 release notes (#479)
  • f18fdd9 restructured our CommittableOffset benchmark (#478)
  • da1498e Performance tuning consumer and committer settings (#472)
  • 5666fbc Update CommitCollectorStageBenchmark.cs
  • c256a88 added CommitCollectorStage benchmark (#477)
  • e5ddd61 fix issues with CommittableSource benchmarks (#476)
  • 125b25a Created CommittableOffsetBatchBenchmark (#474)
  • aa94008 readjust all benchmarks to use just the default values (#473)
  • 592faa6 Add specs for Committing stages (#471)
See More
  • 08a6d5f KakfaConsumeActor: Stopping behavior and shutdown loggingi (#470)
  • 199d0db Adding support for configurable commit collecting / draining (#458)
  • 8ab0d82 make ProducerRecord a record (#469)
  • 4e5ecb3 Migrated ConnectionCheckerSettings to use record types (#468)
  • f079d35 Bump tests and example / benchmark runtime to .NET 9 (#467)
  • 18c525e restructure and expand benchmarks (#466)
  • 2bd3733 fix IControl and ensure clean draining / termination of Akka.Streams consumers (#464)
  • d60c4f7 quality: exclude obsolete and non-essential code from coverage reports (#463)
  • d020b18 quality: generate and publish CRAP scores to AzDo (#462)
  • ced3532 fixed PowerShell (#461)
  • 090e127 Update nightly-builds.yaml for Azure Pipelines
  • b193311 remove FAKE, simplify build system (#460)
  • c82461d Added v1.5.39-beta2 release notes (#459)
  • 0788a54 upgraded to Akka.NET v1.5.39 (#457)
  • 1c72293 Added v1.5.39-beta1 release notes (#456)
  • 359aa54 Resolve ArgumentException upon partition re-assignment (#454) [ #415 ]
  • 71a6606 Change Assign and AssignWithOffsets to use IncrementalAssign (#455)
  • 97fb44a Add RebalanceExtTests for testing partitioned message loss on rebalance (#453)
  • aead3e4 Refactor SubSourceStageLogic; filter messages from revoked partitions in partitioned stream sources (#452)
  • 215a232 Rewrite RebalanceIntegrationTests; filter out buffered records from recently revoked partitions (#450)
  • 573437d Enable nullability (#449)
  • a39e933 [WIP] fix nullability / build warnings (#448)

This list of changes was auto generated.

Assets 3
Loading