Timeout for Kafka consumers/subscribers

[mruslan97]

Hi guys! I've encountered the following issue: there are several subscribers for Kafka under the same consumer group id, all reading from one topic. One of them experienced a problem with a long response time from an external service, which affected the others. Is it possible to set some kind of timeout for processing for each subscriber or consumer group id? Or would it be better not to subscribe them under the same consumer group id?

My case: reading one type of message from a single topic but with different processing logic.

[BEagle1984]

No, Silverback does not have a built-in timeout mechanism for subscribers. However, you can easily implement one using a CancellationTokenSource with a defined timeout and passing the corresponding token to your processing logic.

That said, if you are consuming each message only once using the same consumer, I strongly advise against having multiple subscribers handling the same consumed message. The main issue is that error handling and message acknowledgment cannot be performed consistently. For example, if one subscriber throws an exception, should Silverback acknowledge the message or not? This ambiguity can lead to inconsistent processing behavior.

I recommend two possible approaches:

1. Centralized processing
Use a single subscriber that orchestrates calls to different services. This allows you to control error handling explicitly - deciding when to rethrow an exception, roll back a transaction, or apply any other business logic.

2. Separate consumers per subscriber
If the subscribers are truly independent, you should create separate consumers for each. You can achieve this by:
Assigning different consumer group IDs and either filtering as described in the documentation or using different message models per each consumer (e.g., class MyMessage, class Subscriber1MyMessage : MyMessage, class Subscriber2MyMessage : MyMessage) and configuring each consumer to deserialize the appropriate type.
This ensures that each subscriber processes messages independently while maintaining the at least once semantic.