[Enhancement]: Allow additional flags for Kafka Exporter

[hunter86bg]

Related problem

Currently docker-images/kafka/exporter-scripts/kafka_exporter_run.sh lacks multiple flags that could allow the exporter to respond faster and avoid timeouts in grafana agents when having more than 4000 partitions.

Can we add options to enable 'concurrent.enable' , 'topic.workers' and 'refresh.metadata' ?

Suggested solution

Adjust docker-images/kafka/exporter-scripts/kafka_exporter_run.sh so it can apply additional flags to the Kafka Exporter.
refresh.metadata - the default value of 30s is pretty low
concurrent.enable - can improve response times of the exporter
topic.workers - can improve response times of the exporter

Alternatives

No response

Additional context

No response

[ppatierno]

Not a Kafka Exporter expert, I can understand they could be useful in your use case, not sure being useful to all users but said that, the change is not just about the script but also changing the operator logic to pass those values to the script via additional env vars.

[ppatierno]

Triaged on 23.1.2025: every new flag would need changes in the operator and configuring it for how it works today. We should put more thinking into a way for users to configure easily the flags they want without the need to update the operator each time.

[hunter86bg]

I thought that we could pass the env variable via the KafkaNodePool pod template but I checked it and indeed it doesn't have it.

[scholzj]

I'm not sure if there is anything wrong with simply adding the new options to the CRD API to be honest. It is not like the Kafka Exporter is adding new options twice per week. Alternatively, one can try to do something similar to the additionalKanikoOptions option we have in Kafka Connect. But I'm not 100% sure if we want to go the two different ways for configuring the options. That might also require a proposal as it is a more significant API change.

Environment variable in KafkaNodePool is definitely not the right way as the Node pools have absolutely nothing to do with the Kafka Exporter. So if anything, it would be through Kafka Exporter's own template in the Kafka CR.

[hunter86bg]

What about adding a single variable holding a list of additional params to the exporter.
Something like:

AdditionalKafkaFlags:
  - "flag1=value1"
  - "--flag2"
  - "--no-flag3"

Will become a variable containing them all:

KAFKA_EXPORTER_ADDITIONAL_FLAGS=" flag1=value1 --flag2 --no-flag3 "

Of course , those flags can even be validated against the binary (I guess the output of '--help' should be enough).

[scholzj]

I think that is inconsistent with everything we use anywhere else.

[ppatierno]

To be honest I don't think it's viable adding each time a new Kafka exporter option because some user is asking for that.
I agree that Kafka exporter is not adding new options each time, but right now there are a lot of them we don't support, so after this request, others could come and we should update the operator again.
I see the additionalKanikoOptions way a good way to deal with this. Of course, it would need a proposal and deprecation of the current API.

[scholzj]

Keep in mind that:

• Behind the additionalKanikoOptions there is a list of allowed options. It does not allow you to use any option you want. So it is just a different API construct, but it still needs code change to allow new option.
• There are many Kafka Exporter options we do not want the users to set. So again, they cannot be completely freely configurable and we will never want to support all of the Kafka Exporter options.

[ppatierno]

Sure, but we can always evaluate which ones should be really forbidden (i.e. kafka.server, tls.enabled, ...) and allowing all the others since the beginning if we change the API.

[scholzj]

Sure, but we can always evaluate which ones should be really forbidden (i.e. kafka.server, tls.enabled, ...) and allowing all the others since the beginning if we change the API.

I think we have in the past always opted against it. For Kaniko, IIRC my original proposal suggested to allow by default but the community wanted me to change it to allow only selected options. So I think this is mainly about the API change. Would more flexible CRD API made sense here with the list of allowed options like we have for Kaniko? Maybe it would. But not 100% sure it is worth changing given we already have the API we have. But why not 🤷.