Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Http Sink Connector retry messages when the service has a long-time response #265

Open
NM-Narut opened this issue Sep 24, 2024 · 1 comment

Comments

@NM-Narut
Copy link

I have to created sink connector with configuration like this
{
"connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector",
"http.authorization.type": "none",
"tasks.max": "3",
"name": "{{connector-name}}",
"http.url": "{{service-endpoint-url}}",
"auto.commit.interval.ms": "15000",
"heartbeat.interval.ms": "15000",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"retry.backoff.ms": "30000",
"http.ssl.trust.all.certs": "true",
"topics.regex": "{{topic-name}}",
"max.poll.interval.ms": "3600000"
}

I reviewed the service response log and observed that the duration ranged between 5 and 10 seconds, then I counted the number of requests and I found the number of requests was more than the number of messages. Assume that if I have 500 messages in Kafka, the number of requests is more than 500 requests and I find some messages will be sent duplicate.

Which config will fix this issue ?

Thank you

@NM-Narut NM-Narut changed the title Http Sink Connector retry resume messages when the service has a long-time response Http Sink Connector retry messages when the service has a long-time response Oct 10, 2024
@chatchaisi
Copy link

I already have this issue when a consumer group is rebalancing.
API Response time around 1-2 sec.

I don't know how to fix this. If you have solution to fix this. Please reply how to resolve it to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants