Slow updates with 3K+ objects

[IliaGe]

What happened: After creating a Service (Headless) object - it takes ~15 minutes to update Route53 DNS record

What you expected to happen: Record will be created upon creation

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• External-DNS version (use external-dns --version): v0.13.5
• DNS provider: AWS Route53
• Others: Sources: service, ingress, istio-gateway, istio-virtualservice

We are running on EKS with BatchSizeConfig of 1,000 (default), we have ~3K Isito virtualService objects & 1K Service objects.
The actual update to R53 happens only after 10-15 min:

time="2025-01-19T08:34:20Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T08:42:11Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T08:50:02Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T08:57:53Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T09:05:44Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T09:13:36Z" level=info msg="Applying provider record filter for domains: [xxx]"
time="2025-01-19T09:21:28Z" level=info msg="Applying provider record filter for domains: [xxx]"

In between, K8S objects scanning, which is slow (3-4 VS's every seconds)
time="2025-01-19T09:21:50Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:50Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:50Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:50Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:51Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:52Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:53Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:53Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"
time="2025-01-19T09:21:53Z" level=debug msg="No endpoints could be generated from VirtualService xxx/xxx"

Anything I can Do ?
I have tried enabling the "--events" , but nothing happen upon object creation :(

[IliaGe]

Just to update, that I have created additional external-dns workers, with annotation filtering
to be able to process items faster :/