Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_output_as_buffered_retries: fix flaky test #4877

Merged
merged 1 commit into from
Mar 24, 2025

Conversation

Watson1978
Copy link
Contributor

@Watson1978 Watson1978 commented Mar 23, 2025

Which issue(s) this PR fixes:
Fixes #4873

What this PR does / why we need it:
Before invoke @num_errors_metrics.inc, the does retries correctly when #try_write fails test will be failed if thread will be switched at

def update_retry_state(chunk_id, using_secondary, error = nil)
@retry_mutex.synchronize do
@num_errors_metrics.inc

It is easy to reproduce by adding sleep just before @num_errors_metrics.inc, like:

diff --git a/lib/fluent/plugin/output.rb b/lib/fluent/plugin/output.rb
index a987937b..d97d8ec6 100644
--- a/lib/fluent/plugin/output.rb
+++ b/lib/fluent/plugin/output.rb
@@ -1312,6 +1312,7 @@ def check_slow_flush(start)
 
       def update_retry_state(chunk_id, using_secondary, error = nil)
         @retry_mutex.synchronize do
+          sleep 0.1
           @num_errors_metrics.inc
           chunk_id_hex = dump_unique_id_hex(chunk_id)
 

This patch will wait until @num_errors_metrics.inc is invoked.

Docs Changes:

Release Note:

@Watson1978 Watson1978 marked this pull request as ready for review March 23, 2025 13:40
@Watson1978 Watson1978 requested a review from daipom March 23, 2025 13:40
Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978 Watson1978 force-pushed the test_output_as_buffered_retries branch from 1880b84 to 90a0025 Compare March 23, 2025 13:41
@Watson1978 Watson1978 added the CI Test/CI issues label Mar 23, 2025
@ashie ashie merged commit a10f0b0 into fluent:master Mar 24, 2025
10 checks passed
@Watson1978
Copy link
Contributor Author

Thanks!

@Watson1978 Watson1978 deleted the test_output_as_buffered_retries branch March 24, 2025 02:03
@Watson1978 Watson1978 added the backport to LTS We will backport this fix to the LTS branch label Mar 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport to LTS We will backport this fix to the LTS branch CI Test/CI issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI: Flaky 'does retries correctly when #try_write fails' test
2 participants