Introduce active_job_report_on_retry_error

[solnic]

⚡ TL;DR

• active_job_report_after_job_retries introduced in add report_after_job_retries support for ActiveJob #2500 is replaced by active_job_report_on_retry_error
• it was needed to make reporting on each retry error work otherwise reporting errors when there was no retry would stop working

⏳ Longer explanation

Plot twist: the active_job_report_after_retries turned out to be problematic because if we want to report on each retry failure too, then we must always report on the last retry in the perform patch.

Previously the retry_stopped handler would do the reporting of the final attempt error and the perform patch would skip reporting to avoid duped reports, but that does not work when reporting on each retry error is meant to work too, because there is no way to tell whether the final attempt was already reported by the retry_stopped handler or not. This means that when a job failed but there was no retry, and the reporting on retry errors is turned on, then we would not get any report because the perform patch would skip reporting since reporting on retry errors is turned on.

This is really confusing, I know!

Luckily, this PR changes our original new setting to become active_job_report_on_retry_error which is a simpler solution since it only adds new logic for reporting errors on each retry and stays away from the original final-failure handling implemented in the perform patch.

This setting is disabled by default because it is the original behavior of the SDK and we don't want to change it in a patch release.

Once you enable active_job_report_on_retry_error, Sentry will be reporting exceptions on each job retry error using enqueue_retry.active_job handler, and the final (last retry attempt) failure will be reported by the perform patch.

There is no other way of doing this because of how ActiveJob works. We cannot use around_perform as described here 9303b9a and there are no AS notifications about errors at the moment that we could leverage except the one for retries.

In a future Rails release there errors will be reported using an error reported that we'll be able to hook into though, so we will most likely have a dedicated handling logic for future Rails versions eventually.

Refs #2500
Refs #2598
Refs #2597

[solnic]

@modosc 👋🏻 I ended up reworking the solution that you originally implemented in #2500 after all. Please let me know if this makes sense for you.

@aburgel this most likely addresses the issue you reported in #2573 (comment)