Addressing CI Reporting and Use of Upload/Error Status

[codecovdesign]

Problem to solve

Codecov displayed in the UI that the upload for one of the reports failed, but the CI did not fail and exited successfully. Additionally, they received the pull request comment with coverage data despite the upload failure. This raises the question of whether this could be a processing failure rather than an error in the upload step.

Furthermore, another example reported an engineer pushed a commit with a future date (Aug 1st) due to their system clock being set ahead. This caused Codecov to show the latest upload as occurring sometime in the future. The customer is requesting a way to delete the offending commits from Codecov.

issue 1, upload is shown with X:

then user lands on PR page and sees CI passed, but no data:

Summary of areas to investigate:

• Upload Failure vs. Processing Failure:

  • Investigate if the issue is due to a processing failure rather than an upload error

• Investigate CI Success with Upload Failure:

  • Confirm if the issue lies in the processing phase after the upload.
  • Cross-check logs and reports for any discrepancies.

related issue: #2442

Solution

Investigation note: #2220 (comment)
Figma link: https://www.figma.com/design/4Z7yb2dkIIATkfzpWoMYQq/GH-2220?node-id=1-2

We are going to decouple the fixes in the issues below so we can ship some of the designs sooner.

Implementation tasks

Preview Give feedback

1. Update UI copy for Coverage reports history and the body of the no coverage data description #2521
   Dev-Ready +1
   
2. Implement the Upload processing status UI component #2522
   Dev-Ready +1
   
3. Tooltip for explaining CI status #2523
   Dev-Ready +1
   
4. The ordering of uploads from new to old uploads on front-end display #2524
   Dev-Ready +1
   

[codecovdesign]

Sync with @Adal3n3 @drazisil-codecov @vlad-ko

Related issues

• pr page missing sha/pr name: Project and/or change in commit/pr list #1963
• on the pr/commit list, for the upload:x copy update: upload => report processing
  • we have pending currently. what is the state of error?
  • we should track CI status and report processing status
  • remove the error display as this is untrue
  • report processing is only queued, started, processing, complete or failed
• the upload itself finishes when the CLI says it's finished, anything past that is not an upload error. the usage of word upload is more accurately processing of the reports. IF there was an upload failure the codecov system wouldn't know about it, but if there was a processing issue that is what we can report.
• empty state during "processing" OR dont' show error: Confusion of various states up until processing finalizes #2238

[codecovdesign]

related: #2400

[codecovdesign]

Related feedback internally

This screen feels like a “work in progress”. You have the list of “view CI build” links. But in my case the CI succeeded and also the upload of the coverage report succeeded but only the processing of the upload failed (because the report had broken stuff in it).
Also I can download the uploaded reports on the right side, but I do not get the relation to the CI build that was used doing that upload.
If would be cooler to have one list that shows me what upload was broken, with a link to download the upload and a link to the corresponding CI build.
Oh, and the failed stuff is most of the time just a “still working on it”, so if you refresh the page a couple of minutes later, the failures are gone. Would be cool if it could say that it is still in processing and not failed.

[Adal3n3]

Meeting note w @calvin-codecov 9/6:
Question 1: Why did upload fail, but we still show that the CI passed?

Question 2: In this example, why do we show that the CI passed and the PR states is "all modified lines are covered by tests ✅" even tho the commit has a failed upload?

[codecovdesign]

sync from sept 9th:

• focus on processing statuses across landing pages, commit, PR, (pr comment, ideation: intermediary PR comment and link to manual trigger ) (related in terms of logic / current behavior: Improve Behavior and Consistency of Report Processing Logic Across PR comment and App (Commit and PR) #2400)
• consider alternative use of "upload" copy, consider processings etc
  • are we sure we want to move this to "report"; tradeoff is that at no point ever does codecov know about the upload status fully, until the aggregated report is there.
• consideration of removing CI status OR at least conditionally depending on config
• related issue where patch only would be shown

[Adal3n3]

Investigation note:

1. The “uploads” section is actually a “Coverage reports/uploads version history”, it contains both old and new uploads. If anyone from your team re-runs all jobs, a new version of the uploads will be generated. So any previous uploads with error messages represent older versions, and the errors are glued to those specific versions.
   • Suggestion: We need a mechanism to detect old vs. new uploads. This will require API and database work. As a user, I would expect to focus on the latest upload, so removing older uploads could be helpful.
2. The CI status and upload status are working as intended. They are two different statuses with no direct relationship. The CI status is from the commit/pull checks and they are unrelated to Codecov checks. Codecov considers all non-codecov statuses to be CI statuses. This means your CI status is based on all your non-codecov related checks so if your Codecov checks fail while other checks pass, you will still see a green “CI passed”. Note 1 and 2 explain why you might have a passing CI even your upload failed.
   
   
3. This page primarily shows data from the very first job attempts, which is why we’ve received feedback about mismatched information between the UI, GitHub, and PR comments. GitHub always shows users the latest jobs, but the UI doesn’t update from the first run to the latest run. From a typical user’s perspective, when they see an error, they expect to visit the app, fix the error, and have the issue resolved. However, in the current app experience, Codecov doesn't reflect the re-run of all jobs. Users need to open a fresh commit to display their coverage reports.
   Suggestions:
   1. Consider supporting the “re-run all jobs”, and always display the latest job.
   2. or direct users to open a new commit to display the updated coverage report.
4. Tricky case: The UI currently shows “Upload Failed” when the upload is still processing. Once the upload completes, the failure message disappears, which can be misleading.
   • Solution: Display a "Upload Processing" status while the upload is in progress.
     

(cc @calvin-codecov)

[Adal3n3]

Existing upload status:

• error -
• uploaded -
• processed - means successful for "uploaded" type uploads
• complete - means successful for carryforward type uploads
• started -

[calvin-codecov]

It would be helpful if we could figure out which uploads correspond to which run job number

(https://github.com/codecov/gazebo/actions/runs/10355997185)

[Adal3n3]

From @vlad-ko

I think I've fond another "bug/feature" situation with the CFF. If i hav a flag that's designated as a CFF, if i upload a broken report, we will sort of ignore it and just use the CFF version of the report. It could be a feature, but it could also be a problem, because the customer may not realize that we're not using the latest data.

ToDo: @calvin-codecov can you investigate this? If it's true, we need either reject the CFF broken report, and show an error message, and somehow indicate we are not using the latest data or see if we can support the latest data.

[Adal3n3]

• We can not support the re-run all jobs you created after the initial run, and show the latest job in the UI.
• We can't provide the targeted link to the upload location but we can provide job and build ID.
• We can't detect old and new uploads but we can use flags or timestamps to separate them

[kfbustam]

As a result of these failures, the code highlighting could potentially be inaccurate on the Github Files Changed tab right? Is it possible to surface a message due to that somewhere in the Github UI?

Maybe instead of:

or

"We haven't received the signal to send notifications which might mean reports are still being uploaded, so code highlighting might be inaccurate. Thanks for your patience"
"Not all reports were successfully processed, so code highlighting is likely inaccurate. Please feel free to reach out to a Codecov admin."
etc.

[Adal3n3]

Thanks for the nice suggestion @kfbustam. I have circled it back to my team and we will investigate to see if it's possible.