Downtime with /bash endpoint
Incident Report for Codecov
Postmortem

The root cause of the /bash endpoint’s downtime was the result of an infrastructure migration that completed at 22:00 UTC on Sep 23, 2022. The migration resulted in /bash attempting to resolve against infrastructure that no longer existed. The fix involved directing /bash to the newly migrated infrastructure and was completed at 23:39 UTC on Sep 24, 2022.

Future work will be to improve monitoring around the /bash endpoint to ensure that these lapses in uptime, should they occur again, are detected more quickly.

Through our efforts to direct users away from the bash uploader and toward alternative solutions (e.g., the binary uploader, github action, the circleCI orb, etc); the bash uploader comprises less than 5% of all uploader use and falling. While we will make efforts to monitor the uptime of /bash to prevent these sorts of issues in the future, it is still our plan to fully sunset the bash uploader with time; and we recommend that users migrate away from the bash uploader when/if possible.

Posted Sep 24, 2022 - 23:57 UTC

Resolved
This issue is now resolved. We will be providing additional information soon.
Posted Sep 24, 2022 - 23:38 UTC
Investigating
We are currently investigating downtime with our /bash endpoint. Our binary uploader, github action, circleCI orb, and buildkite build step seem to be unimpacted by this issue.
Posted Sep 24, 2022 - 23:33 UTC
This incident affected: Codecov Backend.