Downtime on API

Incident Report for Codecov

Postmortem

This issue arose due to a rogue lock entering a permanently acquired state. This created a deadlocking scenario in our api infrastructure that led to restart and initialization issues in our api.

We have alleviated the deadlock and will work to understand and correct the source of this issue going forward.

It is suspected that this deadlocking issue caused some degree of degraded performance starting at 2021-10-11 9:40 UTC that eventually progressed to a major api outage. This issue was resolved at 2021-10-11 13:45 UTC.

Posted Nov 11, 2021 - 13:56 UTC

Resolved

We have identified and resolved the issue. All systems appear to be operational.
Posted Nov 11, 2021 - 13:51 UTC

Investigating

Codecov's API is currently experiencing an outage. We are investigating.
Posted Nov 11, 2021 - 13:25 UTC
This incident affected: Codecov Backend.