Deadline Funnel outage

Critical · Started Aug 23, 2022 · 4:00 PM

Duration

< 1m
Severity

Critical
Detection lead

—
User reports

—

Summary

Deadline Funnel outage

At 12pm EST on 8/23/22, a deployment introduced increased load on our production database. This caused three downtime incidents between 12pm and 1pm EST: - 12:12pm - 12:21pm EST - 12:24pm - 12:32pm EST - 12:40pm - 12:49pm EST During these three incidents, the core API for Deadline Funnel stopped responding to requests, which in turn caused core Deadline Funnel functionality to stop working: - Email links - Email timers - Page integrations - Zapier integrations We resolved this issue by reverting the deployment and manually killing the long-running queries that had been introduced and that were causing these issues on the production database. When those steps were completed, both the database and API returned to a healthy state, and all functionality was restored. The root cause has been identified and additional updates are being made over the next 24 hours to prevent this from happening again in the future. We apologize for this issue and the inconvenience it caused. Please contact us at help@deadlinefunnel.com if you have any questions.

Started

Aug 23, 2022 · 4:00 PM
Resolved

Aug 23, 2022 · 4:00 PM
Duration

< 1m
Severity

Critical

Event timeline

How this incident unfolded

✓

Resolved
Aug 23 · 5:46 PM Deadline Funnel

At 12pm EST on 8/23/22, a deployment introduced increased load on our production database. This caused three downtime incidents between 12pm and 1pm EST: - 12:12pm - 12:21pm EST - 12:24pm - 12:32pm EST - 12:40pm - 12:49pm EST During these three incidents, the core API for Deadline Funnel stopped responding to requests, which in turn caused core Deadline Funnel functionality to stop working: - Email links - Email timers - Page integrations - Zapier integrations We resolved this issue by reverting the deployment and manually killing the long-running queries that had been introduced and that were causing these issues on the production database. When those steps were completed, both the database and API returned to a healthy state, and all functionality was restored. The root cause has been identified and additional updates are being made over the next 24 hours to prevent this from happening again in the future. We apologize for this issue and the inconvenience it caused. Please contact us at help@deadlinefunnel.com if you have any questions.