Deadline Funnel admin degradation

Major · Started Feb 23, 2021 · 6:30 PM

Duration

< 1m
Severity

Major
Detection lead

—
User reports

—

Summary

Deadline Funnel admin degradation

We experienced a service degradation in the Deadline Funnel admin from 1:30pm EST to 7:30pm EST on 2/23/21. We apologize for the inconvenience this caused. The issue was related to an underlying database storage issue, but was limited in scope. The Deadline Funnel admin (app.deadlinefunnel.com) was the main service impacted - all other services continued to work as expected during this time. Services that continued working during this time include webhooks, email links, email timers, floating bars, and Portal. The root cause has been identified and resolved. The issue occurred because of a large temporary file that was created by a secondary process in an application database. This large temporary file did not get released automatically and required manual intervention to remove. We are also actively working on implementing additional safeguards and upgrades to prevent this from happening again.

Started

Feb 23, 2021 · 6:30 PM
Resolved

Feb 23, 2021 · 6:30 PM
Duration

< 1m
Severity

Major

Event timeline

How this incident unfolded

✓

Resolved
Feb 24 · 1:07 AM Deadline Funnel

We experienced a service degradation in the Deadline Funnel admin from 1:30pm EST to 7:30pm EST on 2/23/21. We apologize for the inconvenience this caused. The issue was related to an underlying database storage issue, but was limited in scope. The Deadline Funnel admin (app.deadlinefunnel.com) was the main service impacted - all other services continued to work as expected during this time. Services that continued working during this time include webhooks, email links, email timers, floating bars, and Portal. The root cause has been identified and resolved. The issue occurred because of a large temporary file that was created by a secondary process in an application database. This large temporary file did not get released automatically and required manual intervention to remove. We are also actively working on implementing additional safeguards and upgrades to prevent this from happening again.