Outage of AdvicePay Application and API

Critical · Started Apr 19, 2023 · 11:37 AM

Duration

< 1m
Severity

Critical
Detection lead

—
User reports

—

Summary

Outage of AdvicePay Application and API

AWS S3 was having issues with very high latency on the day of the outage. Additionally, AdvicePay was using a default HTTP client without a timeout to request firms' logo images from S3. The combination of those two issues led to a denial of service due to starvation of connections on our servers. AWS has since resolved things on their end. To protect against this issue in the future, AdvicePay has implemented timeouts for all HTTP clients that we use for requesting any resource.

Started

Apr 19, 2023 · 11:37 AM
Resolved

Apr 19, 2023 · 11:37 AM
Duration

< 1m
Severity

Critical

Event timeline

How this incident unfolded

✓

Resolved
Apr 19 · 11:54 AM AdvicePay

At 11:37 AM MDT on 4/19/2023 we received an alert from our uptime monitoring service that the AdvicePay application was unreachable. Functionality was restored as of 11:50 AM MDT. We are investigating the underlying cause.
✓

Postmortem
May 18 · 12:59 PM AdvicePay

AWS S3 was having issues with very high latency on the day of the outage. Additionally, AdvicePay was using a default HTTP client without a timeout to request firms' logo images from S3. The combination of those two issues led to a denial of service due to starvation of connections on our servers. AWS has since resolved things on their end. To protect against this issue in the future, AdvicePay has implemented timeouts for all HTTP clients that we use for requesting any resource.