Elevated Error Rates for Receipts / ECOMM APIs

Summary

Elevated Error Rates for Receipts / ECOMM APIs

Upstream issues with impacted services have been resolved and platform stability has been restored.

• Started

  Dec 7, 2021 · 3:59 PM

• Resolved

  Dec 8, 2021 · 4:16 AM

• Duration

  12h 17m

• Severity

  Major

• ◐

  Investigating

  Dec 7 · 3:59 PM flexEngage

  We are experiencing elevated error rates for Receipts and ECOMM APIs. We are currently investigating the issue.

• ◐

  Investigating

  Dec 7 · 3:59 PM flexEngage

  We are continuing to investigate this issue.

• ◐

  Investigating

  Dec 7 · 4:16 PM flexEngage

  We are continuing to investigate this issue.

• ◐

  Investigating

  Dec 7 · 4:44 PM flexEngage

  We are continuing to experience elevated error rates due to an AWS outage and are reviewing the impacted services. The current impacts are as follows: * Receipts API experiencing elevated error rates * Order Status (ecomm) API experiencing slow response times * Receipt and order/ecommerce notifications are delayed * Shipment tracking page (flexTracker) experiencing slow response times and elevated error rates * You may experience issues accessing sent Order Status (ecomm) messages in our app * Email Validation API has been removed from the list of impacted services

• ◐

  Investigating

  Dec 7 · 4:45 PM flexEngage

  We are continuing to investigate this issue.

• ◐

  Investigating

  Dec 7 · 4:46 PM flexEngage

  We are continuing to investigate this issue.

• ◆

  Identified

  Dec 7 · 5:55 PM flexEngage

  The issues being experienced on our platform are a direct result of a known AWS outage. Our team is continuing to monitor the situation and look for ways to mitigate impact.

• ◆

  Identified

  Dec 7 · 10:10 PM flexEngage

  While issues with AWS persist, the impact on the affected services has started to abate. We are currently seeing improved reliability across all impacted services. However, until the issue is considered resolved on the AWS side, our team will continue to monitor and mitigate wherever possible.

• ◉

  Monitoring

  Dec 7 · 10:44 PM flexEngage

  A fix has been implemented and is being monitored AWS-side, and we continue to experience improvements in the stability of our services. We will be monitoring ongoing and will provide further updates as the situation changes.

• ✓

  Resolved

  Dec 8 · 4:16 AM flexEngage

  Upstream issues with impacted services have been resolved and platform stability has been restored.