Lambda Errors and Runtime Errors Experienced Throughout the Application

Summary

Lambda Errors and Runtime Errors Experienced Throughout the Application

**Summary:** On May 7, 2026, from approximately 07:15 AM to 05:21 PM PT, customers across all regions experienced failures in automated workflows and email notifications within FloQast, resulting in jobs failing to execute as expected. **Root Cause:** A routine update from our cloud infrastructure provider changed how certain internal network traffic was handled. A prior update within our platform had not been fully applied across all parts of our environment, which created a conflict with the provider's change. Affected workflows failed immediately rather than completing normally. **Immediate Resolution:** The infrastructure update was rolled back to a prior stable configuration, and a process was run to fully refresh the affected workflows across all regions. All three regions were subsequently returned to normal operation. **Follow-up Actions:** We are taking several steps to strengthen our resilience to infrastructure changes of this kind. Going forward, updates from our cloud provider will be validated in a dedicated test environment before reaching production, giving us earlier visibility into any compatibility issues. We are also enhancing our internal monitoring to provide greater coverage across our infrastructure, and we are improving how we receive and act on cloud provider advisories so our engineering team can respond proactively to changes that may affect our environment.

• Started

  May 7, 2026 · 7:28 AM

• Resolved

  May 7, 2026 · 4:18 PM

• Duration

  8h 49m

• Severity

  Major

• ◐

  Investigating

  May 7 · 7:28 AM FloQast

  We are currently investigating an issue where various functions within the FloQast application are erroring out, showing either a Lambda error or a Runtime error. This is being treated with the highest priority

• ◆

  Identified

  May 7 · 9:01 AM FloQast

  The issue has been identified and a fix is being implemented.

• ◆

  Identified

  May 7 · 9:32 AM FloQast

  The fix is currently being applied in parallel to all FloQast environments. We expect the full resolution to be in place very soon.

• ◉

  Monitoring

  May 7 · 11:23 AM FloQast

  The fix has been initiated, and we are monitoring the results. We are aware of some errors still occurring due to warm lambdas which have not fully finished restarting yet. If a service is used and a lambda that has not been fully restarted yet is invoked, an in-app error may still occur.

• ✓

  Resolved

  May 7 · 4:18 PM FloQast

  The fix for this incident has finished deploying. A full RCA will be shared in this posting as soon as it is available.

• ✓

  Postmortem

  May 14 · 3:02 PM FloQast

  **Summary:** On May 7, 2026, from approximately 07:15 AM to 05:21 PM PT, customers across all regions experienced failures in automated workflows and email notifications within FloQast, resulting in jobs failing to execute as expected. **Root Cause:** A routine update from our cloud infrastructure provider changed how certain internal network traffic was handled. A prior update within our platform had not been fully applied across all parts of our environment, which created a conflict with the provider's change. Affected workflows failed immediately rather than completing normally. **Immediate Resolution:** The infrastructure update was rolled back to a prior stable configuration, and a process was run to fully refresh the affected workflows across all regions. All three regions were subsequently returned to normal operation. **Follow-up Actions:** We are taking several steps to strengthen our resilience to infrastructure changes of this kind. Going forward, updates from our cloud provider will be validated in a dedicated test environment before reaching production, giving us earlier visibility into any compatibility issues. We are also enhancing our internal monitoring to provide greater coverage across our infrastructure, and we are improving how we receive and act on cloud provider advisories so our engineering team can respond proactively to changes that may affect our environment.