Major Service Outage - API

Summary

Major Service Outage - API

On `2025-05-13` We experienced an outage affecting the Bitly Public API from `18:12Z` to `18:34Z`. This affected the ability to use the Bitly site or create new links, but did not impact usage of any existing links. The root cause was an error with a newly provisioned application secret which caused a fatal API service error that didn't fail our deployment pipeline. Our monitoring systems detected the issue quickly and we successfully rolled back to restore services. We have since taken steps towards resolving root causes, including improving secrets validation and faster failure detection during service initialization.

• Started

  May 13, 2025 · 2:30 PM

• Resolved

  May 13, 2025 · 2:37 PM

• Duration

  7m

• Severity

  Critical

• ◐

  Investigating

  May 13 · 2:30 PM Bitly

  We are currently investigating reports of a major service outage impacting our public API. Our on-call team is working to identify and resolve this issue. We will provide an update within 30 minutes.

• ◐

  Investigating

  May 13 · 2:30 PM Bitly

  We are continuing to investigate this issue.

• ◆

  Identified

  May 13 · 2:34 PM Bitly

  The issue has been identified and we are working to restore the service.

• ✓

  Resolved

  May 13 · 2:37 PM Bitly

  This incident has been resolved.

• ✓

  Postmortem

  Jun 11 · 11:21 AM Bitly

  On `2025-05-13` We experienced an outage affecting the Bitly Public API from `18:12Z` to `18:34Z`. This affected the ability to use the Bitly site or create new links, but did not impact usage of any existing links. The root cause was an error with a newly provisioned application secret which caused a fatal API service error that didn't fail our deployment pipeline. Our monitoring systems detected the issue quickly and we successfully rolled back to restore services. We have since taken steps towards resolving root causes, including improving secrets validation and faster failure detection during service initialization.