Linode Status · History · Incident #4141

RESOLVED

Service Issue - Object Storage

Minor · Started May 29, 2026 · 6:23 PM

  • Duration

    < 1m

  • Severity

    Minor

  • Detection lead

  • User reports

Summary

Service Issue - Object Storage

Between 10:13 UTC and 10:22 UTC on May 29, 2026, customers using Object Storage \(OBJ\) in London experienced 5xx errors when trying to access content stored in that location. While attempting to isolate a separate packet loss issue, Akamai sequentially upgraded 2 pairs of routers. The first pair of routers was upgraded successfully, but due to human error the second router pair's upgrade command was executed before the first pair had completed route convergence. This resulted in a temporary loss of routing paths for OBJ traffic in the London datacenter. The issue was quickly detected via an automated alert, and service was restored automatically once the firmware upgrades and routing convergence completed. Akamai's Change Management Policy requires all router upgrade activities to be performed in accordance with established procedures and predefined implementation instructions. Akamai will conduct refresher training on the Change Management Policy to reinforce compliance with the required processes and ensure strict adherence going forward. This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.


  • Started

    May 29, 2026 · 6:23 PM

  • Resolved

    May 29, 2026 · 6:23 PM

  • Duration

    < 1m

  • Severity

    None

Event timeline

How this incident unfolded

  • Resolved

    May 29 · 6:23 PM Linode

    Starting at around 10:07 UTC on May 29, 2026, Object Storage in London became unavailable. Affected customers were unable to access Object Storage during this time, and may have seen 500 or similar errors. This occurred as the result of an issue with a planned router upgrade. The router has since been brought back online, and we can confirm that the issue was mitigated at 10:19 UTC on May 29, 2026 and is no longer occurring. We apologize for the impact and thank you for your patience and continued support. We are committed to making continuous improvements to make our systems better and prevent recurrence. If you continue to experience problems, please <a href="https://cloud.linode.com/support/tickets">open a Support ticket</a> for assistance.

  • Postmortem

    Jun 8 · 9:13 AM Linode

    Between 10:13 UTC and 10:22 UTC on May 29, 2026, customers using Object Storage \(OBJ\) in London experienced 5xx errors when trying to access content stored in that location. While attempting to isolate a separate packet loss issue, Akamai sequentially upgraded 2 pairs of routers. The first pair of routers was upgraded successfully, but due to human error the second router pair's upgrade command was executed before the first pair had completed route convergence. This resulted in a temporary loss of routing paths for OBJ traffic in the London datacenter. The issue was quickly detected via an automated alert, and service was restored automatically once the firmware upgrades and routing convergence completed. Akamai's Change Management Policy requires all router upgrade activities to be performed in accordance with established procedures and predefined implementation instructions. Akamai will conduct refresher training on the Change Management Policy to reinforce compliance with the required processes and ensure strict adherence going forward. This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.

Get alerted before the next Linode outage.

Pulsetic catches degradations minutes before vendors acknowledge them.

Start monitoring free
Hey there 👋  Friends from designmodo are here to help!