Performance issue affecting UK customers

Summary

Performance issue affecting UK customers

**Summary** Across the period of Friday 9th - Saturday 10th June, several Enable Trading programs channels, hosted in the UK Live environment, experienced general slowness and performance issues. Examples of this were slow load times, slow program line saves, and long waits for reports to run. These processes seemed to be slow, however, they would not complete during the incident. Additionally, some users also found that any actions they undertook would fail instantly. ‌ **Cause** The root cause of this problem was located within Microsoft’s Azure servers, on which Enable is hosted upon. Database locking was occurring, which prevented any tasks requested in the user interface from being completed and stored on the database. As the problem was with the server itself, Microsoft were required to fix it. ‌ **Preventative measures** To improve communication during incidents, going forward, we will improve both the transparency and frequency of our communication during any severe incidents or outages in the platform. This will provide clarity that Enable are aware of the issue and are working as a priority to resolve it. Our status page will be much more prominent in our communication and handling of incidents. To improve the speed at which we identify incidents, we will extend our monitoring and tracking. These targeted improvements will plug monitoring gaps that were identified by our teams, whilst investigating the root cause of this incident. This will allow us to identify the specific cause of the issue sooner, should this happen again.

• Started

  Jun 9, 2023 · 2:45 PM

• Resolved

  Jun 10, 2023 · 6:49 PM

• Duration

  1d 4h 3m

• Severity

  Minor

• ◐

  Investigating

  Jun 9 · 2:45 PM Enable

  We're aware of a performance issue affecting many UK users of the Enable platform and have a team of Engineers working to resolve this with urgency.

• ◐

  Investigating

  Jun 9 · 2:47 PM Enable

  Our current working theory is that a replication process is preventing writes to the database. The configuration of this is managed by Microsoft, and we're working with them to identify a fix for the root cause. In the meantime, we're investigating other approaches to work around the underlying issue.

• ◉

  Monitoring

  Jun 10 · 12:17 PM Enable

  Microsoft has applied some fixes and we are seeing connectivity and performance returning to normal levels. We are continuing to monitor the situation.

• ✓

  Resolved

  Jun 10 · 6:49 PM Enable

  The connectivity and performance issues have been resolved and all systems are functioning normally.

• ✓

  Postmortem

  Jun 16 · 10:42 AM Enable

  **Summary** Across the period of Friday 9th - Saturday 10th June, several Enable Trading programs channels, hosted in the UK Live environment, experienced general slowness and performance issues. Examples of this were slow load times, slow program line saves, and long waits for reports to run. These processes seemed to be slow, however, they would not complete during the incident. Additionally, some users also found that any actions they undertook would fail instantly. ‌ **Cause** The root cause of this problem was located within Microsoft’s Azure servers, on which Enable is hosted upon. Database locking was occurring, which prevented any tasks requested in the user interface from being completed and stored on the database. As the problem was with the server itself, Microsoft were required to fix it. ‌ **Preventative measures** To improve communication during incidents, going forward, we will improve both the transparency and frequency of our communication during any severe incidents or outages in the platform. This will provide clarity that Enable are aware of the issue and are working as a priority to resolve it. Our status page will be much more prominent in our communication and handling of incidents. To improve the speed at which we identify incidents, we will extend our monitoring and tracking. These targeted improvements will plug monitoring gaps that were identified by our teams, whilst investigating the root cause of this incident. This will allow us to identify the specific cause of the issue sooner, should this happen again.