Ably Outage History

This incident has been resolved.

This incident has now been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

The incident is now resolved. The elevated protocol adaptor error rate affected EU regions for 36 minutes from 11.25am-12.01pm UTC, for 15 minutes in US regions from 11.50am-12.05pm UTC, and for 14 minutes in AP regions from 11.48am-12.02pm UTC.

The incident has been resolved

We have identified the issue which relates to an internal error rate. While this was elevated, there was no impact to customers.

Service is restored in us-east-1 and all services are operating normally.

Push notification errors have reduced and this issue is now resolved.

Error rates have been back to normal since 0255 (UTC).

This incident has been resolved.

This incident has been resolved.

The fix for this has been fully rolled-out.

This incident has now been resolved.

For most of the production cluster, between 13:05 and 14:02 UTC on 2024-10-01, a small proportion of SSE connections were disconnected 30 seconds after establishing a connection. For the duration of that 30s they still worked fine, and connection resumption was working correctly, so this should not have caused any missed messages. The net effect would have been a small increase in subscribe latency about once every 30s. Only the SSE endpoint was affected; websocket traffic (including anyone using an Ably SDK) was not. For customers connected to ap-southeast-2 (Sydney), the duration of the effect was much longer, between 10:06 on 2024-09-30 to 14:02 on 2024-10-01.

Cloudflare is indicating that the incident is resolved, and all Ably services appear to be operating normally. We will continue to monitor the situation.

15:00 UTC - We are currently investigating increased latency for Realtime endpoints in our Europe (Ireland) region. 15:25 UTC - Latencies are back to expected levels and the service is operating as normal.

This issue is now resolved and queue and push services are operating as expected.

Between 22:43 and 22:50 UTC, us-east-1 and eu-central-1 datacenters experienced network issues communicating between each other. Seemingly no other pairs of regions were affected, only those two. Messages published in one of those two regions during that time may have experienced increased latency before being received by the other. (You can find out your local ably datacenter by querying rest.ably.io/metadata)

Between 18:58 and 19:02 UTC the eu-central-1 suffered a partial netsplit from US regions (us-east-1, us-west-1, us-east-2). Customers in some parts of the world may have experienced increased error rates and latencies during this time.

9th August 12:57 UTC We have become aware of an issue with our Ably Reactor Queues in eu-west-1 and are investigating. 9th August 13:30 UTC We have identified the issue and rectified it. The coordinator node for our eu-west-1 Reactor Queue cluster became unresponsive. This meant that the eu-west-1 endpoint - https://eu-west-1-a-queue.ably.io:15671 - was also unresponsive. 9th August 13:36 UTC We are monitoring the cluster to ensure it is working as expected. 9th Aug 13:44 UTC We can confirm that Ably Reactor Queues in eu-west-1 are now operational.