Ably had 21 outages in the last 2 years totaling 18h 31m of downtime — averaging 0.9 incidents per month.
There were 21 Ably outages since October 8, 2024 totaling 18h 31m of downtime. Each is summarised below — incident details, duration, and resolution information.
We're experiencing an elevated level of API errors and are currently looking into the issue.
Between 15:30 UTC to 15:36 UTC there were elevated errors affecting token requests in all regions resulting in 503 error responses.
Customers may be experiencing issues viewing documentation pages on our website ably.com/docs. Our realtime services are not affected by this issue at this time.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're experiencing an elevated level of API errors in LiveObjects and are currently looking into the issue.
We're mitigating the impact while investigating the root cause.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We're experiencing an elevated level of errors in ap-southeast-1 and are currently looking into the issue.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
A fix has been implemented and we our monitoring the result. Error rates have decreased to normal levels and the service is currently fully operational.
This incident has been resolved.
We're experiencing an elevated level of errors in the ap-southeast-1 region and are currently looking into the issue.
This incident has been resolved.
We're experiencing an elevated level of realtime publish errors in ap-southeast-1 and are currently looking into the issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're experiencing an elevated level of attachment errors in eu-west-1 and are currently looking into the issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has now been resolved.
We are currently investigating an issue impacting SMS based 2FA login to the Ably website ably.com.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Increased level of push delivery errors for messages published in ap-southeast-2 only since 15:28 UTC, we're remediating
The issue has been identified and a fix is being implemented.
This incident has been resolved.
Customers may be experiencing connectivity issues affecting our website (ably.com), customer dashboards, the Control API (control.ably.net), the Ably blog and CMS. Our realtime services are not affected by this issue at this time.
The service is recovering, we are continuing to monitor.
This incident has been resolved.
There was a high error rate with pusher, pubnub, and mqtt protocol adaptors in one region for a duration of 11 minutes from 11:25 UTC in the eu-west-1 (Ireland) region. This has now been resolved. No other region was affected
There is a high error rate for all translator protocol adaptors globally (pusher, pubnub and mqtt). We have identified the issue and are working on a fix.
We have resolved the issue and the error rate has now dropped across all affected regions. We are continuing to monitor.
The incident is now resolved. The elevated protocol adaptor error rate affected EU regions for 36 minutes from 11.25am-12.01pm UTC, for 15 minutes in US regions from 11.50am-12.05pm UTC, and for 14 minutes in AP regions from 11.48am-12.02pm UTC.
We are currently investigating this issue.
Issue has been resolved, we are monitoring.
The incident has been resolved
We're experiencing an elevated level of realtime publish errors in eu-west-1 and are currently looking into the issue.
We're experiencing an elevated level of realtime publish errors in eu-west-1 and are looking into the issue.
We are continuing to investigate an elevated level of realtime publish errors in eu-west-1 for a single customer. There is no wider impact.
We have identified the issue which relates to an internal error rate. While this was elevated, there was no impact to customers.
We're experiencing an elevated level of errors in the us-east-1 region and are currently looking into the issue.
We are continuing to investigate. We have also identified issues with push delivery in us-east-1 and are investigating.
The us-east-1 region has been disabled, so realtime connections and API traffic should be handled without issue in other regions. Push delivery is also back to normal. We are continuing to monitor.
We are continuing to monitor for any further issues.
Service is restored in us-east-1 and all services are operating normally.
We're experiencing an elevated level of push notification errors and are currently investigating the issue.
Some push notification messages may be delayed. We have deployed a fix and are now monitoring.
Push notification errors have reduced and this issue is now resolved.
We're investigating elevated error rates for some applications in us-east-2. We will update with more information when we have it
Error rates are back to normal. We are continuing to investigate what the issue was.
Error rates have been back to normal since 0255 (UTC).
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
This incident has been resolved.
At 13:47 UTC we observed elevated failure rates of channel attaches to some channels in the ap-southeast-2 (Sydney) region. We have identified what we believe to be the cause and rolled out a mitigation, which completed at 14:08. No other regions were affected.
This incident has been resolved.
Since Monday 7th, a REST publish that was refused by Ably's servers (for example, for being above the maximum message size) had a small probability of contributing an incorrect, anomalously high message inbound refused stat (key `messages.inbound.rest.messages.refused`) to app and account stats. Since refused messages do not count towards a customer's chargeable message quota, this bug did not result on anyone being charged more or having their quota used up. However, in a couple of cases it did momentarily trigger a customer's account-wide instantaneous message rate limit (`messages.maxRate`), for which we apologise. We are rolling out a fix for this currently.
The fix for this has been fully rolled-out.
We have been alerted to an issue where our edge proxy (CloudFront) is responding to requests with 502 errors at an elevated (but still low) rate, and are investigating. If you are experiencing any issues, please let us know at https://ably.com/support.
We have identified and mitigated this issue and are now monitoring.
This incident has now been resolved.