Is Ably down?

We are seeing an elevated rate of connection errors in the us-east-1 region, seemingly as a result of a partial loadbalancer failure. We are disabling the region so traffic will be routed to other US sites.

Traffic is no longer being routed to us-east-1, but instead to us-east-2 and us-west-1. We are continuing to investigate the root cause of the networking issue in us-east-1.

us-east-1 remains disabled. All traffic is being routed to other regions so there is no ongoing impact on service.

us-east-1 is reenabled and the service is operating normally. The root cause has been confirmed to be a defective Network Load Balancer that was causing a small fraction of connection attempts in the region to fail. We are awaiting a postmortem from AWS on the issue, and will publish our own postmortem here once we have that.

Between about 14:00 UTC on the 20th and 17:00 on the 21st, on channels with the persist last message (https://ably.com/docs/channels#rules) channel rule enabled, attachments with `rewind=1` in situations where the most recent message was published over two minutes ago but still somewhat recently (and where the channel has been continuously active in a given region since the message was published) may not have had that rewound message correctly delivered to the attaching client. We apologise to anyone whose application was affected by this.

Between 10:35 and 12:20 UTC today, MongoDB ingress rules may have experienced errors publishing changestream events. If you were affected by this, please contact [email protected].

We're experiencing an elevated level of API errors and are currently looking into the issue.

Between 15:30 UTC to 15:36 UTC there were elevated errors affecting token requests in all regions resulting in 503 error responses.

On October 1, a distributed denial-of-service \(DDoS\) attack caused elevated token-request errors across all Ably Realtime regions for approximately six minutes and intermittent unavailability of [ably.com](http://ably.com). We restored token-request service by increasing capacity and added targeted traffic-blocking and rate-limiting rules to mitigate the attack. We are strengthening endpoint protection and reviewing our firewall controls to reduce the risk of recurrence.

Customers may be experiencing issues viewing documentation pages on our website ably.com/docs. Our realtime services are not affected by this issue at this time.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.