Ably incident

Occasional anomalously-high `messages.inbound.rest.messages.refused` stat when a rest publish is refused

Notice Resolved View vendor source →

Ably experienced a notice incident on October 11, 2024 affecting Ably Platform, lasting 2h 7m. The incident has been resolved; the full update timeline is below.

Started
Oct 11, 2024, 10:12 AM UTC
Resolved
Oct 11, 2024, 12:20 PM UTC
Duration
2h 7m
Detected by Pingoru
Oct 11, 2024, 10:12 AM UTC

Affected components

Ably Platform

Update timeline

  1. identified Oct 11, 2024, 10:12 AM UTC

    Since Monday 7th, a REST publish that was refused by Ably's servers (for example, for being above the maximum message size) had a small probability of contributing an incorrect, anomalously high message inbound refused stat (key `messages.inbound.rest.messages.refused`) to app and account stats. Since refused messages do not count towards a customer's chargeable message quota, this bug did not result on anyone being charged more or having their quota used up. However, in a couple of cases it did momentarily trigger a customer's account-wide instantaneous message rate limit (`messages.maxRate`), for which we apologise. We are rolling out a fix for this currently.

  2. resolved Oct 11, 2024, 12:20 PM UTC

    The fix for this has been fully rolled-out.