OnlineOrNot incident

Degraded performance for Uptime Checks

Resolved after 4 hours 32 minutes, on January 16, 2025.

resolved Jan 17, 2025, 08:14 AM UTC

On 2025-01-16, between 18:05 UTC and 21:32 UTC, OnlineOrNot had degraded performance in its uptime checker. During this incident, uptime checks would take longer than usual to complete (often taking several minutes rather than the usual 30 seconds between checks). The root cause was a loss of connectivity between our Cloudflare Workers service, and the AWS database that stores metadata about uptime checks. There was a replica system ready to run in AWS, however due to a misconfiguration it could not start automatically. The misconfiguration has been fixed, and OnlineOrNot will run regular (monthly) drills to ensure the fallback system functions as expected.

Started
Jan 16, 2025, 05:00 PM UTC
Resolved
Jan 16, 2025, 09:32 PM UTC
Duration
4h 32m
Detected by Pingoru
Jan 16, 2025, 05:00 PM UTC

Update timeline

  1. resolved Jan 17, 2025, 08:14 AM UTC

    On 2025-01-16, between 18:05 UTC and 21:32 UTC, OnlineOrNot had degraded performance in its uptime checker. During this incident, uptime checks would take longer than usual to complete (often taking several minutes rather than the usual 30 seconds between checks). The root cause was a loss of connectivity between our Cloudflare Workers service, and the AWS database that stores metadata about uptime checks. There was a replica system ready to run in AWS, however due to a misconfiguration it could not start automatically. The misconfiguration has been fixed, and OnlineOrNot will run regular (monthly) drills to ensure the fallback system functions as expected.

Your product runs on more than OnlineOrNot

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →