Blues incident
notehub.io and Notehub API down; other services may be degraded
Blues experienced a critical incident on July 15, 2025 affecting Notehub UI and Notehub API, lasting 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 15, 2025, 07:52 AM UTC
Notehub.io and the Notehub API are currently unavailable. Event routing appears to be working. The incident started at 2:41AM. We will publish further updates as we learn more.
- investigating Jul 15, 2025, 07:56 AM UTC
Services appear to be returning to normal, but we are continuing to investigate.
- resolved Jul 15, 2025, 08:33 AM UTC
The incident was resolved as of 3:43AM ET. Event routing was not impacted. Only access to notehub.io and api.notefile.net
- postmortem Jul 15, 2025, 08:55 PM UTC
Last night’s outage was caused by a default OS upgrade automation. While each Notehub infrastructure service is supported by multiple, redundant server instances, the automated upgrade incorrectly impacted multiple servers at the same time. This caused some key services used by the UI and API to go offline. To insure the issue doesn’t recur, we have disabled this default automation. Then, to ensure we still maintain our OS, we will implement a custom OS upgrade automation to prevent redundant server instances going offline at the same time.