Notabene Outage History

This incident has been resolved.

During planned infrastructure maintenance in our EU region, our internal message system was briefly unable to accept writes. Between 08:15 and 10:10 UTC, some transactions submitted through the API were delayed. A smaller number were accepted but never processed, so they stayed in a pending state. The API stayed available throughout. Most of the impact fell between 08:54 and 09:10 UTC and between 09:37 and 09:50 UTC.

Impact: From 14 Sep 18:14 UTC to 15 Sep 08:41 UTC, calls to POST /entities/{did}/address-ownership/discover on api.eu1.notabene.id timed out without a response. All other API endpoints, including transfer creation and authorization, were unaffected. Sandbox was unaffected. Customers using discovery as a step in their withdrawal flow saw those withdrawals held for the duration. Timeline - 14 Sep 18:14 — A network security rule permitting our EU production cluster to reach the internal address hash service was removed during infrastructure work. Discovery requests began timing out. - 15 Sep 08:21 — Customer reports received. Incident opened. - 15 Sep 08:26 — Root cause identified as a missing network rule. - 15 Sep 08:41 — Rule restored. Discovery recovered immediately. No backlog processing was required, as failed requests had not been queued. - 15 Sep 10:15 — Permanent fix to the infrastructure module merged.

This issue has been resolved and everything should be working normally now. We're continuing to monitor closely to make sure everything stays stable. Thank you so much for your patience while we worked through this.

While introducing more security layers to our systems, we blocked part of the communication between two services that ended up making the UI and API unavailable

We've now resolved the incident. Thanks for your patience.

We've now resolved the incident. Thanks for your patience.

Resolved

We've now resolved the incident.

API has recovered. A short spike in 500 errors for approximately 1% of API calls between 05:35 and 05:45 UTC due to the downstream provider issues. We will continue to monitor.

We've now resolved the incident. Thanks for your patience.

We're continuing to monitor, but as of 14:32 UTC all systems are up and running.

We've now resolved the incident, and are continuing to monitor.

We've now resolved the incident. Thanks for your patience. The recovery started at 05:22:00 UTC