Omnea Outage History

The downstream AWS Cloudfront outage has been marked as resolved. Monitoring confirmed no further effect on the Omnea platform.

Confirmed errored tasks are no longer above the expected rate. We will continue to monitor environments to ensure errored tasks are successfully retrying without intervention.

The issue has been resolved. During routine database maintenance activity to remove legacy objects, a transaction was unintentionally left open. This resulted in locks being held on a number of database objects for data-integrity reasons, which impacted availability of certain endpoints. The open transaction was identified, and then committed immediately, releasing the locks and restoring normal service. We are reviewing our maintenance procedures and safeguards to reduce the likelihood of similar incidents during future operational activities.

Issue resolved by fixing a parallel query conflict.

Performance confirmed to have recovered.

Loading timeouts were being caused due to consistently retrying authentication calls unsuccessfully in a selection of environments. This has been mitigated. We will update with a root cause once fully investigated. Cause & Mitigation: There was a missing NextJS asset in some Omnea environments in the most recent deployment. This was caused by an AWS S3 internal server error during bulk resource upload. SST Custom Resource Handler v2 handles retries for most failure modes, but not AWS internal errors - instead of aborting the deploy, it continued which led to missing assets that prevented the frontend from loading. As mitigation we will migrating NextJS to SSTv4 which will handle an AWS internal server error more elegantly.

The issue is resolved. A spike in authorization requests exceeded our configured throughput limit. We intentionally throttled this spike due to pre-configured limits, however the requests failed to recover correctly because that failure was cached, which prolonged the impact until those instances were restarted. We have now mitigated the issue by deploying a follow-up change to ensure startup configuration failures are retried safely rather than persisted.

Issue has been resolved. The errors were related to a small selection of IP addresses - no discernible external impact.

The failure cause has been resolved. This issue was caused by an application version mismatch during deployment, where a frontend change deployed before dependent backend configuration updates had fully applied. That created a temporary incompatibility between the frontend and backend, which resulted in errors for authenticated users to access their own data. We mitigated the issue by reverting the change and have now implemented tighter deployment staging controls, compatibility validation for dependent changes, and faster rollback mechanisms to prevent recurrence and shorten future recovery time.

Confirmed as resolved. This was a short lived issue where over zealous security screening on uploaded documents was timing out and rejecting the upload. The timeout issue was resolved by additional pre-flagging of safe patterns.

Some users experienced intermittent “Internal Server Error” responses when loading pages with high resource load. These errors disappeared on refresh but then would reappear intermittently on similar pages. On investigation these page load errors were caused by the frontend application exhausting a resource limit in our hosting environment when under high load. We have implemented a memory boost whilst investigating where to remove load. The errors have now abated.

Confirmed no further knock-on effects.

Monitoring has shown no further degradation. The third-party services affected by the AWS outage are showing a return to operational status.

All pages are showing a consistent return to full performance.

All pages are showing a return to full performance. This incident is now resolved.

We experienced an issue earlier where emails were not being sent or were only delivered intermittently. The issue has now been identified and resolved. Email delivery is back to normal. We’re continuing to monitor to ensure full stability. Thank you for your patience.

Our provider has resolved their outage. If you still experience this issue, please try refreshing the page.