IRIS experienced a major incident on May 16, 2023 affecting Manager, lasting 22h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 16, 2023, 10:58 AM UTC
We are currently investigating an issue with the Order centre view in GXP taking longer than normal to display. Our engineering team are investigating, and we expect an update in the next hour. Services affected: GXP - Order centres - Order details Workaround: Emails notifications, are still being sent and contain full order details, and order statuses can still be updated. Next update: 13:00 GMT
- investigating May 16, 2023, 12:06 PM UTC
The team continue to investigate the problem and as soon as an update becomes available we will update this thread. Workaround: Emails notifications, are still being sent and contain full order details, and order statuses can still be updated. Next update: 14:00 GMT
- identified May 16, 2023, 01:01 PM UTC
The engineering team have identified the cause of the issue to be with one of our partner provided services, Microsoft Azure, and are currently working with them to resolve the issue as quick as possible. Emails notifications, are still being sent and contain full order details, and order statuses can still be updated. Next update: 15:00 GMT
- identified May 16, 2023, 02:13 PM UTC
The engineering team are working on a hotfix to resolve the issue and this is currently in testing prior to production release. We will update you as we get further news on their progress. Workaround: Emails notifications, are still being sent and contain full order details, and order statuses can still be updated. Next update: 16:00
- identified May 16, 2023, 02:58 PM UTC
The engineering team are continuing testing on the potential hotfix, prior to release. As soon as we have more news on this progress we will update again. Workaround: Emails notifications, are still being sent and contain full order details, and order statuses can still be updated. Next update: 17:00
- monitoring May 16, 2023, 03:33 PM UTC
We have now put in place a hotfix to resolve the issue caused by the extra load on the database. All services are now operational and should now return to normal. The team will continue to monitor closely for the next 24hrs.
- monitoring May 16, 2023, 03:33 PM UTC
We are continuing to monitor for any further issues.
- resolved May 17, 2023, 09:28 AM UTC
This issue has now been resolved