IRIS experienced a notice incident on October 22, 2019 affecting Manager and Europe and 1 more component, lasting 2h 47m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 22, 2019, 10:03 PM UTC
Our monitoring system has alerted us to the fact that orders are not being processed and manager may not respond. We are investigating.
- identified Oct 22, 2019, 10:31 PM UTC
We have isolated the fault to a problem with storage. Storage operations are currently failing, we will update here once we have further information.
- identified Oct 22, 2019, 11:08 PM UTC
We have isolated additional services as being affected by this storage problem and we have updated the affected components list. We are working with our storage supplier to rectify this fault.
- monitoring Oct 23, 2019, 12:26 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 23, 2019, 12:51 AM UTC
This incident is now resolved. A postmortem is expected in the next 48 hours.
- postmortem Oct 24, 2019, 04:00 PM UTC
During the evening of the 22nd of October one of the storage services that we depend on entered an inconsistent state. In order to ensure the integrity of the stored data the storage service was brought offline. A number of our services that depend on the underlying storage failed as a result and this prevented guest messaging, orders and service requests from being processed. Our guest facing applications delivered content throughout with reduced performance. Once the storage inconsistencies were resolved and storage was brought online, all other services recovered. We have investigated the reliance on the storage services and how this is implemented throughout GXP and we will make adjustments that will mean that future potential storage failures will not affect the product to this extent. These changes will be delivered as part of our normal product update process. We would like to take this opportunity to apologise to our customers and partners for any inconvenience that this outage may have caused. Please contact your account manager should you have further questions or require more information. Regards, iRiS Systems Engineering