IRIS experienced a notice incident on March 28, 2019 affecting Manager API, lasting 2h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 28, 2019, 11:21 AM UTC
The Manager API has failed while applying a hotfix. This will affect customers using the Order Board product.
- identified Mar 28, 2019, 12:04 PM UTC
We are continuing to work on a fix for this issue.
- identified Mar 28, 2019, 12:29 PM UTC
We have isolated the failing service and all other components are unaffected. We have also found the root cause and we are working on a fix. We estimate that this endpoint will be available again at 13:30 UTC
- resolved Mar 28, 2019, 01:22 PM UTC
This incident has been resolved.
- postmortem Mar 28, 2019, 01:24 PM UTC
# Postmortem During the deployment of a hotfix, one of the application services hosting the Manager API, failed to deploy. This was due to the files in the service folder having their permissions changed so that they could not be modified. After engaging with our suppliers it transpired that this was a general fault in the EU West region. This fault was not visible to us as we consult the global status view when making a decision to deploy. In order to ensure that these faults are captured so that we do not deploy when infrastructure is degraded, we will now consult the regional status boards. Customers was not affected by this deployment failure but our backend components briefly ended up running different versions of our code. Regards, iRiS Systems Engineering