INKY experienced a minor incident on March 8, 2022 affecting Dashboard Services US, lasting 4h 55m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 08, 2022, 04:28 PM UTC
Inky engineers are investigating an increase in error rates for accessing the Inky Dashboard.
- monitoring Mar 08, 2022, 04:55 PM UTC
Engineers have observed that error rates have returned to normal we are monitoring to ensure that all systems remain normal.
- resolved Mar 08, 2022, 09:24 PM UTC
This incident has been resolved.
- postmortem Mar 10, 2022, 01:28 PM UTC
# Post incident report: Start: 8-March-2022 1323 UTC End: 9-March-2022 1630 UTC Duration: 27 hr 7 min ## Summary: The Inky Dashboard is having intermittent loading issues for some customers. ## Root Cause: Resource usage on the servers was higher than expected. Systems put in place to help in cleaning up the resources used were so busy they started to use more resources than expected. The cycle continued until some systems were failing when users were trying to access the dashboard. ## Customer Impact: Some users were unable to access the Inky Dashboard intermittently. ## Mitigation Action: Separated each portion of the process out to its own server, so they can scale independently and quickly. ## Follow-up Items and Preventative Measures: 1. The entire process has been split with each step having its own resource pool to draw from. This should prevent any one step overwhelming the entire process. 1. Monitoring has also been broken down to quickly spot an issue with any individual step in the process in addition to monitoring the health of the process as a whole. This should assist us with spotting intermittent issues like this one.