Blip experienced a major incident on November 14, 2024 affecting Desk and Portal, lasting 5h 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 14, 2024, 11:56 AM UTC
We are currently experiencing low performance on Blip Portal. Impact: Slowness when using Blip Portal. Update: Our support team is actively working on it.
- investigating Nov 14, 2024, 11:57 AM UTC
We are continuing to investigate this issue.
- investigating Nov 14, 2024, 12:07 PM UTC
Incident Summary: The Desk and Portal are inaccessible for some customers. Impact Analysis: In short, the Portal presents a completely gray screen for some users and the Desk displays the Blip icon loading page (which keeps loading indefinitely). Current status: Our technical experts are gathered in a war room investigating the scenario with full focus. We are validating the integrations that are part of the redirection process for user login. As soon as we have new information, we will keep everyone updated.
- identified Nov 14, 2024, 01:38 PM UTC
Incident Summary: The Desk and Portal are inaccessible for some customers. Impact Analysis: In short, the Portal presents a completely gray screen for some users and the Desk displays the Blip icon loading page (which keeps loading indefinitely). Current status: Our technical experts are gathered in a war room investigating the scenario with full focus. The analysis revealed that the failure was due to a failure in our supplier that controls and releases resources on the Blip platform. We opened a ticket with them and requested priority service. At the same time, we found a workaround in our network infrastructure to mitigate the problem and have already received positive feedback from some customers that this action resolved the problem for them. Given this validation, we are currently implementing the same solution for other customers.
- monitoring Nov 14, 2024, 02:25 PM UTC
Mitigation Measures: We applied a workaround to our network infrastructure to mitigate the issue for all customers and received positive feedback from them that the platform is back to normal. Given this feedback, we monitor the environment to make sure it remains stable.
- resolved Nov 14, 2024, 03:08 PM UTC
The workaround in our environment has restored the functionality and usability of our platform.
- postmortem Nov 14, 2024, 03:11 PM UTC
**Start Time:** 14/11/2024 ~05:21 \(GMT-3\) **End Time:** 14/11/2024 ~11:35 \(GMT-3\) **Incident Summary:** The Desk and Portal are inaccessible for some customers. **Impact Analysis:** In short, the Portal presents a completely gray screen for some users and the Desk displays the Blip icon loading page \(which keeps loading indefinitely\). **Mitigation Measures:** We applied a workaround to our network infrastructure to mitigate the issue for all customers and received positive feedback from them that the platform is back to normal. **Root Cause Analysis:** In the face of the normalization of the environment, we classify the incident as resolved and now enter the root cause analysis stage. This process will be conducted internally, as quickly as the situation requires, and will be published here on the status page as soon as possible. If you have any questions or need additional clarification, we recommend that you contact the CSM of your account for support.