DRACOON experienced a major incident on January 25, 2024 affecting Branding, lasting 1h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 25, 2024, 08:11 AM UTC
We are currently investigating an issue with the Branding service. Our team is working to gather more information and resolve the issue as quickly as possible. We apologize for any inconvenience this may cause and will provide updates as soon as we have them.
- monitoring Jan 25, 2024, 08:16 AM UTC
The issue with the Branding Service has been resolved, and we are monitoring the situation to ensure it remains stable. We apologize for any inconvenience this may have caused and appreciate your patience.
- resolved Jan 25, 2024, 09:16 AM UTC
The issue with the Branding Service has been fully resolved. All systems are now operating normally. We apologize for any inconvenience this may have caused and appreciate your patience. If you continue to experience any issues, please don't hesitate to reach out to our support team for assistance.
- postmortem Feb 14, 2024, 01:19 PM UTC
We experienced an issue with the Branding Service on 2024-01-25 09:00 - 10:15. Our team has worked diligently to identify the root cause and implement a resolution. In this post-mortem, we want to share the details of what happened, why it happened, what we did to resolve it, and what we will do to prevent similar incidents in the future. What happened? During the time of the incident, the default branding was displayed on the login page for some requests instead of the configured custom branding. Why did this happen? The underlying issue with the incorrect branding being displayed was a timing issue introduced with a recent update of our authentication service. Because the issue was an edge case, not all running instances of the authentication service were affected and therefore not every response contained the incorrect branding. What did we do? We applied a temporary workaround to fix the issue immediately and will permanently fix the issue in the next version of our authentication service. What can we do to improve? We will further analyze the root cause of the underlying issue and improve our automated testing procedures to prevent similar incidents from happening in the future. We apologize for any inconvenience this incident may have caused. We are committed to ensuring the stability and reliability of our services and will continue to take proactive measures to prevent similar incidents from happening in the future. If you have any questions or concerns, please don't hesitate to reach out to our support team for assistance.