Hivebrite incident

Global outage on all the productions and sandboxes

Critical Resolved View vendor source →

Hivebrite experienced a critical incident on December 23, 2024 affecting European production servers and US production servers and 1 more component, lasting 13h 47m. The incident has been resolved; the full update timeline is below.

Started
Dec 23, 2024, 08:42 PM UTC
Resolved
Dec 24, 2024, 10:30 AM UTC
Duration
13h 47m
Detected by Pingoru
Dec 23, 2024, 08:42 PM UTC

Affected components

European production serversUS production serversApac production serversEuropean sandbox serversUS sandbox serversSwitzerland production servers

Update timeline

  1. investigating Dec 23, 2024, 08:42 PM UTC

    Our platform is currently experiencing technical issues that are preventing users and admins from accessing the platform. There is no impact to existing customer data. We apologize for the inconvenience and are working to resolve this as soon as possible.

  2. identified Dec 23, 2024, 08:45 PM UTC

    The issue has been identified and a fix is being implemented.

  3. identified Dec 23, 2024, 08:46 PM UTC

    We are continuing to work on a fix for this issue.

  4. monitoring Dec 23, 2024, 08:49 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved Dec 24, 2024, 10:30 AM UTC

    The incident has been resolved. Please reach out to your CSM if you have any questions.

  6. postmortem Dec 24, 2024, 02:43 PM UTC

    Dear customer, ## Incident Overview Our platform experienced an unexpected incident that prevented you from accessing Hivebrite for several minutes. We sincerely apologize for the inconvenience and here are more details on the incident. | INCIDENT PROPERTIES | | | --- | --- | | **Started** | Dec 23, 2024 08:26 pm UTC | | **Impact Ended** | Dec 23, 2024 08:49 pm UTC | | **Impact Duration** | 23 minutes | | **Environment** | prod | ## What Happened Our traffic handler that manages redirection to our marketing website URLs was not configured properly. Because of that, in case of worldwide high traffic, it loses the connection with our server. We had identified the issue a few weeks ago and were planning a maintenance window in January to correct it. Everything was ready for the resolution. We thought there would be no issue to wait as much as this configuration existed for one year and a half and it had never led to any issue until now. However, the probabilities went against us. ## Impact on You There was no data loss and there is no long lasting impact after this incident. ## What was done Thanks to everything being already ready, the team could react and correct the behaviour straightaway. The infrastructure has been corrected during the incident. It will not reoccur. We sincerely appreciate your patience and understanding during this time. We are committed to continually improving our services to deliver a more resilient and reliable experience. If you have any lingering concerns or questions, please do not hesitate to reach out to your CSM or our support team. Sincerely, Hivebrite