Bugfender incident

Service Disruption: Datacenter Outage

Critical Resolved View vendor source →

Bugfender experienced a critical incident on March 6, 2024 affecting Public website (bugfender.com) and Web dashboard (dashboard.bugfender.com) and 1 more component, lasting 6h 47m. The incident has been resolved; the full update timeline is below.

Started
Mar 06, 2024, 03:00 AM UTC
Resolved
Mar 06, 2024, 09:47 AM UTC
Duration
6h 47m
Detected by Pingoru
Mar 06, 2024, 03:00 AM UTC

Affected components

Public website (bugfender.com)Web dashboard (dashboard.bugfender.com)Log ingestion API

Update timeline

  1. investigating Mar 06, 2024, 05:05 AM UTC

    We are currently experiencing an unexpected outage at our datacenter affecting all services. Our team is actively investigating the issue and working to restore services as quickly as possible. We apologize for the inconvenience and appreciate your patience. Updates will be provided as soon as more information is available.

  2. investigating Mar 06, 2024, 05:06 AM UTC

    We are continuing to investigate this issue.

  3. identified Mar 06, 2024, 06:28 AM UTC

    The issue has been identified. We expect to restore normal service shortly.

  4. monitoring Mar 06, 2024, 07:04 AM UTC

    A fix has been implemented, and we are currently monitoring the results. Please be aware that intermittent outages may occur as our engineers continue to work on the system.

  5. identified Mar 06, 2024, 07:26 AM UTC

    The in-place fix did not resolve the issue. Our engineers are continuing to work on finding a viable alternative.

  6. monitoring Mar 06, 2024, 07:41 AM UTC

    A new fix has been implemented, and we are currently monitoring the results. Please be aware that intermittent outages may occur as our engineers continue to work on the system.

  7. monitoring Mar 06, 2024, 08:43 AM UTC

    The issue has been traced to a power outage in our main datacenter. Backup infrastructure has been activated, and services are mostly fully functional. However, please be advised that performance may be degraded, and some intermittent outages may occur while everything comes back to a stable state.

  8. resolved Mar 06, 2024, 09:47 AM UTC

    This incident has been resolved, and all systems are now operational. Please note that there may be some pending logs to be processed, so expect a delay in logs appearing as the systems process all pending information.