Tideways incident

MySQL master is being migrated to new host

Resolved after 8 hours 20 minutes, affecting Ingest / Data Collection and User Interface, on May 28, 2024.

resolved May 29, 2024, 04:51 AM UTC

The backlog is resolved and we will work on updating you with more information about what happened and once our workday starts in Europe/Berlin timezone.

Started
May 28, 2024, 08:31 PM UTC
Resolved
May 29, 2024, 04:51 AM UTC
Duration
8h 20m
Detected by Pingoru
May 28, 2024, 08:31 PM UTC

Affected components

Ingest / Data CollectionUser Interface

Update timeline

  1. investigating May 28, 2024, 08:31 PM UTC

    Our hosting company is moving the MySQL database master to another host and the web-interface is not reachable. We are investigating how this impacts data ingestion.

  2. investigating May 28, 2024, 09:22 PM UTC

    Web Interface is reachable again, but data ingestion is currently on hold as worker servers are being migrated to other hosts. Data is accepted by our servers and put into a queue. We are working with our hoster to get enough servers online that processing can continue in a degraded state.

  3. investigating May 28, 2024, 09:39 PM UTC

    Processing is now continuing at a degraded level, we are processing data based on priority of the organization's plans and expect to reach normal levels in about an hour.

  4. investigating May 28, 2024, 10:09 PM UTC

    Monitoring data is now up to date, traces are still being processed and while take a while longer until they are not lagging behind anymore.

  5. monitoring May 29, 2024, 12:40 AM UTC

    Traces are still being processed, the backlog has shrunk to 25% of its original peak, expect the proccesing to take 30-60 minutes.

  6. resolved May 29, 2024, 04:51 AM UTC

    The backlog is resolved and we will work on updating you with more information about what happened and once our workday starts in Europe/Berlin timezone.

Your product runs on more than Tideways

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →