YourCloudTelco incident

Outage

Major Resolved View vendor source →

YourCloudTelco experienced a major incident on January 4, 2023 affecting YourCloudTelco Calling Platform (Network) and Arena and 1 more component, lasting 17m. The incident has been resolved; the full update timeline is below.

Started
Jan 04, 2023, 12:38 AM UTC
Resolved
Jan 04, 2023, 12:56 AM UTC
Duration
17m
Detected by Pingoru
Jan 04, 2023, 12:38 AM UTC

Affected components

YourCloudTelco Calling Platform (Network)ArenaYourCloudTelco Management CAPYourCloudPBX

Update timeline

  1. investigating Jan 04, 2023, 12:38 AM UTC

    an error has occured on a core database server. We have identified the issue and are working to resolve,

  2. resolved Jan 04, 2023, 12:56 AM UTC

    This incident has been resolved. A core dB crashed and failed to automatically fail to its redundant High Availability (HA) database. We are working to resolve the underlying HA failure.

  3. postmortem Jan 12, 2023, 12:05 AM UTC

    The immediate cause of this issue was a database failure and a faulty database proxy service which should typically have failed the failed system to its redundant host. In reviewing this outage, we have identified both the cause of the initial database failure and secondly the proxy failure. ‌ **Addendum** As result of this outage we lost approximately four days of detailed billing records \(referred to as CDR records\). Unfortunately these records were irretrievable. Mike Johnstone