Whereby incident

Cloud recordings stop unexpectedly

Minor Resolved View vendor source →

Whereby experienced a minor incident on February 29, 2024, lasting —. The incident has been resolved; the full update timeline is below.

Started
Feb 29, 2024, 10:35 AM UTC
Resolved
Feb 29, 2024, 10:35 AM UTC
Duration
Detected by Pingoru
Feb 29, 2024, 10:35 AM UTC

Update timeline

  1. resolved Feb 29, 2024, 10:35 AM UTC

    Type: Incident Affected Components: Embedded API, Meeting rooms Feb 29, 10:35:55 GMT+0 - Resolved - We are noticing that cloud recordings end prematurely, roughly after 5 mins. We are deploying a potential fix. Feb 29, 11:47:21 GMT+0 - Resolved - An update to the underlying OS/runtime of our recording service caused recordings to crash after a few minutes. This crash was sadly not detected by our automated nor manual tests, which are always run before such deployments. The reason why we did not pick this up was the error only presented itself after a certain time, which is outside the time we have encoded in for our test runs. We have now rolled back the change and the feature should be working as expected. We are yet to find the root cause of the issue, and have scheduled time to identify what went wrong, and also discuss how our test setup can be adjusted to catch similar regressions in the future.