PreVeil incident

Increased backend CPU usage and increased timeouts when connecting to the servers

Minor Resolved View vendor source →

PreVeil experienced a minor incident on November 4, 2019 affecting Backend and Admin Console and 1 more component, lasting 3h 33m. The incident has been resolved; the full update timeline is below.

Started
Nov 04, 2019, 03:24 PM UTC
Resolved
Nov 04, 2019, 06:58 PM UTC
Duration
3h 33m
Detected by Pingoru
Nov 04, 2019, 03:24 PM UTC

Affected components

BackendAdmin ConsoleDriveMail

Update timeline

  1. investigating Nov 04, 2019, 03:24 PM UTC

    Since about 9am this morning, we've been seeing a very high CPU load on the servers. In the same timespan, we've seen an increase in errors, some of which are timeouts when connecting to the backend. It is unclear how many of our customers are affected, and we are continuing to investigate the root cause.

  2. investigating Nov 04, 2019, 03:57 PM UTC

    We are continuing to investigate this issue; some users may experience performance issues in the browser app.

  3. monitoring Nov 04, 2019, 06:34 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Nov 04, 2019, 06:58 PM UTC

    This incident has been resolved.