LuxSci incident

Issue With LuxSci Web Server rs125.luxsci.com

Critical Resolved View vendor source →

LuxSci experienced a critical incident on November 6, 2019 affecting Hosted Web Services, lasting 1h 22m. The incident has been resolved; the full update timeline is below.

Started
Nov 06, 2019, 05:38 PM UTC
Resolved
Nov 06, 2019, 07:00 PM UTC
Duration
1h 22m
Detected by Pingoru
Nov 06, 2019, 05:38 PM UTC

Affected components

Hosted Web Services

Update timeline

  1. investigating Nov 06, 2019, 05:38 PM UTC

    We are currently investigating an issue with LuxSci web server rs125.luxsci.com. Web sites on this server may be appearing down at this time. We will update this post when more information becomes available.

  2. identified Nov 06, 2019, 06:11 PM UTC

    We have identified an issue with the database associated with LuxSci Web Sever rs125.luxsci.com as the root cause. We are in the process of resolving the issue at this time. We will provide further updates as information becomes available.

  3. identified Nov 06, 2019, 06:40 PM UTC

    We are still working to resolve this issue. We will provide more information as it becomes available.

  4. resolved Nov 06, 2019, 07:00 PM UTC

    We have resolved the issue with LuxSci Web Server rs125.luxsci.com. We will update this post with more detailed information as to the cause when it becomes available.

  5. postmortem Nov 06, 2019, 07:16 PM UTC

    We have determined that there was a permissions issue on the file system that was preventing the database server from running properly on this server. This permissions issue has been resolved and we have added assertions into our software to automatically detect and resolve that issue should it ever arise in the future.