LuxSci experienced a major incident on September 24, 2019 affecting LuxSci Network Infrastructure, lasting 3h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 24, 2019, 08:27 PM UTC
We are investigating an issue with LuxSci server rs160.luxsci.com. We will provide updates here as more information becomes available.
- investigating Sep 24, 2019, 09:23 PM UTC
We are still investigating the issue with LuxSci email server rs160.luxsci.com at this time. We will provide further information as it becomes available.
- investigating Sep 24, 2019, 09:29 PM UTC
The cause of the downtime on this shared email server, rs160, has been identified as some file system corruption on the main customer data drive. We can not re-boot the server in a way where this drive can be mounted properly on the server and made accessible. The current solution we are pursing is to start the server in recovery mode and run a repair on the drive to fix any issues. Once that is complete we will once more try to reboot the server and get it back online. The downtime expected for repairing a 1200 GB drive can not be clearly predicted ... it could be several hours. We will update this status message once the disk repair is complete and we attempt to reboot the server once more. We anticipate that this will resolve the issue. If for some reason it does not, we have backup solutions for restoring services. While the down time persists, inbound email is being queued for you on LuxSci's inbound servers. This inbound email will be automatically unqueued and delivered to your server once the downtime is over.
- identified Sep 24, 2019, 09:30 PM UTC
Status updated to "issue identified."
- resolved Sep 24, 2019, 11:29 PM UTC
At this time, the issue with LuxSci email server rs160.luxsci.com is resolved. Any inbound email received during this downtime is now being delivered. We're still investigating the root cause of this issue and will update this post with any further information as we get it.