LuxSci incident
Some Businss Class Servers in RackSpace DFW Inaccessible
LuxSci experienced a minor incident on August 10, 2020 affecting Shared WebMail and SecureForm and 1 more component, lasting 1h 52m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 10, 2020, 09:59 AM UTC
We are currently experiencing and issue where Rackspace-based Business Class servers are inaccessible due to a networking at at Rackspace. This affects servers in Dallas, Texas. We are currently escalating with Rackspace for a resolution and ETA.
- identified Aug 10, 2020, 09:59 AM UTC
We are continuing to work on a fix for this issue.
- identified Aug 10, 2020, 10:33 AM UTC
This incident impacts only some of the Business Class servers in Rackspace, DFW. Customers with email hosting and without Premium email filtering (i.e., Proofpoint) will be experiencing delayed delivery of new email. Additionally, for customers using AWS-based business class servers, outbound email will be queued on your server until this issue is resolved, as that email is relayed out through affected Rackspace servers.
- identified Aug 10, 2020, 11:07 AM UTC
The issue has been identified as an issue with an upstream aggregation router in the Rackspace general infrastructure that is affecting the traffic routing to/from these servers. This issue impacts a wide band of Rackspace customers. Their network engineering team is actively engaged in resolving the issue.
- resolved Aug 10, 2020, 11:51 AM UTC
Rackspace has resolved the issue with the networking as of 7:33am ET. The root cause of the issue was a problem with an update in a general backbone aggregation router which serves a large section of the Rackspace infrastructure. It took them some time to resolve that maintenance error. Generally, this would not have been a service impacting event; however, it seems that a Rackspace networking issue on Thursday left part of their infrastructure in a state of reduced redundancy ... so when this issue occurred, the normally redundant routing could not compensate. it is not clear from my conversations with Rackspace, but I would imagine the two issues were related.