Layer27 experienced a notice incident on September 18, 2014, lasting 3h 15m. The incident has been resolved; the full update timeline is below.
Update timeline
- monitoring Sep 18, 2014, 02:25 PM UTC
We were alerted to a latency issue on our side and have discovered an issue affecting one of our outbound internet connections. We have traced the issue and quarantine for now. We are working through root cause analysis and will update once we have more information. For now, the issue has be cleared up, but RCA is not complete.
- resolved Sep 18, 2014, 05:41 PM UTC
The POD01 internet latency issue was resolved and corrected this morning and no additional flags have surfaced. The latency period appears to have started at approximately 9:20AM EDT and was completely remediated by 10:04AM EDT.
- postmortem Aug 02, 2018, 04:37 PM UTC
# Latency Issue Postmortem Date: 09/18/2014 - 09:25AM to 10:04AM EDT ## Issue Description Latency was found on a particular upstream path to the Internet. Some Customers may have experience slow connectivity to the POD01 cloud environment. ## Cause Through troubleshooting, it was discovered that a secondary firewall/VPN appliance began to drop broadcast packets onto the network and flood the affected firewalls ingest port. No configuration changes were applied, and so the issue appears to have been a bug/issue. ## Resolution The VPN tunnel causing the broadcast packet drops to occur was terminated and isolated into a separate environment for monitoring. The VPN was torn down and recreated. We are currently working with the manufacturer to gain more clarity on the issue. If additional maintenance or configuration is required, affected customers will be notified of a scheduled maintenance window.