LimaCharlie incident

DNS issues

LimaCharlie experienced a minor incident on August 29, 2024 affecting Web Endpoints (limacharlie.io) and Web Endpoints (app.limacharlie.io) and 1 more component, lasting 10d 17h. The incident has been resolved; the full update timeline is below.

Started: Aug 29, 2024, 09:05 PM UTC
Resolved: Sep 09, 2024, 02:07 PM UTC
Duration: 10d 17h
Detected by Pingoru: Aug 29, 2024, 09:05 PM UTC

Affected components

Web Endpoints (limacharlie.io)Web Endpoints (app.limacharlie.io)Web Endpoints (jwt.limacharlie.io)API Canada (API Canada Telemetry Ingestion)API Canada (API Canada Insight)API Canada (API Canada External Telemetry)API Canada (API Canada Cloud Adapters)API Canada (API Canada Services)API Canada (API Canada Enrollment)API Canada (API Canada Management)

Update timeline

investigating Aug 29, 2024, 09:05 PM UTC

The limacharlie.io domain experienced DNS issues which caused unavailability of services.
resolved Sep 09, 2024, 02:07 PM UTC

Summary Root Cause Analysis Report - Post-Mortem Incident Overview: On August 29 2024 the limacharlie.io domain experienced DNS issues which caused unavailability of services. This report summarizes the timeline of events, root causes, and corrective actions taken. Timeline of the Events: August 29, 2024 - 2:05pm PT: DNS issues began. - 2:10pm PT: Reports received of customer receiving errors and no longer being able to access orgs. - 2:21pm PT: Customers were notified via community Slack that a DNS issue existed and the LimaCharlie team was working on a fix. There was a phone call directly to our upstream provider to help remediate the issue. The issue was escalated to upstream provider management and was resolved. - 2:30pm PT: Customers were notified via community slack that the DNS issue had been resolved and that services would be back to normal within 15 minutes. Note that DNS propagation times at local nameservers impacted when customers saw full resolution. - 3:38pm PT: Confirmed full resolution at all global root nameservers. Root Cause: 1. Human Error at upstream provider: LimaCharlie had requested a change with an upstream provider which required manual intervention by the provider. Unfortunately the specific instructions provided were not followed which caused DNS nameservers to change. Upon finding the issue the nameservers were reverted to their original state which corrected the issue. 2. System Status: All Underlying services remained up and running. Domain name resolution was not completing as expected due to incorrect nameservers being set. Actions Take: - Vendor Communication: The LimaCharlie team was able to communicate with the upstream provider and quickly remediate the issue. Lessons Learned / Future Recommendations: - Improve internal communication for backend operations changes, ensuring monitoring for deviations and expected outcomes are implemented. - Request written confirmation of proposed manual changes by upstream providers prior to execution.