Lumigo incident
Node.js Lambda functions failing in eu-west-1
Lumigo experienced a critical incident on March 14, 2023 affecting Tracer, lasting 7h 53m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 14, 2023, 10:52 AM UTC
CRITICAL: Lambda functions in eu-west-1 with the Lumigo Node.js tracer are failing. To fix this, please update to the latest version of the Lumigo Node.js tracer ASAP. As a precaution, we strongly advise updating the Node.js tracer in your Lambda functions in all regions. For assistance and/or guidance in updating your functions, please reach out immediately to [email protected]. At approximately 01:06 PM PST (GMT-7) on 13/03, the format of the AWS Lambda environment variable X-AMZN-TRACE-ID changed in AWS region eu-west-1. This is causing Lambda functions in this region with the Lumigo Node.js tracer installed to fail. - This issue only affects functions in eu-west-1 - This issue only affects functions with the Lumigo Node.js tracer installed. - The fix has been implemented at 04:16 PM PST (GMT-7) 13/03, and is included in tracer version 1.80.1 or newer.
- identified Mar 14, 2023, 12:15 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Mar 14, 2023, 04:16 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 14, 2023, 06:45 PM UTC
This incident has been resolved. A fix has been deployed, tested and verified. We advise any customer who has not yet updated their Node.js tracer version to 1.80.1 or above, to do so with urgency.