Reown incident

ENS API returning `derek.ens` for most addresses

Notice Resolved View vendor source →

Reown experienced a notice incident on November 20, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Nov 20, 2023, 02:45 PM UTC
Resolved
Nov 20, 2023, 02:44 PM UTC
Duration
Detected by Pingoru
Nov 20, 2023, 02:45 PM UTC

Update timeline

  1. resolved Nov 20, 2023, 02:45 PM UTC

    We deployed a faulty PR. We disregarded our integration tests and the feature is not tested. We will follow up with a postmortem.

  2. postmortem Nov 21, 2023, 11:10 AM UTC

    **TL;DR** On Nov 20 2023 from 1pm CET to 3pm CET our ENS resolver returned a static name. **Summary** Customers discovered the issue and operator initiated a swift rollback after the issue was discovered. **Root Cause** A bug was deployed to production. **5 Whys** _**Why was a bug deployed to production?**_ We occasionally make mistakes. We know this and hence and try to catch these in lower stages. _**Why was the bug not discovered in lower stages?**_ The name service feature was not covered in integration tests. _**Why was the feature not covered in integration tests?**_ We never prioritized. _**Why was this never prioritized?**_ The mix of effort and fun to write of the tests didn’t immediately check out. _**Why did it not check out?**_ The integration test suite is difficult to maintain. **What could we have done better?** * The feature was not covered by integration tests * Other integration tests were failing for a while not alerting operator to dig into what’s going on * The integration test suite was not fun/fast to work with * The rollback took a long time due to the images not being tagged * Discovery could have been quicker **Action items** 1. Write a more fun to work with integration test suite @Derek Rein ✅ 2. Cover the Name Service in integration tests @Derek Rein ✅ 3. Cover transaction history in integration tests @Max Kalashnikov ✅ 4. Make sure integration tests pass again @Max Kalashnikov ✅ 5. Deploy only tagged images @Max Kalashnikov ✅