Think Lumo incident
Database performance issues
Resolved after 4 hours 13 minutes, on February 17, 2024.
Confirming that we are back to BAU. We will be sending our customers a Root Cause Analysis in the coming days as soon as we have confirmation from AWS. If you have any questions about this incident, please reach out to [email protected].
Update timeline
- monitoring Feb 17, 2024, 11:05 AM UTC
The upgrade to the database went smoothly but we are observing poor performance from the database which is leading to a slowdown in alerts. We are investigating and have raised the issue with AWS.
- monitoring Feb 17, 2024, 12:47 PM UTC
AWS has provided some potential solutions to the issue which involve restarting the database, which may result in downtime of around 10 minutes.
- monitoring Feb 17, 2024, 01:11 PM UTC
We are seeing some improvement in db performance but still not back to normal. We are still working with AWS to identify the root cause.
- monitoring Feb 17, 2024, 02:17 PM UTC
We are adding a few more replica databases that will likely ease the load temporarily so we can return to service faster.
- monitoring Feb 17, 2024, 02:51 PM UTC
Adding new databases seems to have helped significantly. We should see alerts start to flow soon.
- monitoring Feb 17, 2024, 03:06 PM UTC
We can confirm alerts are flowing again. We will continue to monitor the metrics and confirm once we are back to BAU.
- monitoring Feb 17, 2024, 03:20 PM UTC
Confirming that we are back to BAU. We will be sending our customers a Root Cause Analysis in the coming days as soon as we have confirmation from AWS. If you have any questions about this incident, please reach out to [email protected].
Your product runs on more than Think Lumo
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services