Resolved after 4 hours 42 minutes, affecting Platform, on February 2, 2025.
We are continuing to see superior performance across the board. We will continue to closely monitor the database upgrade. We can confirm total downtime for the database upgrade was about 2 minutes. Given we brought the database upgrade forward we will be cancelling the planned maintenance tonight. We are sorry for the disruption to your day and will share a post mortem at a later time.
Affected components
Update timeline
- investigating Feb 02, 2025, 11:51 PM UTC
We're aware of a high volume of timeouts causing disruption to a variety of platform behaviour. Active investigation is underway.
- identified Feb 03, 2025, 12:21 AM UTC
We identified a database query causing undue stress on the primary database. We're actively managing it's impact. Reports, exports, and notifications have been delayed, but are starting to be processed again.
- identified Feb 03, 2025, 12:52 AM UTC
Reports, exports, and various finance processes will continue to experience considerable delays as we monitor platform load.
- identified Feb 03, 2025, 01:55 AM UTC
We've provisioned additional resources to process the reporting backlog. Reporting delays are being managed between 5-10 minutes.
- identified Feb 03, 2025, 02:38 AM UTC
The incident team has decided to bring forward the planned database upgrade that was scheduled for later this evening, to sustain the increased traffic. We anticipate there will be disruption to the platform for approximately ten minutes during the upgrade. We will commence the upgrade at 12:45PM AEST. We expect platform functionality to return to normal within minutes following the upgrade.
- identified Feb 03, 2025, 02:45 AM UTC
We have commenced the database upgrade.
- monitoring Feb 03, 2025, 02:53 AM UTC
The database upgrade has been successfully performed and we are closely monitoring the application.
- monitoring Feb 03, 2025, 03:09 AM UTC
We are continuing to see application performance return to normal levels across the board. We will continue to closely monitor the database upgrade.
- monitoring Feb 03, 2025, 03:46 AM UTC
We are continuing to see superior performance across the board. We will continue to closely monitor the database upgrade. We can confirm total downtime for the database upgrade was about 2 minutes. Given we brought the database upgrade forward we will be cancelling the planned maintenance tonight. We are sorry for the disruption to your day and will share a post mortem at a later time.
- resolved Feb 03, 2025, 04:33 AM UTC
We're satisfied with platform recovery post-upgrade.
Your product runs on more than The Lookout Way
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services