The Lookout Way Outage History

Started Jun 10, 2025, 06:45 AM UTC· resolved Jun 11, 2025, 06:44 AM UTC

1. investigating Jun 10, 2025, 06:45 AM UTC

   Our monitoring has alerted us to a consistent, low volume of request timeouts affecting varied platform clients and pages. We're actively investigating.

2. investigating Jun 10, 2025, 07:47 AM UTC

   We are continuing to investigate the issue.

3. identified Jun 10, 2025, 08:18 AM UTC

   There's an issue with our platform provider. We're continuing to investigate.

4. identified Jun 10, 2025, 09:14 AM UTC

   No updates from our provider as of yet. We are continuing to investigate.

5. identified Jun 10, 2025, 10:17 AM UTC

   Our platform provider has identified an issue and is currently investigating.

6. identified Jun 10, 2025, 11:14 AM UTC

   We are still seeing < 1% of requests timeout due to a problem identified by our platform provider. If you experience a timeout you can safely try again as 99% of requests are performing normally. They continue to investigate.

7. identified Jun 10, 2025, 11:53 AM UTC

   We saw ~15% of requests timeout due to a problem identified by our platform provider between 9:16PM AEST and 9:48PM AEST. We are now seeing < 1% of requests timeout. They continue to investigate.

8. identified Jun 10, 2025, 12:09 PM UTC

   Our platform provider is continuing to investigate. Next update will be at 6am AEST.

9. identified Jun 10, 2025, 08:03 PM UTC

   Our platform provider is still looking into the issue but has restored some functionality.

10. monitoring Jun 10, 2025, 09:21 PM UTC

    We are now in a position to provision resourcing again. We have made an operational change and are seeing performance return to target levels. Some background jobs, such as sending emails and notifications, may have been delayed but we are also seeing performance return to target levels here too. We are continuing to work with our platform provider who is experiencing a major outage. Therefore, while our site is up, we continue to monitor. The latest from our platform provider: "We don’t have additional information to share at this time. Our efforts remain focused on internal testing and validation as we continue to see incremental improvements. We will provide a resolution timeline as soon as possible, and we apologize for the continued trouble.."

11. monitoring Jun 10, 2025, 11:10 PM UTC

    Our monitoring continues to show an operational Lookout platform. We are also observing encouraging signs of recovery with our upstream infrastructure provider. We will continue to monitor.

12. resolved Jun 11, 2025, 06:44 AM UTC

    This incident has been resolved.

Read the full incident report →

Started May 19, 2025, 09:18 PM UTC· resolved May 19, 2025, 10:02 PM UTC

1. investigating May 19, 2025, 09:18 PM UTC

   We're aware of an issue affecting email notifications delivered by the platform and are investigating.

2. identified May 19, 2025, 09:31 PM UTC

   There's an issue with our email delivery provider. We're continuing to investigate.

3. monitoring May 19, 2025, 09:52 PM UTC

   Our provider has implemented a fix. We're observing successful email deliveries. We're continuing to monitor.

4. resolved May 19, 2025, 10:02 PM UTC

   We've seen no failed email deliveries since 7:28am (AEST). This incident is resolved.

Read the full incident report →

Started May 19, 2025, 04:02 AM UTC· resolved May 19, 2025, 05:17 AM UTC

1. identified May 19, 2025, 04:02 AM UTC

   Essential dashboards and custom dashboards in Analytics are currently showing data as of 8AM AEST this morning. We currently schedule the refresh of data every 2 hours. There was an issue which prevented the data from being refreshed. We have fixed the issue and are currently refreshing the data. At this stage we expect the refresh to complete at about 2:30PM AEST. We will update here.

2. resolved May 19, 2025, 05:17 AM UTC

   The refresh of data successfully completed at 3:00PM AEST. The next refresh is scheduled to commence at 4:00PM AEST and takes about 47 minutes to complete.

Read the full incident report →

Started May 13, 2025, 04:55 AM UTC· resolved May 13, 2025, 05:04 AM UTC

1. identified May 13, 2025, 04:55 AM UTC

   An issue delaying the delivery of webhooks has been identified.

2. monitoring May 13, 2025, 04:56 AM UTC

   Additional infrastructure has been introduced to help clear the backlog.

3. resolved May 13, 2025, 05:04 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 17, 2025, 04:36 AM UTC· resolved Apr 17, 2025, 06:00 AM UTC

1. investigating Apr 17, 2025, 04:36 AM UTC

   We are investigating a delay in notifications and reminders.

2. identified Apr 17, 2025, 04:46 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Apr 17, 2025, 06:00 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 20, 2025, 04:02 AM UTC· resolved Mar 20, 2025, 06:16 AM UTC

1. investigating Mar 20, 2025, 04:02 AM UTC

   We're aware of a contained issue affecting SCIM management (single sign-on related), and are actively investigating. At this stage impact to the platform appears minor.

2. monitoring Mar 20, 2025, 04:28 AM UTC

   We've identified the issue as affecting a single customer, and have applied a mitigation to protect platform functionality. We're continuing to monitor.

3. resolved Mar 20, 2025, 06:16 AM UTC

   The mitigation remains in place while we continue to seek performance improvements for the affected customer.

Read the full incident report →

Started Mar 18, 2025, 01:47 AM UTC· resolved Mar 18, 2025, 01:58 AM UTC

1. monitoring Mar 18, 2025, 01:47 AM UTC

   We're aware of an issue affecting Care app users. The cause has been identified, and the fix has already being applied. We're monitoring the recovery.

2. resolved Mar 18, 2025, 01:58 AM UTC

   The fix was successful.

Read the full incident report →

Started Mar 04, 2025, 03:01 AM UTC· resolved Mar 13, 2025, 06:36 AM UTC

1. monitoring Mar 04, 2025, 03:01 AM UTC

   We have been tracking Cyclone Alfred since last week and have made the decision to activate our business continuity plan. We have designed our business to withstand these kinds of events so Lookout can remain highly available to you. Our infrastructure is in Sydney, with redundant backups in Melbourne. We do not anticipate any disruption to your instance of Lookout. You can remain focused on your clients and workers. Our staff are distributed around the country with redundancies across key roles. We have made plans to accomodate staff who are travelling to, or working from the affected area to ensure their safety. We do not anticipate any disruption to inbound support, but will reach out to stakeholders of projects where planned activities may be disrupted. Should Brisbane and the South-East Queensland region be significantly impacted, we have clear lines of responsibility in place for the ongoing management and performance of our business. For any support enquiries, please use our official inbound channels: - [email protected] - https://support.thelookoutway.com/en/ We remain committed to providing highly available service and will continue to keep you informed as the situation evolves in this incident.

2. monitoring Mar 10, 2025, 01:24 AM UTC

   First reports this morning indicate our staff are safe. Some are experiencing power outages, internet outages, minor flooding, or have moved to safer locations. We are grateful for this outcome. Our platform has continued to remain highly available and supported. The weather event has not yet finished so we'll keep this incident open to continue providing updates.

3. monitoring Mar 13, 2025, 06:36 AM UTC

   All our staff are safe. The weather event has now finished.

4. resolved Mar 13, 2025, 06:36 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 19, 2025, 01:09 AM UTC· resolved Feb 19, 2025, 01:22 AM UTC

1. identified Feb 19, 2025, 01:09 AM UTC

   We're aware of any issue with our published Lookout Platform API documentation. It's currently unavailable as the provider is experiencing an issue. Unfortunately this also affects creating or updating recipes in Workato (but existing recipes remain unaffected).

2. resolved Feb 19, 2025, 01:22 AM UTC

   The provider's issue has been resolved. API documentation is available again.

Read the full incident report →

Started Feb 19, 2025, 12:13 AM UTC· resolved Feb 19, 2025, 01:09 AM UTC

1. investigating Feb 19, 2025, 12:13 AM UTC

   We are currently investigating this issue.

2. investigating Feb 19, 2025, 12:44 AM UTC

   We believe we have found the source of the issue and are deploying a remediation.

3. identified Feb 19, 2025, 12:44 AM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Feb 19, 2025, 12:56 AM UTC

   The fix has been deployed. We're continuing to monitor.

5. resolved Feb 19, 2025, 01:09 AM UTC

   The deployed fix is effective and failed attempts to calculate shifts have been successfully retried. The incident is resolved.

Read the full incident report →

Started Feb 14, 2025, 02:40 AM UTC· resolved Feb 14, 2025, 05:50 AM UTC

1. investigating Feb 14, 2025, 02:40 AM UTC

   Reports and data exports are currently taking a long time to generate. We are looking into the issue.

2. monitoring Feb 14, 2025, 02:53 AM UTC

   We have increased the resources available and the report queue is now empty, though some long running reports remain. We are monitoring the system.

3. resolved Feb 14, 2025, 05:50 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 03, 2025, 10:35 AM UTC· resolved Feb 03, 2025, 12:04 PM UTC

1. investigating Feb 03, 2025, 10:35 AM UTC

   We are aware of an issue affecting the "Helpers" and "Scheduling" page in some instances. Our on-call engineer has been paged and is responding. We do not have an ETA or work around at this stage. The rest of the platform continues to operate normally.

2. investigating Feb 03, 2025, 11:05 AM UTC

   We are still investigating the issue.

3. identified Feb 03, 2025, 11:26 AM UTC

   We have identified the issue and are deploying a fix. We expect service to return to normal in about 20 minutes.

4. monitoring Feb 03, 2025, 11:45 AM UTC

   Service has returned to normal. We will continue to monitor.

5. resolved Feb 03, 2025, 12:04 PM UTC

   The issue is now resolved and service is normal.

Read the full incident report →

Started Feb 02, 2025, 11:51 PM UTC· resolved Feb 03, 2025, 04:33 AM UTC

1. investigating Feb 02, 2025, 11:51 PM UTC

   We're aware of a high volume of timeouts causing disruption to a variety of platform behaviour. Active investigation is underway.

2. identified Feb 03, 2025, 12:21 AM UTC

   We identified a database query causing undue stress on the primary database. We're actively managing it's impact. Reports, exports, and notifications have been delayed, but are starting to be processed again.

3. identified Feb 03, 2025, 12:52 AM UTC

   Reports, exports, and various finance processes will continue to experience considerable delays as we monitor platform load.

4. identified Feb 03, 2025, 01:55 AM UTC

   We've provisioned additional resources to process the reporting backlog. Reporting delays are being managed between 5-10 minutes.

5. identified Feb 03, 2025, 02:38 AM UTC

   The incident team has decided to bring forward the planned database upgrade that was scheduled for later this evening, to sustain the increased traffic. We anticipate there will be disruption to the platform for approximately ten minutes during the upgrade. We will commence the upgrade at 12:45PM AEST. We expect platform functionality to return to normal within minutes following the upgrade.

6. identified Feb 03, 2025, 02:45 AM UTC

   We have commenced the database upgrade.

7. monitoring Feb 03, 2025, 02:53 AM UTC

   The database upgrade has been successfully performed and we are closely monitoring the application.

8. monitoring Feb 03, 2025, 03:09 AM UTC

   We are continuing to see application performance return to normal levels across the board. We will continue to closely monitor the database upgrade.

9. monitoring Feb 03, 2025, 03:46 AM UTC

   We are continuing to see superior performance across the board. We will continue to closely monitor the database upgrade. We can confirm total downtime for the database upgrade was about 2 minutes. Given we brought the database upgrade forward we will be cancelling the planned maintenance tonight. We are sorry for the disruption to your day and will share a post mortem at a later time.

10. resolved Feb 03, 2025, 04:33 AM UTC

    We're satisfied with platform recovery post-upgrade.

Read the full incident report →

Started Dec 02, 2024, 01:30 AM UTC· resolved Dec 02, 2024, 06:13 AM UTC

1. identified Dec 02, 2024, 01:30 AM UTC

   We're aware of performance issues regarding an area of the Admin dashboard. We've temporarily disabled the Lookout Connect dashboard widgets while we investigate improvements.

2. resolved Dec 02, 2024, 06:13 AM UTC

   The Lookout Connect dashboard widgets will remain disabled while we continue to pursue performance improvements. Considering this incident resolved. Admin dashboard behaviour has otherwise returned to usual.

Read the full incident report →

Started Nov 11, 2024, 04:17 AM UTC· resolved Nov 13, 2024, 12:00 AM UTC

1. investigating Nov 11, 2024, 04:17 AM UTC

   This affects customers using the Lookout Digital Agreements add-on powered by BoldSign. All open agreements been revoked due to a configuration change we initiated. As a workaround agreements can be re-issued manually, but we are currently investigating how we can automatically re-open all agreements again.

2. investigating Nov 11, 2024, 06:49 AM UTC

   We are still compiling a list of impacted agreements that need to be reviewed and resent. We will be in touch once we have this information. New agreements sent after 1:20p.m. AEST today (11 November 2024) are unaffected.

3. resolved Nov 13, 2024, 12:00 AM UTC

   Unfortunately it doesn't appear we can automatically re-activate the revoked agreements from 11 November 1:20PM AEST. We have reached out to affected people to support them with re-activating agreements. If you come across a revoked agreement in the interim, it will need to be cancelled from the ticket and a new agreement created. Any agreements created after 11 November 1:20PM AEST are unaffected and will continue to work as normal. We're sorry about this.

Read the full incident report →