Thanx Outage History

Started Mar 26, 2025, 12:19 AM UTC· resolved Mar 26, 2025, 01:18 AM UTC

1. investigating Mar 26, 2025, 12:19 AM UTC

   We're experiencing an outage with a downstream provider which is causing degraded reporting performance

2. resolved Mar 26, 2025, 01:18 AM UTC

   The downstream provider has resolved the issue and reporting performance has returned to a normal state.

Read the full incident report →

Started Mar 19, 2025, 05:46 PM UTC· resolved Mar 19, 2025, 06:17 PM UTC

1. investigating Mar 19, 2025, 05:46 PM UTC

   We are seeing errors from Mastercard card enrollment API affecting card enrollments. We are working on a solution to address the issue.

2. resolved Mar 19, 2025, 06:17 PM UTC

   The issue has been resolved.

Read the full incident report →

Started Mar 14, 2025, 06:47 PM UTC· resolved Mar 14, 2025, 07:58 PM UTC

1. investigating Mar 14, 2025, 06:47 PM UTC

   We are investigating reports of issues with reward redemption impacting Toast and Olo merchants online and in-store. We will provide prompt updates with more information soon.

2. monitoring Mar 14, 2025, 07:37 PM UTC

   Reward redemption is now fully operational. We are continuing to monitor this to ensure there are no further issues.

3. resolved Mar 14, 2025, 07:58 PM UTC

   This has been resolved.

Read the full incident report →

Started Mar 14, 2025, 12:47 AM UTC· resolved Mar 19, 2025, 04:01 PM UTC

1. investigating Mar 14, 2025, 12:47 AM UTC

   We've identified a delay in our dashboard reporting and campaign targeting. This appears to be an issue with AWS infrastructure. We're currently working with AWS's technical team to resolve.

2. identified Mar 14, 2025, 05:33 AM UTC

   We've identified the root issue and are working on a resolution.

3. identified Mar 14, 2025, 10:07 AM UTC

   In collaboration with the AWS team, we've that the issue is due to the degradation of the hardware of a key replication database. We are actively working on system restoration. Due to the data scale of the underlying system, data migration is expected to take until EOD Friday.

4. identified Mar 17, 2025, 06:36 PM UTC

   In collaboration with the AWS team, we've made the changes necessary to our key replication database and are seeing our platform recover. We are closely monitoring its recovery and in parallel also exploring way to further speed up its recovery time if it is technically possible.

5. monitoring Mar 18, 2025, 01:21 AM UTC

   Our team is in the final stages of recovering full reporting and targeting functionality. We will provide updates periodically as we get closer to full resolution.

6. resolved Mar 19, 2025, 04:01 PM UTC

   Our database replication has fully caught up and all our reporting data is up to date.

Read the full incident report →

Started Mar 05, 2025, 01:03 AM UTC· resolved Mar 05, 2025, 04:10 PM UTC

1. investigating Mar 05, 2025, 01:03 AM UTC

   We are seeing an elevated number of errors from American Express APIs affecting card enrollments. We are investigating this issue with American Express.

2. resolved Mar 05, 2025, 04:10 PM UTC

   The issue has been resolved.

Read the full incident report →

Started Feb 28, 2025, 04:19 PM UTC· resolved Feb 28, 2025, 08:33 PM UTC

1. investigating Feb 28, 2025, 04:19 PM UTC

   We're experiencing degraded performance in our merchant experience and are actively investigating

2. identified Feb 28, 2025, 04:58 PM UTC

   We have identified the root cause as long running APIs to our data provider. We are working on a fix to address these API calls

3. monitoring Feb 28, 2025, 05:33 PM UTC

   We've rolled out a fix and performance is looking healthy. Continuing to monitor

4. resolved Feb 28, 2025, 08:33 PM UTC

   The service has recovered and the issue has been resolved

Read the full incident report →

Started Jan 25, 2025, 11:34 PM UTC· resolved Jan 26, 2025, 01:27 AM UTC

1. investigating Jan 25, 2025, 11:34 PM UTC

   We are seeing an elevated number of errors from American Express APIs affecting card enrollments. We are investigating this issue with American Express.

2. resolved Jan 26, 2025, 01:27 AM UTC

   The issue has been resolved.

Read the full incident report →

Started Dec 02, 2024, 06:49 PM UTC· resolved Dec 03, 2024, 03:16 PM UTC

1. investigating Dec 02, 2024, 06:49 PM UTC

   We have identified a delay in data reporting synchronization since November 26, 2024. We are currently investigating the issue and will provide updates as soon as possible.

2. monitoring Dec 02, 2024, 08:33 PM UTC

   We have addressed the issue and are seeing latency recovering. We will continue monitoring it and provide updates as soon as possible.

3. monitoring Dec 02, 2024, 11:14 PM UTC

   We are continuing to monitor the issue as it recovers historical data.

4. resolved Dec 03, 2024, 03:16 PM UTC

   Historical data has been fully recovered. We will continue to enhance our platform to ensure stability.

Read the full incident report →

Started Nov 22, 2024, 07:04 PM UTC· resolved Nov 22, 2024, 09:52 PM UTC

1. investigating Nov 22, 2024, 07:04 PM UTC

   We are seeing an elevated number of errors from American Express APIs affecting card enrollments. We are investigating this issue with American Express.

2. resolved Nov 22, 2024, 09:52 PM UTC

   The issue has been resolved.

Read the full incident report →

Started Nov 15, 2024, 05:20 PM UTC· resolved Nov 15, 2024, 06:05 PM UTC

1. identified Nov 15, 2024, 05:20 PM UTC

   We're experiencing intermittent issues on our ordering platforms. We have identified the issue with one of our downstream providers and are working on restoring our ordering service.

2. identified Nov 15, 2024, 05:48 PM UTC

   We've restored ordering services for merchants on Toast ordering.

3. resolved Nov 15, 2024, 06:05 PM UTC

   The ordering service has been restored.

Read the full incident report →