Tremendous Outage History

Type: Incident Duration: 1 hour and 27 minutes Affected Components: Reward redemptions Mar 10, 21:24:00 GMT+0 - Identified - Recipients were unable to redeem their rewards during this timeframe. Mar 10, 22:51:00 GMT+0 - Resolved - We identified a change in our bot detection as the root cause for this incident, and have rolled back the change. We apologize for this incident. Recipients can now retry their redemption.

Type: Incident Duration: 1 hour and 3 minutes Affected Components: Reward redemptions, Dashboard, API Mar 2, 16:25:00 GMT+0 - Investigating - The website and the API are unresponsive; we're investigating the cause. Mar 2, 17:19:54 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 2, 17:28:29 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 5 hours and 18 minutes Affected Components: Reward redemptions Jan 30, 09:30:00 GMT+0 - Identified - Prepaid card images aren't loading for a small subset of recipients. Jan 30, 14:48:00 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 day, 13 hours and 28 minutes Affected Components: Reward redemptions Dec 19, 00:38:48 GMT+0 - Monitoring - Venmo has fixed the issue and we have released all delayed payouts. We are still working on refunding the incorrectly charged fees. Dec 19, 17:38:03 GMT+0 - Resolved - We've confirmed new payouts work correctly. We're marking this incident as resolved, but we're still working on refunding affected recipients. We will communicate that resolution directly to the affected recipients. Dec 18, 04:10:00 GMT+0 - Identified - We're aware that Venmo is incorrectly charging recipients fees. We are delaying Venmo redemptions while we work to fix this issue. We are also working on a resolution for recipients who received less than the correct amount. We'll update this as soon as we know more.

Type: Maintenance Duration: 30 minutes Affected Components: Reward redemptions, Dashboard, API Nov 22, 10:00:00 GMT+0 - Identified - We will upgrade a critical part of our infrastructure, and that will require up to 30 minutes of downtime. Nov 22, 10:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 22, 10:30:00 GMT+0 - Completed - Maintenance has completed successfully

Type: Incident Duration: 3 hours and 3 minutes Affected Components: Reward redemptions, Marketing site, Dashboard, API, Sandbox Nov 18, 12:00:00 GMT+0 - Investigating - Due to a global Cloudflare outage, we are currently experiencing disruption in our services. We will share updates as soon as more information becomes available. Cloudflare is aware of the issue and is currently investigating on their end: Nov 18, 15:02:56 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 7 hours and 5 minutes Affected Components: API, Reward redemptions Oct 20, 14:32:28 GMT+0 - Identified - A small portion of physical and digital prepaid cards are experiencing delays, preventing order placement and some cards from being displayed. The problem has been identified and we are working with our vendor to resolve this issue. Oct 20, 21:37:03 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 7 hours Affected Components: Sandbox Sep 15, 07:10:00 GMT+0 - Identified - An issue occurred during an automatic upgrade of our infrastructure in Google Cloud that required our manual intervention to fix. The Sandbox environment was not accessible during this period. Sep 15, 14:10:00 GMT+0 - Resolved - We've fixed the issue, and the environment is back up.

Type: Incident Duration: 17 hours and 25 minutes Affected Components: Reward redemptions Jun 20, 15:13:42 GMT+0 - Resolved - Our provider \*\*Runa\*\* has traced the problem to Cloudflare mistakenly flagging some redemption links. Most customers can redeem normally, but a subset may still encounter errors. For real-time visibility into which gift-card brands are currently redeemable and future updates, please check . Jun 19, 21:48:32 GMT+0 - Identified - **Impact:** Some gift-card redemptions processed through our upstream provider **Runa** are currently failing. **Status:** We’re working closely with Runa and monitoring their recovery efforts. We’ll post another update as soon as redemptions are fully restored.

Type: Incident Affected Components: Reward redemptions, Dashboard, API Apr 24, 00:13:00 GMT+0 - Resolved - SendGrid says email delivery is back to 100%. Email that was delayed has now been delivered. [You can also check SendGrid's status page directly for updates.](https://status.sendgrid.com/)

Type: Incident Duration: 37 minutes Affected Components: API, Reward redemptions, Dashboard Apr 16, 20:03:02 GMT+0 - Resolved - This incident has been resolved. Apr 16, 19:26:00 GMT+0 - Monitoring - We've identified an issue causing high database load which is leading to slow response times across the board. We've identified a fix and are expecting performance to improve shortly.

Type: Incident Duration: 5 hours and 14 minutes Affected Components: Reward redemptions Apr 14, 00:07:32 GMT+0 - Monitoring - The upstream vendor outage has ended, and we are monitoring things as they resolve to normal. Apr 13, 21:20:50 GMT+0 - Identified - Our primary provider for US-based gift cards is experiencing a production outage. We're monitoring the situation and hope to have reward redemptions reinstated shortly. Apr 14, 02:34:25 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 32 minutes Affected Components: Reward redemptions Apr 8, 21:19:48 GMT+0 - Monitoring - We are monitoring the situation. Apr 8, 21:52:16 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 48 minutes Affected Components: Reward redemptions Mar 24, 19:28:51 GMT+0 - Investigating - We are currently investigating this incident. Mar 24, 21:17:18 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 20 minutes Affected Components: API, Reward redemptions, Dashboard Jan 13, 21:59:00 GMT+0 - Resolved - This incident has been resolved. Jan 13, 21:39:00 GMT+0 - Investigating - Partially degraded performance on the dashboard and API.

Type: Incident Duration: 19 days, 12 hours and 44 minutes Affected Components: API, Dashboard Oct 25, 20:03:43 GMT+0 - Resolved - We are seeing emails delivered immediately at this point. Oct 25, 12:15:00 GMT+0 - Monitoring - The service provider we use for email delivery, SendGrid, is experiencing an incident that is causing delays in email delivery. This can affect all emails we send, including reward deliveries and transactional emails such as login verification emails. SendGrid is assuring us that no email is lost, but this is currently causing significant delays in emails being delivered. We are monitoring the SendGrid incident and will verify that emails are delivered correctly.

Type: Incident Duration: 17 minutes Affected Components: Dashboard Sep 12, 15:32:03 GMT+0 - Investigating - We're investigating order placement issues in the dashboard, where the final review screen shows "0" amounts for orders. Sep 12, 15:34:43 GMT+0 - Identified - We identified the problem and are deploying a fix. We expect things should be working again once the new code is in production, perhaps 15-20 minutes. Sep 12, 15:49:16 GMT+0 - Resolved - This incident has been resolved. Sep 12, 15:50:12 GMT+0 - Resolved - The incident has been resolved - if you are still experiencing issues, you may need to do a browser refresh (CMD + R or CTRL + R). Thank you for your patience.

Type: Incident Duration: 5 minutes Affected Components: API Jul 24, 19:18:00 GMT+0 - Resolved - This incident has been resolved. Jul 24, 19:12:00 GMT+0 - Identified - Today at around 3:13pm ET there was a problem with a database change we deployed that caused a number of actions to start failing across the site. The following actions failed during this time: * Placing new orders * Approving orders * Redemptions This problem lasted for about five minutes from 3:12:30 pm to 3:18:22 pm ET. Failed redemptions retry automatically and recipients should have received their payout by now. **However, you will have to retry newly placed orders as well as order approvals via the API.**