Superhuman experienced a major incident on August 20, 2020 affecting Email sending, lasting 11h 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 20, 2020, 05:01 AM UTC
A small portion of emails sent from Superhuman are failing at the moment due to an upstream outage on Gmail's API. We're investigating and will update when we have a better idea of the impact and potential resolution steps.
- identified Aug 20, 2020, 05:12 AM UTC
Currently about 10% of emails sent via Superhuman are failing due to errors communicating with the Gmail API. If this happens to you it will show up in the client with a red dot that says "send failed". Tapping on the "send failed" will let you re-send your email if you'd still like to.
- identified Aug 20, 2020, 06:00 AM UTC
We're now seeing close to 40% of emails fail to send, and a similar number of reminders are not returning on time (as they rely on a similar Gmail API). By design Superhuman will retry about 10 times over the course of several minutes until sending succeeds. While that's happening we optimistically include the emails in your Sent Items (usually this makes everything feel blazingly fast!). The retry logic means that during an outage like this some emails will send more slowly than usual (when it retried many times before working), while others will not send and re-appear in your Inbox as failed after several minutes. (Pro-tip: to tell if an email in your sent items is really sent, or still being tried, try to add a label to it cmd+k "Label" will not show up until an email is sent successfully). Google's incident coverage is here: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a45de3b26d6c5872f4cfe8e3424d7a82.
- monitoring Aug 20, 2020, 08:47 AM UTC
The error rate we're observing on the Gmail API has been trending down significantly, and for the last 30 minutes our send failure rate has stayed below 1%. Although we're not ready to close this issue yet (we're still seeing many slow sends, and a much-higher-than-normal failure rate) most people should see everything working again. If you had emails that failed to send, they'll show up in Superhuman and you can tap on "Send Failed" to retry. Any reminders that should have returned to your inbox during the time that Gmail's API was down will have returned just now. Google continues to work on the underlying issue, and they post updates here: https://status.cloud.google.com/incident/zall/20008
- resolved Aug 20, 2020, 04:50 PM UTC
And we're officially back! We stopped seeing elevated error rates at around 3am PST, and Google officially resolved the incident at 4:19am. If you had emails that failed to send, they'll show up in Superhuman and you can tap on "Send Failed" to retry. A small subset of reminders that should have triggered between 10pm and 2am PST were delayed until 2am PST.