MetaPulse experienced a major incident on July 13, 2023 affecting Background Workers and Web Site, lasting 4h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 13, 2023, 02:53 PM UTC
Email is not being sent from our web server. We have identified the issue and are working to resolve it as soon as possible.
- identified Jul 13, 2023, 02:53 PM UTC
We are continuing to work on a fix for this issue.
- identified Jul 13, 2023, 05:27 PM UTC
We are continuing to work on fixing this issue. We will attempt to resend the unsent emails once this issue is fixed.
- resolved Jul 13, 2023, 07:15 PM UTC
This issue has been fixed. We will be sending out missed emails shortly.
- postmortem Jul 20, 2023, 11:22 PM UTC
On July 13, 2023, [metapulse.com](http://metapulse.com) was unable to send email for several hours. This was due to the Amazon SES authentication keys being rotated while the servers were using the original keys. The extended downtime was due to difficulty getting the servers to recognize the new keys. After the issue was resolved we were able to resend most emails including notifications and mailings. However some emails were unrecoverable including users registration and resetting passwords. Due to this issue, we have taken steps to make MetaPulse less dependent on email. A new Data Exports page is now available which lists all data exports and provides download links instead of relying on email.