Alchemer incident
US Application Outage is Resolved. Responses continue to be processed.
Alchemer experienced a minor incident on August 10, 2020 affecting Application - US and Response Processing - US and 1 more component, lasting 1d 9h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 10, 2020, 04:36 PM UTC
We are investigating recurring issues with the US application not loading. More information to follow shortly.
- identified Aug 10, 2020, 04:44 PM UTC
We are seeing performance improve at this time. The issue has been isolated to a portion of the service and we are continuing to mitigate the issue going forward. More information to follow. Thank you for your patience.
- monitoring Aug 10, 2020, 04:52 PM UTC
We are monitoring the resolution put in place to restore normal service. More information to follow. If you are still experiencing intermittent error messages in the application at this time, please reach out to our Support Team to discuss.
- identified Aug 10, 2020, 05:19 PM UTC
The US application is still experiencing issues with access and producing error messages upon moving between pages. Our Administration team is performing maintenance on the corresponding systems to restore normal service. Thank you for your patience.
- identified Aug 10, 2020, 05:47 PM UTC
Our team continues to work to restore access to the application with consistent fashion. Progress to resolution is being made and next steps for restoring consistent performance are being taken. EU and CA application are not affected at this time. Respondent experience for survey taking is not affected at this time. We are seeing delays in response processing which is causing data to be delayed in appearing in the application. Data is being gathered safely and securely. More information to follow.
- identified Aug 10, 2020, 06:55 PM UTC
The US application continues to be unavailable at this time. Users are unable to access the application. Survey taking is not impacted by this. We are working to resolve this.
- identified Aug 10, 2020, 08:20 PM UTC
Most users are now able to log in successfully. There are still intermittent issues for some customers. Work continues to fully resolve this for all of our customers. Please reach out to [email protected] if you have any questions. We are here to help!
- identified Aug 10, 2020, 09:49 PM UTC
We are continuing to progress through maintenance and troubleshooting to return stability and consistent US app performance. Thank you for patience.
- monitoring Aug 10, 2020, 10:35 PM UTC
We have been able to restore access for a majority of affected accounts as of now and are seeing consistency in function there. We are continuing to work to mitigate continued issues for a portion of accounts experiencing slow load times on the main dashboard, including alert messages noting "Sorry! There was an error. Please try again later." Troubleshooting is continuing to restore full service as soon as possible. Thank you for your patience.
- monitoring Aug 11, 2020, 02:15 AM UTC
System Administration teams are continuing to progress through maintenance to resolve all instances of errors appearing in the application. Thank you for your patience while we complete work to restore service to all affected accounts.
- monitoring Aug 11, 2020, 12:44 PM UTC
System Administration teams are continuing to progress through maintenance to resolve all instances of errors appearing in the application. Thank you for your patience while we complete work to restore service to all affected accounts.
- monitoring Aug 11, 2020, 04:42 PM UTC
As many of you know, SurveyGizmo experienced an outage yesterday at 10 AM Mountain Time (MT) due to a corrupted database on failed hardware. We have restored service to most of our customers, but 6% (~850 customers) remain affected. We are working to safely and securely restore access and all data for these customers. While system access has been impacted, survey taking / collecting responses continues on active surveys. SurveyGizmo has a solid data backup from 8 AM MT (2 hours prior to the outage) with no risk of data loss. Once the system has been fully restored, queued survey responses will be processed through the present time. We will continue to provide hourly updates on our status page. We apologize for any inconvenience this has caused and we will do everything in our power to restore your confidence in our platform.
- monitoring Aug 11, 2020, 05:59 PM UTC
We continue work to fully restore systems and process all queued response data. As we have further updates on the progress, we will update you here. Previous update: As many of you know, SurveyGizmo experienced an outage yesterday at 10 AM Mountain Time (MT) due to a corrupted database on failed hardware. We have restored service to most of our customers, but 6% (~850 customers) remain affected. We are working to safely and securely restore access and all data for these customers. While system access has been impacted, survey taking / collecting responses continues on active surveys. SurveyGizmo has a solid data backup from 8 AM MT (2 hours prior to the outage) with no risk of data loss. Once the system has been fully restored, queued survey responses will be processed through the present time. We will continue to provide hourly updates on our status page. We apologize for any inconvenience this has caused and we will do everything in our power to restore your confidence in our platform.
- monitoring Aug 11, 2020, 07:40 PM UTC
Access to the SurveyGizmo application is now being restored for all users. Queued survey response data has begun to process and will continue over the next few hours. Application load times may vary as this process completes.
- monitoring Aug 11, 2020, 09:32 PM UTC
Access to the SurveyGizmo application has been restored for all users at this time. Queued survey response data continues to process and we are monitoring progress. Application load times may vary as this process completes.
- monitoring Aug 12, 2020, 12:21 AM UTC
Access to the SurveyGizmo application has been restored. Queued survey response data continues to process and we are monitoring progress. Application load times may vary as this process completes.
- monitoring Aug 12, 2020, 01:34 AM UTC
Access to the SurveyGizmo application has been completely restored. Queued survey response data continues to process and is almost complete. We continue to monitor progress.
- resolved Aug 12, 2020, 02:11 AM UTC
All queued Survey Response data has successfully been processed and the application remains stable. Monitoring will remain in progress.