Torii experienced a minor incident on January 20, 2022 affecting Application Console and API Server, lasting 3d 17h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 20, 2022, 02:32 PM UTC
The issue has been identified and a fix is being researched.
- identified Jan 20, 2022, 02:49 PM UTC
Data integrity is not affected. Once the issue is resolved, application will be back to normal. We are seeing performance issues with one of our databases. This may affect certain functionality of the app, such: - Creating users - Creating API keys
- identified Jan 20, 2022, 02:50 PM UTC
Data integrity is not affected. Once the issue is resolved, application will be back to normal.
- identified Jan 20, 2022, 06:35 PM UTC
Our Dev Team has begun deploying a fix. The fix does require some unscheduled database maintenance. During this maintenance time, users may experience intermittent issues with the app including: - Logging into the application - Slow Loading Pages - Inviting New Users - Creating New API Keys At this time, we do not have an ETA as to when the fix will be complete, but we will continue to provide regular updates.
- identified Jan 20, 2022, 09:35 PM UTC
The deployment of the fix is still underway and our Development Team is monitoring it closely. We do not have an ETA as to when the fix will be complete, but we will continue to provide regular updates.
- identified Jan 21, 2022, 12:36 AM UTC
Our Development Team continues to closely monitor the deployment of the fix. We do not have an ETA as to when the fix will be complete, but we will continue to provide regular updates.
- identified Jan 21, 2022, 05:22 AM UTC
Our Development Team continues to closely monitor the deployment of the fix. We do not have an ETA as to when the fix will be complete, but we will continue to provide regular updates.
- identified Jan 21, 2022, 07:57 AM UTC
Our Development Team continues to closely monitor the deployment of the fix. We expect that the majority of the functionality will be restored within one hour.
- identified Jan 21, 2022, 09:13 AM UTC
The following functionality has been restored: - Logging into the application - Inviting new users - Creating new API keys - Integration syncs Workflows are still paused and will be operational again in about 2 hours. The team is working to restore the rest of the functionality.
- monitoring Jan 21, 2022, 01:16 PM UTC
All functionality with the exception of usage data has been restored. Collection of usage data is delayed and may not be shown as the system is still analyzing the data.
- monitoring Jan 22, 2022, 02:59 PM UTC
The team is still monitoring the fix. All functionality with the exception of usage data has been restored. Collection of usage data is delayed and may not be shown as the system is still analyzing the data.
- monitoring Jan 23, 2022, 02:07 PM UTC
Collection of usage data is delayed and may not be shown as the system is still analyzing the data.
- monitoring Jan 23, 2022, 09:54 PM UTC
Usage data is now being collected. The following workflow triggers are disabled until collection of usage data has been completed: License not in use, App not in use. We expect all data to be collected within 24 hours.
- resolved Jan 24, 2022, 08:03 AM UTC
Collection of usage data has been completed. All systems are operational again.