eazyBI experienced a major incident on May 3, 2018 affecting Web App, lasting 22h 54m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified May 03, 2018, 10:09 AM UTC
Today we are experiencing instability in eazybi.com services. We are working to get them back in order as soon as possible.
- identified May 03, 2018, 10:09 AM UTC
We are continuing to work on a fix for this issue.
- identified May 03, 2018, 12:50 PM UTC
eazyBI data imports are still running slowly and, in some instances, end with an error. We will rerun failing imports automatically when performance problems will be resolved.
- identified May 03, 2018, 02:45 PM UTC
We got an update from Google engineers. The main cause of the problem is located in database servers. Problem is still being investigated. Next update will be in 2 to 4 hours
- monitoring May 04, 2018, 04:39 AM UTC
The database servers are working stable. All failing imports are restarted. Most of them are already finished. We will continue monitoring system performance
- resolved May 04, 2018, 09:03 AM UTC
The Google Cloud Engineering team was able to find the root cause of the restarts. There was a noncritical service update that contained an issue causing MySQL memory to be overused and thus, causing the unwanted restarts. This change was rolled back and to resolved this incident. There are no monitored incidents for past 10 hours.