Keypup Outage History

Google Cloud Platform is currently having issues with services related to background jobs, on which Keypup relies. This results in an elevated error rate in the data processing jobs, with a high number of jobs failing or being retried. We are monitoring the situation and will reprocess application updates since Thursday once the incident is over on the GCP side.

Data updates have been paused while we are processing a large backlog of jobs. Job processing is slowed down due to continued issues on Google Cloud Platform.

We have re-enabled data updates from third-party applications and are continuing to clear the backlog of jobs. On the reporting side, the loading of insights may feel slower due to the following: - The auto-refresh mechanism of dashboard insights is currently disabled. Insights will only be refreshed upon reloading the page (or visiting another dashboard, then coming back) - The ahead-of-time caching of dashboard insights is currently disabled. Insights are live-loaded if data has changed since the last load.

We continue the process the pending backlog and will provide another update in the next few hours.

The processing of the remaining backlog is almost over. Once processed, the team will trigger a job to reprocess the application updates for the last few days to ensure no updates were missed.

The processing backlog has been cleared, and our team is now reprocessing the last few days of updates to ensure no data updates were missed.

The team has performed maintenance activities on the database to restore performance. The system is currently catching up on updates from the last two hours.

Cloudflare has implemented a fix and is currently monitoring traffic through their network.

The issue retrieval flow has resumed normal operations. We are currently monitoring the logs to ensure proper activity. A resync of Jira projects will take place during the weekend to ensure the data are up to date.