pingoru pingoru Grain had 23 outages in the last 2 years totaling 4298h 14m of downtime, averaging 0.9 incidents per month.
There were 23 Grain outages since October 29, 2024 totaling 4298h 14m of downtime. Each is summarised below. Incident details, duration, and resolution information.
Our primary transcription provider is experiencing an outage, the team is monitoring for impact and any required recovery.
The upstream incident has been resolved, the team will work through the backlog of any impacted recordings for reprocessing.
An outage with an upstream provider caused recording bots to be delayed or fail to join altogether. The team is investigating potential mitigations for the future.
Our transcript partner experienced an outage causing some captures to fail to generate a transcript. The team is actively working to reprocess all impacted captures.
All failed transcripts have been reprocessed.
The team identified a crashing bug during a deployment, a hot fix was applied and the application is once again loading. The team is monitoring for further issues.
The team has no observed further issues.
Some local captures are stuck in a processing state. The team is investigating the root cause but believe that all data will be fully recoverable.
All hung local captures were recovered.
An outage with an upstream provider is causing instability for bots including delays in scheduled bots joining and ad-hoc bot failures. The team is investigating potential mitigations.
At the half hour, the team observed minimal delays in scheduled bot and no ad-hoc failures. We are in communication with the upstream provider about any further potential issues.
The team has not observed any further degradation in the bot service.
Upstream provider is experiencing an outage causing out instability in our bot recorders including delayed joined times and recording failures. The team is investigating the scope and any potential mitigations.
The outage has subsided. The team has identified a set of recordings currently stuck in processing state that require manual recovery and will be working to recover them as quickly as possible.
All recordings that were successfully captured have now been recovered. Future improvements around being more robust to upstream failures are being researched by the team.
An issue with an upstream provider is causing some bot recordings to be stuck processing. We are working with the upstream provider to uncover the root cause an to recover the recordings.
Most recordings impacted by the issue have successfully recovered, a small set are still pending further investigation. The recorder has returned to full functionality.
All recordings with recoverable media have now be successfully recovered.
Platform is experiencing stability due to increased database load. The team is investigating and looking for mitigations.
The team has deployed a fix and the database load is now back to normal. We are monitoring for any further issues.
The team determined the root cause was related to index changes on a hot job queue table, leading to high CPU usage. The correct indexes were applied and the database returned to full health.
An outage with an upstream provider is causing failures for new recordings (both bot and local capture). The team is investigating potential mitigations.
The team has moved to a new region for the upstream provider and new bot recordings are starting to succeed.
We are seeing signs that the main region is coming back online. Local captures should start capturing correctly.
We've moved back to the main region and we're seeing new bots successfully be launched. Some issues are lingering around double scheduled bots and we are making adjustments to clean those up.
The team rolled back to the backup region due to continued higher than normal error rates.
Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.
The platform is currently experiencing instability, the team is deploying upgraded infrastructure and investigating the root cause.
The team has rolled out mitigations and we're not longer seeing node failures. However we are still investigating the root cause of the issue.
The mitigation has prevented further instability but the team is still investigating the root cause of the issue. The issue caused intermittent and cascading excess memory pressure on the application nodes.
The team has deployed a fix around large memory spikes trigging out-of-memory failures. Since the latest fix, no further memory spikes have occurred.
An issue with an upstream vendor is causing a partial outage of Meet recorders, investigation is underway.
A mitigation has been deployed and we are monitoring for for further failures.
We have not seen subsequent failures.
We are monitoring an incident with an upstream provider that is causing some meeting recorders to fail to join calls. We are expecting a resolution in the near term.
This incident has been resolved.
AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
We are continuing to monitor for any further issues.
AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
This incident has been resolved.
Due to an unexpected change in a sub processors API, recording processing experienced a temporary delay. This change has been reverted and we are seeing processing move forward. We are monitoring to ensure all backed up processing completes as expected.
All delayed recordings have successfully processed.
Recording capture is currently experiencing issues because of instability with the Grain database. The team is investigating the root cause and deploying mitigations.
The system has stabilized and the team will now be working on recovering failed recordings where possible.
This incident has been resolved.
We are currently investigating an issue that is causing bots to be unable to join meetings.
Our affected upstream provider has identified the issue and is in the process of deploying mitigations.
Our affected upstream provider has deployed mitigations and we are now observing bots joining meetings at normal rates. We will continue to monitor for stability.
This incident has been resolved.
Upstream providers including Google services and vendors relying on Google Cloud Platform are experiencing widespread outages which is causing instability and processing delays for Grain. Once the upstream issues are resolved, recording processing should fully complete automatically.
The upstream outage has resolved and Grain systems are back to full operation.
From 05/18 ~21:30 UTC until 05/19 ~10:00 UTC, Microsoft calendars were failing to sync due to an expired Microsoft OAuth Client credential. Due to the nature of the failure, alarms were not correctly setup which lead to the delayed resolution. The sync failure led to some events not having their auto-record rules applied if the event was created or edited during the outage window. The team has put in place mitigations to prevent this failure state in the future.
Our recording service is experiencing a high rate of failures, causing issues with scheduling and booting up new recorders. Some meetings may not be captured.
Mitigation efforts have brought the recording service to a stable place, we are monitoring for further issues and will be evaluating any possibilities for recovery of incorrectly handled recordings.
We've seen another uptick in errors after the top of the hour and new mitigations are being explored.
This incident has been resolved.
The application has been experiencing instability due to increased database load caused by a dependency update. This is causing some functions to fail to process including calendar syncing. We're working on rolling back the dependency to resolve the issue.
The fix has been deployed and database load has returned to normal. We're monitoring for further issues and evaluating processes that need to be re-run.
This incident has been resolved.
Our transcription partner is experiencing an outage, causing delays in transcript processing. We are awaiting further information.
The outage is beginning to clear and we are seeing transcripts complete processing. We will begin backfilling any failed transcripts once we've received confirmation that the outage is fully resolved.
This incident has been resolved.
The marketing page is currently offline due to a certificate expiration issue on our upstream provider. We are working to restore the page.
The upstream provider has fully resolved the issue.