Documo experienced a major incident on June 12, 2025 affecting Fax Services, lasting 5h 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 12, 2025, 06:35 PM UTC
We are currently investigating an issue where customers are experiencing an 'internal server error' when attempting to view, print, and send outbound faxes. Updates to follow.
- investigating Jun 12, 2025, 07:02 PM UTC
We are currently investigating an issue due to a global GCP outage. This is causing customers to receive an 'internal server error' when attempting to view, print, or send faxes. Our team is continuing to monitor for updates and will provide them as we have them.
- identified Jun 12, 2025, 08:28 PM UTC
We are aware of an outage that is affecting GCP across all regions globally and is affecting many other applications. For the Documo app, it is causing users to receive an 'internal server error' when attempting to view, print, or send faxes. Our team has been in contact with representatives from Google, and understand that their engineers have identified the root cause and have begun to apply the appropriate fixes. It will take some time (exact ETA unknown) for this to propagate across all regions. We will post updates here as soon as we know more.
- identified Jun 12, 2025, 08:58 PM UTC
As of 4:31 PM EDT, we are seeing evidence of recovery and restoration of the flow of faxes through our systems. Enqueued faxes may take some time to be completed, and the update of fax statuses in the portal may be somewhat delayed. Our contacts at Google have advised us that they are not yet 100% certain that the systems are fully recovered and stable. We will provide further updates when we have been provided the all-clear from Google engineers.
- monitoring Jun 12, 2025, 09:37 PM UTC
Our internal fax queues have been recovering well over the last hour and we look to be in a stable state. Our contacts at Google advise us that full recovery will be around 6 PM EDT. More updates to follow.
- resolved Jun 13, 2025, 12:08 AM UTC
This incident has been resolved.