Heron Data had 12 outages in the last 2 years totaling 61h 4m of downtime — averaging 0.5 incidents per month.
There were 12 Heron Data outages since June 17, 2024 totaling 61h 4m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are currently investigating an issue for bank statements not parsing
A fix has been deployed for new deals, we are currently backfilling any unprocessed workflows.
All failed workflows have now been retried and incident is resolved
We are currently experiencing an outage with submissions getting stuck in processing in our system. We have identified the issue and are working with highest priority to get this resolved.
We are currently re-processing submissions that were stuck.
All processing should be back to normal. Submissions received between 08:19 and 17:25 UTC on Apr 21 may be impacted - please reach out to [email protected] if you would like these submissions reprocessed
This incident has been resolved.
The issue has been identified and a fix is being deployed. Some delay in workflow processing for small subset of users
Fix has been deployed, we are now backfilling the failed workflows
All effected workflows have now been successfully re-run. This incident is now resolved.
There is an ongoing intermittent issue with our DNS provider cloudflare: https://www.cloudflarestatus.com/ - we are actively investigating for updates / root cause.
Cloudflare is starting to recover, access to dashboard and API should start to become available.
Cloudflare still experience issues, and are resolving. Dashboard and API access will still be intermittent.
We are still actively monitoring the cloudflare and are attempting to implement workarounds
We have implemented some work arounds for our API and dashboard and are starting to see some requests recover
We're seeing service restored on both API and dashboard access and will continue to monitor
This issue is now resolved, API and dashboard access should be completely restored.
We are having issues with GCP provisioning compute instances for async processing. Async processing is impacted
GCP is denying "spot instances" for our async processing kubernetes cluster. This means that no async tasks are being processed. We are creating a new node pool that does not rely on spot instances to get us back up & running
New node pools are up & running, tasks are running again. We are recovering.
This incident has been resolved. Our async processing uses GCP "spot" instances, and GCP unexpectedly started denying the creation of these spot instances in our kubernetes cluster. We have now created non-spot instances that will allow us to serve async tasks as normal, and have both spot and non-spot instances in place for future
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
We have remediated the issue and all new submissions will process normally. We are recovering on older submissions that were created during the outage
Incident has been resolved. Postmortem to follow. We had an issue where our asynchronous processing capability was compromised by an infinite looping async task from a new feature launch. The root cause is now understood, and a fix is planned. For now the feature is disabled across Heron platform
We are experiencing issues with one of our cloud providers, GCP. Namely, we are seeing degradation in the cloud storage, cloud run, and AI services
We are continuing to investigate this issue.
Read requests are succeeding, but post requests are failing
GCP services have recovered. Our systems are recovering
GCP has advised that they are experiencing varying degrees of impact on different components. We will continue to monitor and recover as possible https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Google Cloud Platform had a system outage that impacted our systems: [https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW) . This mainly impacted our ability to send data to GCP such as storing files or other information All data sent via email-based integrations have been recovered \(emails / files / end users have been reprocessed\) and no data was lost during the outage. Any API-based requests that Heron indicated were successful during this time were indeed successful, but customers may have experienced a higher-than-normal error rate during this time. We are pleased to report that no data was lost during this outage, and have made some changes in the Heron system to make it easier to recover from similar outages in the future.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue where Heron async processing is experiencing slowness
Systems are stable
The incident has been resolved
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
Our email server provider, Postmark, experienced an issue with processing queued messages. This meant that all submissions forwarded to Heron Data were not being received by the Heron system. No data was lost in this process, and the provider system has mostly recovered -- all incoming messages are being processed and most historical messages are now processed. We are monitoring to ensure complete recovery https://status.postmarkapp.com/notices/zno1dlxjdjmblc0d-service-issue-outbound-sending-and-inbound-processing-messages-are-being-accepted-and-queued
This incident has been resolved.