Heron Data Outage History

This issue is now resolved. For any questions, please contact [[email protected]](mailto:[email protected]).

This issue is now resolved. Submissions between 10:15-11:15 ET / 15:15 - 16:15 UTC were affected. All affected submissions have been rerun. For any questions, please contact [[email protected]](mailto:[email protected]).

All impacted end users have been reprocessed

All failed workflows have now been retried and incident is resolved

We ran an upgrade of our production database to handle increased traffic which resulted in about 1 minute of intermittent downtime for our API users. The API performance is back to normal.

Some users might have experienced intermittent 500 response codes between 7PM and 8PM BST. This was caused by a monitoring tool we added to better understand our usage and latency which unfortunately interfered with external API calls which we rely on for some API endpoints and users. Once we identified the issue we reverted the monitoring changes and all systems returned to normal. Moving forward we will deploy and babysit all monitoring changes (on top of code deploys and core infra changes) in staging before releasing to production.

As part of releasing new ML models to improve our merchant extraction service, we hit scaling issues when exposed to a higher than normal throughput. All the relevant checks were made in testing, local and staging environments but unfortunately nothing could reproduce production-level traffic. We have reverted the release, and all systems are fully functional again.

Our REST API responses experienced increased latency, peaking at 60s. Some customers using our async enrichment flow will have seen webhooks take longer to arrive than usual. All systems are back to normal now.

Our automated SSL renewal process failed. This meant our API was down between 10:08 and 10:49AM BST. We provisioned a new SSL certificate manually, restored automatic SSL certificate renewals and added monitoring to prevent this from happening again.

In an attempt to optimise our network traffic routing, we provisioned some config changes to our load balancer. The change caused our Kubernetes routing to become unhealthy and thus blocking traffic coming into our app.herondata.io domain. This was tested in staging environments and worked correctly, which means production-level traffic made it impossible to apply this change properly. We quickly reverted the config to stabilise the API which is now operating normally.

Google Cloud has resolved this issue. Our API is fully functional again.

We noticed intermittent increased latency yesterday afternoon due to unprecedented levels of traffic. While debugging the root cause and how to handle this, we increased our compute resources in multiple areas to handle the traffic. We have now identified the root cause and solved the issue. Response times are back to normal but we will continue to monitor to ensure the fix worked.

From 00:10-00:19 UTC (9 minutes), we had a major outage in our API when we experienced a high amount of 502 errors that rendered our backend API unusable. This was caused by issues in our load balancer, which was caused by usually high amounts of traffic and unavailable pods to service the traffic. We are in the process of adding better scalability to our backend to handle such spikes in traffic moving forward, including better horizontal scaling metrics.

We experienced API outage (<5mins) from a prolonged database migration. All systems are back to normal

Response times have been restored.

This incident has been resolved.

We apologise for the increased 500 errors and timeouts experienced in the last 10 hours. We introduced a new index to our database which was partially built and resulted in an “invalid” state. This resulted in long running queries which blocked subsequent reads. We dropped the index, as well as blocking database processes, and are now recovered.

This incident has been resolved.

This incident has been resolved. We experienced a high number of API requests that were using a high amount of CPU, overwhelming our infrastructure. We shipped a change that reduced the CPU usage, and have seen normal CPU usage for the last 8 hours

The API has been stable and working since the fix. This is now been marked as resolved.

This incident has been resolved.

This incident has been resolved.

All effected workflows have now been successfully re-run. This incident is now resolved.

This issue is now resolved, API and dashboard access should be completely restored.