Artie Technologies Outage History

Started Sep 30, 2026, 12:56 AM UTC· resolved Sep 30, 2026, 01:16 AM UTC

1. identified Sep 30, 2026, 12:56 AM UTC

   Some customers received data processing alerts for running pipelines in us-east-1 -- we've identified the issue and are actively monitoring pipelines to restore availability..

2. resolved Sep 30, 2026, 01:16 AM UTC

   The incident has been resolved and streaming is operational -- we will follow up with an incident breakdown shortly..

3. monitoring Oct 09, 2026, 12:58 AM UTC

   From 5:56 to 6:16 PM PDT on September 29, streaming was degraded for pipelines in our us-east-1 data plane. Right after a planned migration of our compute nodes, an engineer applied an unrelated infrastructure change built on an outdated configuration which undid the migration and replaced those nodes in bulk, causing new nodes to come up that were not attached to public IPs and sending data processing alerts for affected pipelines. We identified the issue internally at 5:58 PM, and most pipelines recovered by 6:16 PM as the replacement nodes came online with our static egress IPs. We restarted the few remaining pipelines by hand and confirmed all pipelines were healthy by 6:45 PM. Data was delayed, not lost, and no action is required. We apologize for the disruption and have implemented the following changes to ensure it doesn't happen again: \- additional guardrails on the infrastructure deploy process to catch rogue changes that aren't based on the latest configuration or introduce unintended drift \- process improvements to tighten infrastructure reviews across the team \- exploring foundational infrastructure improvements to reduce pipeline time to recovery when underlying nodes are replaced.

Read the full incident report →

Started Apr 07, 2026, 06:00 PM UTC· resolved Apr 07, 2026, 07:00 PM UTC

1. investigating Apr 07, 2026, 06:00 PM UTC

   We are currently investigating this incident..

2. resolved Apr 07, 2026, 07:00 PM UTC

   This incident has been resolved. We will prepare a post-mortem for this incident shortly..

Read the full incident report →

Started Mar 25, 2026, 11:00 PM UTC· resolved Mar 25, 2026, 11:00 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Feb 20, 2026, 01:22 AM UTC· resolved Feb 20, 2026, 02:10 AM UTC

1. investigating Feb 20, 2026, 01:22 AM UTC

   We are currently investigating this incident..

2. resolved Feb 20, 2026, 02:10 AM UTC

   Root cause: our NGINX controller became unresponsive, which caused the Dashboard API to be unavailable. As a result, customers were unable to log into the UI or make pipeline changes. Service has now been fully restored and systems are operating normally. We’ll continue to monitor closely to ensure stability. Please reach out to our team if you experience any lingering issues..

Read the full incident report →

Started Jan 31, 2026, 12:30 AM UTC· resolved Jan 31, 2026, 08:24 AM UTC

1. identified Jan 31, 2026, 12:30 AM UTC

   A config change has affected networking for pipelines in the us-east-1 data plane. We are continuing to work on a fix for this incident..

2. monitoring Jan 31, 2026, 03:26 AM UTC

   There's a CNI error. We are seeing some clients recover and are still working towards a full recovery..

3. monitoring Jan 31, 2026, 04:39 AM UTC

   We’ve identified intermittent DNS issues on a specific node and are actively working to resolve them. We’re working to restore full reliability..

4. monitoring Jan 31, 2026, 05:14 AM UTC

   We are now working with AWS to diagnose the partial DNS failure..

5. monitoring Jan 31, 2026, 07:42 AM UTC

   We are continuing to see intermittent DNS errors. The issue has been escalated with AWS, and an AWS engineer is joining the investigation shortly. We’ll share another update as we learn more..

6. resolved Jan 31, 2026, 08:24 AM UTC

   We’ve identified the root cause of the DNS instability: an out-of-order VPC CNI upgrade led to an undocumented version mismatch, which caused conflicts in our DNS routing rules. The issue has now been resolved, and we are closely monitoring all pipelines to ensure continued stability..

7. monitoring Feb 10, 2026, 01:34 AM UTC

   ### Incident Overview * **Date:** January 30, 2025 * **Affected AWS region:** us-east-1 * **Duration:** partial outage 6.5 hours, total outage 1 hour ### Summary On January 30, a routine infrastructure upgrade to our Kubernetes networking layer caused a service disruption in one of our primary cloud regions. What began as a partial outage escalated into a full outage affecting data pipelines for customers in the us-east-1 region. We sincerely apologize for the impact this had on your operations. Service was fully restored the same evening after our team identified and resolved the underlying issue. It was a version mismatch between critical networking components in our cluster. ### Impact We understand how important reliability is to your business, and we are deeply sorry for the disruption. During the outage: * **Service was unavailable** in the us-east-1 region for 7.5 hours. * A small number of affected customers experienced **data replication interruptions**, which our team worked to remediate promptly. We recognize that any downtime is unacceptable, and we take full responsibility for this incident. ### What happened Our engineering team was performing a planned upgrade to a networking add-on (vpc-cni) to increase capacity in our Kubernetes infrastructure. During the upgrade: 1. The change was applied successfully in a non-production environment but encountered compatibility issues in the production cluster. 2. A rollback was not possible because the previous component version was no longer supported by the underlying platform. 3. The issue was compounded by version mismatches across several interdependent networking components, which made diagnosis more complex than expected. Our team identified the root cause as an outdated networking component (kube-proxy) that was incompatible with the rest of the upgraded stack. Once the fix was applied, service was restored. ### What we're sorry about We want to be transparent; this incident was avoidable. We moved forward with a change that, in hindsight, required more thorough validation and coordination. Specifically: * The production environment differed from the test environment in ways we did not fully account for. * We did not have sufficient safeguards in place to catch the version incompatibility before it caused an outage. * Our incident response process was slower than it should have been, and we did not escalate to our cloud provider's engineering team quickly enough. We are sorry for the disruption and for falling short of the reliability you expect from us. ### Our plan going forward We are actively carrying out a comprehensive set of improvements to prevent incidents like this from happening again: **Stronger change controls** * We have strengthened our change management process so that all planned infrastructure migrations require formal approval and a minimum of two engineers working together to apply changes. * Non-reversible changes are restricted to business hours (Monday through Thursday) to ensure full team availability and vendor support coverage. **Improved testing and validation** * We are auditing and standardizing component versions across all of our clusters to eliminate hidden drift between environments. * Production and non-production clusters are being aligned to ensure test results are representative. **Faster incident response** * We are formalizing a war room procedure with clearly defined roles: a primary engineer making changes, a secondary approving them, an incident commander coordinating, and a communications lead keeping customers informed. * We are establishing clear severity definitions and escalation timelines so that vendor support is engaged earlier when needed. **Infrastructure resilience** * We are working toward running multiple clusters per region so that workloads can be migrated if a single cluster is compromised. * We are building the capability to move customer pipelines between clusters with minimal disruption. * We are proactively engaging our cloud provider to review and approve infrastructure upgrade plans before they are executed. --- We value your trust and are committed to earning it back through these concrete actions. If you have any questions or concerns about this incident or our remediation plan, please do not hesitate to reach out to your account contact or our support team..

Read the full incident report →