Sardine Outage History

Started Oct 01, 2026, 02:49 PM UTC· resolved Oct 01, 2026, 07:58 PM UTC

1. investigating Oct 01, 2026, 03:36 PM UTC

   We are investigating elevated latency and increased response times affecting Customer API, Feedback API and Business API requests in our PROD-EU environment. Our engineering team is actively investigating, and we will provide an update as soon as we have more information.

2. investigating Oct 01, 2026, 03:39 PM UTC

   We are continuing to investigate this issue.

3. identified Oct 01, 2026, 04:39 PM UTC

   A mitigation has been applied and latency is improving. We are continuing to monitor and will post another update later.

4. monitoring Oct 01, 2026, 05:03 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Oct 01, 2026, 09:18 PM UTC

   The elevated latency and increased response times affecting the APIs in PROD-EU have been resolved. Response times have returned to normal levels and have remained stable since the fix was applied. We will continue to monitor the environment closely. A root cause analysis (RCA) will be shared once it is complete. We apologize for the disruption.

6. postmortem Oct 09, 2026, 02:29 PM UTC

   ## Summary On 1 October 2026, from approximately 14:20 to 15:45 UTC, our EU environment experienced increased latency and timeouts on the `/v1/customers`, `/v1/issuing/risks`, and `/v1/feedbacks` endpoints. We added service capacity to end the main customer-impact period and subsequently deployed a code change that resolved the underlying cause. ## Customer impact Customers using the affected EU endpoints experienced slow responses, errors, and timeouts. The verified affected endpoints were `POST /v1/customers`, `POST /v1/issuing/risks`, and `POST /v1/feedbacks`. Other measured endpoints did not show comparable latency increases. The US environment and other environments were not affected. ## What happened and why Under peak EU traffic, a service supporting these endpoints allocated memory faster than it reclaimed it. Part of this work involved unnecessary database reads and writes to calculate a crypto-recipient feature even when the request contained no crypto recipient address. The service's memory management was also not configured to account for its operating memory limit. As instances reached that limit, they became unhealthy and restarted. Their traffic moved to the remaining instances, increasing the load and causing further restarts. The EU environment had insufficient spare capacity, and our automatic scaling settings did not add capacity quickly enough to contain the failure. Our monitoring did not give us a clear, sustained signal of the underlying problem. Some latency alerts cleared on their own and were not thoroughly investigated, and we lacked dedicated alerts for this service's memory pressure and repeated restarts. A customer report escalation brought the disruption to our attention. We also mistook a temporary improvement for recovery while requests were still affected. ## Resolution At 15:33 UTC, we increased service capacity in the EU environment. Tail latency across all three affected endpoints returned to its usual level at approximately 15:42 UTC, and service restarts had stopped by approximately 15:45 UTC, ending the main customer-impact period. This capacity increase was a mitigation rather than the root-cause fix. At 17:59 UTC, we deployed a code change that skips unnecessary crypto-recipient database operations when no recipient address is present. Post-deployment measurements showed a 79% reduction in memory use, a 40% reduction in CPU time, and a 67% reduction in garbage-collection activity. This change eliminated more than 99% of the unnecessary database reads and writes and resolved the underlying memory-allocation cycle. ## Timeline \(UTC\) All events below occurred on 1 October 2026. * 14:20: Customer-impact window began. * 14:39: Our on-call engineer received and acknowledged a latency alert for this incident. * 14:44: A customer reported the disruption to us. * 15:03: We declared an incident and began a coordinated response. * 15:33: We increased service capacity. * 15:42: Tail latency across the affected endpoints returned to its usual level. * 15:45: The customer-impact window ended; service restarts had stopped and the expanded capacity was ready. * 17:59: We deployed the root-cause code fix to skip unnecessary database operations. ## Prevention and action items * We have added capacity in the EU and raised the memory available to the service. * We have changed the service so it manages memory against its real limit. * We are correcting the automatic scaling trigger so it adds capacity before instances fail. * We are adding alerts for memory pressure, repeated restarts and out-of-memory terminations, and per-client latency so we can detect this type of issue ourselves and notify customers first. * We deployed the root-cause code fix that skips unnecessary database operations when no crypto-recipient address is present. * We are continuing work to reduce memory use per request for high-volume business workloads.

Read the full incident report →

Started Sep 27, 2026, 09:56 PM UTC· resolved Sep 27, 2026, 10:36 PM UTC

1. investigating Sep 27, 2026, 09:56 PM UTC

   Australia instance (api.au.sardine.ai) is degraded for v1/customers API and v1/issuing/risks API. Team is actively looking into it.

2. monitoring Sep 27, 2026, 10:21 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 27, 2026, 10:36 PM UTC

   This incident has been resolved. We'll follow up with post mortem.

4. postmortem Sep 29, 2026, 09:43 PM UTC

   ## Summary On September 27–28, 2026, Sardine experienced two periods of elevated latency and request timeouts in our **Australian production region**. Requests to `/v1/issuing/risks` , `/v1/customers`, and `/v1/feedbacks` were affected from 12:16–22:00 UTC on September 27 and again from 23:20 UTC on September 27 until 00:23 UTC on September 28. A separate high-volume data-processing workload overloaded a shared regional database. An unbounded historical aggregation initially consumed most of the database capacity. As records accumulated, stale database statistics caused another frequently used transaction lookup to select an inefficient query plan. Query timeouts then recycled database connections, repeatedly reintroducing that plan and amplifying the load. The incident affected API availability, transaction persistence, and the risk level evaluation. ## Timeline _All times UTC._ * **September 27, 12:16** — A high-volume data-processing workload begins. Database CPU reaches capacity and `/v1/issuing/risks` timeouts start. * **14:20** — We receive the first customer report. * **14:29** — The primary support on-call acknowledges a page, but the issue is not escalated effectively. * **18:02** — We send the first substantive customer impact update. * **18:05** — The need for engineering escalation is identified. * **21:06** — Sardine declares an incident. * **21:38** — Active engineering investigation and remediation begin. * **22:03** — A database resize and primary switchover restore customer-facing service. * **22:12** — The high-volume workload is paused and database utilization returns to normal. * **22:40** — The first event is marked resolved. * **22:49** — Database CPU monitors are added for all production regions. * **23:20** — The workload resumes and triggers a second degradation. * **23:41** — The new database monitor pages the on-call team and is acknowledged within 13 seconds. * **September 28 00:14** — A second database resize and primary switchover resolve the inefficient transaction lookup. * **00:23** — Optional aggregation work is disabled and the remaining customer-facing timeouts end. * **00:26 onward** — Database statistics are refreshed and an optimized composite index is deployed. ## Root Cause The incident resulted from the interaction of four conditions: 1. **Unbounded aggregation:** A synchronous query scanned the full transaction history for given condition. This consumed most database capacity when the high-volume workload began. 2. **Inefficient transaction lookup:** As new records accumulated, database statistics did not refresh quickly enough. The query planner selected an index that scanned a customer’s growing transaction history rather than performing a targeted lookup. 3. **Connection recycling:** Timed-out queries caused database connections to close. Replacement connections repeatedly began with the same inefficient query plan, creating a feedback loop of expensive scans, timeouts, and reconnects. ## Contributing Factors * Our API monitoring excluded the relevant client-timeout responses, and we did not alert on query cancellations or sustained database-connection churn. Database CPU monitor was only configured for non-pagerduty alert * The customer detected the first event before our monitoring did. Delayed engineering escalation extended the response time. * Our first recovery addressed the immediate pressure but did not prevent the workload from triggering a second event when it resumed. ## Resolution and Recovery For the first event, we resized the database and switched to the new primary, then paused the high-volume workload. For the second event, we again resized and switched the primary, then disabled optional aggregation to remove the remaining load. After service recovery, we: * Refreshed database statistics. * Deployed a composite index that gives both new and established connections an efficient transaction-lookup path. * Added database CPU monitors for every production region. * Removed optional, unused aggregation from the `/v1/customers` path * Kept the Australian production database at a higher baseline capacity while we complete the permanent query fixes and capacity review. ## Corrective Actions We are taking the following actions: ### Database and application resilience * Replace the affected transaction lookup with an atomic, consistently indexed operation. * Skip unnecessary historical aggregation and bound any required fallback query. * Evaluate a read pool so issuing-risk reads do not compete with customer-event writes on the primary. * Review steady-state database capacity after the query fixes are complete. ### Monitoring and response * Expand monitoring for client timeouts and database metrics * Improve incident routing so customer reports, support tickets, and on-call escalation converge quickly.

Read the full incident report →

Started Sep 17, 2026, 08:12 AM UTC· resolved Sep 17, 2026, 11:25 AM UTC

1. identified Sep 17, 2026, 08:12 AM UTC

   Severity: Minor Status: Identified / Investigating Affected Component: Connection Graph (Production - EU) Impact: Users may be unable to view Connection Graph data for the past three days. Graphs generated during this window are not currently visible. No other functionality is affected. Description: We have identified a delay in the Connection Graph pipeline in our EU production environment, which began approximately three days ago. As a result, graph data from this period is not currently reflected. We are actively investigating the root cause and working on a resolution. We will post updates here as more information becomes available.

2. resolved Sep 17, 2026, 11:25 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 16, 2026, 12:08 PM UTC· resolved Sep 16, 2026, 03:19 PM UTC

1. investigating Sep 16, 2026, 12:08 PM UTC

   The Sandbox EU environment is experiencing degraded performance. We are working on resolution.

2. investigating Sep 16, 2026, 12:08 PM UTC

   We are continuing to investigate this issue.

3. resolved Sep 16, 2026, 03:19 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 04, 2026, 10:19 PM UTC· resolved Sep 05, 2026, 03:50 AM UTC

1. investigating Sep 04, 2026, 10:19 PM UTC

   We are currently investigating an issue affecting Advanced Aggregations, where some customers may experience inaccurate results. We will post updates as we learn more.

2. investigating Sep 04, 2026, 11:11 PM UTC

   We are continuing to investigate this issue

3. identified Sep 05, 2026, 12:38 AM UTC

   We have identified the root cause of the issue affecting Advanced Aggregations and our engineering team is actively working on a fix. We will provide a further update once the fix has been deployed and verified.

4. identified Sep 05, 2026, 01:49 AM UTC

   team is actively looking into it. Issue is partially mitigated, and we're still working on fixing the root cause.

5. monitoring Sep 05, 2026, 03:15 AM UTC

   fix is being validated and being rolled out

6. resolved Sep 05, 2026, 03:50 AM UTC

   incident is resolved. We'll provide post mortem.

7. postmortem Sep 08, 2026, 10:48 PM UTC

   ### Summary On September 4–5, 2026 \(UTC\), we experienced an issue affecting **Advanced Aggregations** **in which some customers saw inaccurate aggregation results because recent card transaction data may have been missing**. We fixed **the issue, and backfilled the data**. Advanced Aggregations results are expected to be correct again after **2026-09-05 03:50 UTC**. ### What happened A subset of card transaction records failed to be written into one of our tables because the column used to store the payment method id could not support the size of newly generated card IDs. As a result, those specific transactions were missing from the aggregated dataset, which could lead to incorrect results for customers ingesting this type of data. An impact assessment is available upon request. No data was permanently lost: failed writes were routed to a durable deadletter queue for later reprocessing. After mitigation was deployed, we replayed the queued events and completed a backfill so that aggregations reflected the correct underlying transaction data. Incident started at 2026-09-04 01:02 UTC and resolved at 2026-09-05 03:50 UTC. ### Why it happened The root cause was a **data type mismatch**: the `cards` table `id` values grew beyond the maximum value supported by a **4-byte integer**. `id` field has 8-byte integer so there was no error storing cards. However, `card_id` field in the Advanced Aggregations schema was 4-byte, causing inserts to fail for transactions related to newer cards. To remediate safely at scale, we: * Added new **8-byte integer \(**`bigint`\) columns for the affected identifiers and updated write logic to use them. * Introduced a **compatibility read path** so reads return the correct identifier whether data is stored in the old or new columns, along with new indexes to preserve query performance. ### What we are doing about it * We’re reviewing improvements to monitor schema limit risks earlier and prevent similar issues in the future. * We’ll enhance monitoring on deadletter queue so we can detect issue like this sooner.

Read the full incident report →

Started Jul 24, 2026, 12:42 AM UTC· resolved Jul 24, 2026, 02:07 AM UTC

1. investigating Jul 24, 2026, 12:42 AM UTC

   We are currently investigating this issue.

2. identified Jul 24, 2026, 12:58 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 24, 2026, 01:19 AM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Jul 24, 2026, 01:41 AM UTC

   New sessions are being shown OK, a few older ones might still be missing, they'll be shown shortly.

5. resolved Jul 24, 2026, 02:07 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 16, 2026, 12:40 PM UTC· resolved Jul 16, 2026, 02:25 PM UTC

1. investigating Jul 16, 2026, 12:40 PM UTC

   We are currently experiencing degraded performance in retrieving the latest customer sessions in the Customer Intelligence page. Root cause has been determined. We are working on resolution.

2. identified Jul 16, 2026, 12:40 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Jul 16, 2026, 12:41 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Jul 16, 2026, 12:41 PM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Jul 16, 2026, 01:13 PM UTC

   A fix has been implemented and results are being displayed in the Customer Intelligence page. We are monitoring the process until full service is restored.

6. monitoring Jul 16, 2026, 01:13 PM UTC

   We are continuing to monitor for any further issues.

7. resolved Jul 16, 2026, 02:25 PM UTC

   The issue has been resolved.

Read the full incident report →

Started Jul 15, 2026, 02:34 PM UTC· resolved Jul 15, 2026, 07:56 PM UTC

1. investigating Jul 15, 2026, 02:34 PM UTC

   We are currently investigating this issue.

2. resolved Jul 16, 2026, 02:36 AM UTC

   This incident has been resolved. A Postmortem will be published soon.

3. postmortem Jul 16, 2026, 02:49 AM UTC

   **Summary** On 2026-07-15, API requests experienced elevated error rates and increased tail latency \(p99\). Median latency \(p50\) was not affected and there was little effect on 95th percentile latency \(p95\), so most requests performed normally. We began investigating at 14:34 UTC and applied a fix at approximately 19:56 UTC, after which performance returned to normal. **What happened?** Through the affected period, a portion of API requests returned errors, responded with reason code SITO \(Sardine internal timeout\) or timed out rather than completing, with intermittent spikes during which a larger share of requests were affected. **Why did it happen?** A scaling configuration prevented part of our platform from adding capacity as traffic increased. As traffic rose through the day, available capacity was exhausted, causing timeouts and errors for some requests. **What are we doing about this?** We corrected the configuration at **19:56 UTC**, which immediately restored normal performance. We are also making the affected part of the platform more resilient to sudden traffic increases and adding proactive monitoring so we can detect and respond to this class of issue faster in future. We apologize for the disruption.

Read the full incident report →

Started Jun 25, 2026, 03:20 PM UTC· resolved Jun 25, 2026, 04:23 PM UTC

1. investigating Jun 25, 2026, 03:56 PM UTC

   We are currently investigating this issue.

2. resolved Jun 25, 2026, 06:13 PM UTC

   The incident regarding the latency increased has been resolved. It started at 15:20 UTC and lasting until 16:23 UTC. We'll include a postmortem into this status page in the future.

3. postmortem Jun 30, 2026, 06:51 PM UTC

   ### Summary Due to a database performance issue, API calls to `/v1/customer` , `/v1/issuing/risks` and `/v1/feedbacks` in June 25th faced increased latency. ### What happened One of the database we use to load certain customer and transaction data experienced slow latency for queries used in our customer and feedback APIs, causing latency issues on those APIs from 15:20 UTC to 16:23 UTC at June 25th. We've identified the root cause and are working on fixes to mitigate the issue. ### Why it happened One of our clients has very high volume and a particular usage pattern that didn't match well with our current database design for this data, causing the entire cluster to degrade in performance. ### What We’re Doing About It We are actively working on a solution to this issue, which includes moving the affected customer/transaction data to more performant databases, eliminating the risk altogether. In parallel we are also looking into actions to gracefully handle the usage patterns that cause degradation in the entire cluster.

Read the full incident report →

Started Jun 02, 2026, 05:43 PM UTC· resolved Jun 02, 2026, 06:34 PM UTC

1. investigating Jun 02, 2026, 05:43 PM UTC

   We are currently investigating this issue.

2. resolved Jun 02, 2026, 06:34 PM UTC

   This incident has been resolved.

3. postmortem Jun 08, 2026, 08:41 PM UTC

   ### Summary On June 2, 2026, some customers experienced elevated latency on `/v1/issuing/risks` and `/v1/customers` requests in `prod-eu`. We’re sorry for the disruption and for any delays this caused in customer risk decisions. ### What Happened A subset of issuing risk requests became slower than expected due to delayed customer data lookups. During the main impact window, affected requests could take up to approximately 5 seconds before continuing through a soft-fail path. ### Why It Happened The issue was caused by a customer lookup path not enforcing its intended short timeout. When some customer lookups stalled, they waited on the broader request deadline instead of failing fast at the operation-level timeout. ### What We’re Doing About It We deployed a hotfix to reduce the customer-load timeout and limit how long issuing requests can wait on stalled lookups. We are also adding follow-up improvements to reduce Datastore dependency in low-latency paths and investigate the underlying cause of the lookup stalls.

Read the full incident report →

Started May 22, 2026, 10:00 PM UTC· resolved May 22, 2026, 11:05 PM UTC

1. resolved May 22, 2026, 11:31 PM UTC

   From 10:22 UTC to to 11:05 UTC elevated latency could have affected some API requests.

2. postmortem May 26, 2026, 10:56 PM UTC

   #### Summary Our cache service experienced intermittent latency issues on May 22, 2026. Service has been fully restored. #### What Happened Our cache infrastructure experienced three distinct periods of elevated latency: * 22:23-22:31 UTC * 22:46-22:52 UTC * 23:00-23:05 UTC The system partially recovered between incidents but experienced cascading failures before full restoration at 23:05 UTC #### Why It Happened The incident began with an unusual spike in cache sets and cache deletes that stressed the caching infrastructure. #### What We're Doing About It * Implementing infrastructure improvements to prevent similar incidents resulting from noisy neighbor pattern * Enhancing cache monitoring, alerting and run books * Working with SME’s from our cloud provider to identify and address any contributing factors We apologize for the disruption and appreciate your patience.

Read the full incident report →

Started May 20, 2026, 09:48 AM UTC· resolved May 20, 2026, 10:07 AM UTC

1. investigating May 20, 2026, 09:48 AM UTC

   We are currently investigating the documentation unavailability issue.

2. resolved May 20, 2026, 10:07 AM UTC

   This has been resolved, documentation is available.

Read the full incident report →

Started May 12, 2026, 11:16 PM UTC· resolved May 13, 2026, 12:16 PM UTC

1. identified May 12, 2026, 11:16 PM UTC

   The issue has been identified and a fix is being implemented.

2. resolved May 13, 2026, 12:16 PM UTC

   This incident has been resolved.

3. postmortem May 15, 2026, 12:51 PM UTC

   **Impact:** During incident window * **Customer Intelligence Search** latency was degraded for queries spanning **>30 days** of data. * **Session Details** and **Customer Details** pages load were slow * **Connections Graph** and **Timeline** features were also impacted ## Executive Summary As part of infrastructure optimization, our development team performed multiple operations to our search databases to optimize index structure and data storage. This resulted in inefficient provision of our warm data cluster, and resulted in degraded performance. The team ultimately resolved the incident by updating data cluster configuration. Due to the volume of data, simple rollback was not possible, resulting in the long incident. ## Incident Details ### What Happened Our development team performed multiple operations to our search databases to optimize index structure and data storage. Due to bug in migration script, we migrated more data than initially anticipated. The destination cluster didn’t have sufficient storage and computing resources assigned. Latency started rising slowly as more data was migrated. This was initially dismissed as expected as we’re moving older data to separate clusters that are indeed slower but should remain within acceptable bounds. Two days later, on May 12, as the warm indices filled up as the migration completed, users began reporting that dashboard search was very slow. We then attempted upsizing the cluster but it was not able to upsize due to high traffic and large amount of data. Incident was resolved by our team manually reverted some of the operation. ## Timeline | Time \(PT, May 12\) | Event | | --- | --- | | **May 10, 23:38** | Automated operation around data migration was initiated, team was monitoring and didn’t report any issue | | **May 11, 00:00** | Latency starts climbing. Alerts were triggered but assumed as expected. | | **May 12, 6:02 AM** | Support reports dashboard slowness; on-call begins investigation | | **9:04 AM** | Incident formally created | | **10:56 AM** | First code fix deployed for customer details \+ session details | | **11:18 AM** | Deploy complete, pages still slow | | **12:25 PM** | Removed search dependency on Customer Profile \+ Session Details. Page Loads improved, Network Graph \+ Customer search still slow. | | **1:09 PM** | Root cause identified: indices incorrectly in warm tier; direct hot-tier migration initiated \(~10h estimated\) | | **3:05 PM** | Warm tier upsized aggressively migration still not converging | | **7:00–7:08 PM** | search cluster repeatedly auto-cancels in-flight shard recovery; direct migration abandoned | | **7:19 PM** | Switched to another approach of spinnig up new cluster | | **7:41 PM** | April indicies restored from snapshot; last-30d queries drop to ~15ms | | **9:03 PM** | February \+ March indicies restores complete | | **10:14 PM** | Replicas added to hot copies; search queue drops to 0. Incident resolved. | ## Action Items Immediate: * Manually rollback problematic resource allocation * Ensure all node pools have enough resources Medium Term Process Improvements: * Runbook and Migration process for search database upgrade operation * Better review process for Infra changes * Runbook for monitoring upgrade and immediate rollback * Observability in order to know if latency is expected

Read the full incident report →

Started May 12, 2026, 04:11 PM UTC· resolved May 12, 2026, 08:40 PM UTC

1. identified May 12, 2026, 04:11 PM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring May 12, 2026, 07:26 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 12, 2026, 08:40 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 06, 2026, 05:01 PM UTC· resolved May 06, 2026, 06:11 PM UTC

1. investigating May 06, 2026, 05:01 PM UTC

   We are currently investigating this issue.

2. investigating May 06, 2026, 05:15 PM UTC

   We are continuing to investigate this issue.

3. identified May 06, 2026, 05:40 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring May 06, 2026, 06:07 PM UTC

   We have mitigated the service degradation that began at approximately 16:48 UTC. Our team identified a failure in one of our database read replica instances on Google Cloud Platform and performed a manual failover to redirect traffic to a healthy replica with additional capacity. Error rates and latency have returned to normal levels as of approximately 17:57 UTC. The incident lasted approximately 69 minutes. We are actively working with our cloud provider to determine the root cause of the replica failure and prevent recurrence.

5. resolved May 06, 2026, 06:11 PM UTC

   This incident has been resolved.

6. postmortem May 06, 2026, 11:27 PM UTC

   **Summary** On May 6, 2026 from approximately 16:48 to 17:57 UTC, customers using Sardine's `/v1/customers` and `/v1/issuing/risks` APIs experienced elevated latency and degraded responses. We sincerely apologize for the disruption this caused. This document summarizes what happened, why it happened, and the steps we are taking to prevent recurrence. **What Happened** During this window, requests to the affected endpoints experienced one of two behaviors: * Elevated latency with a `SITO` reason code, indicating that Sardine was unable to compute certain risk signals within the expected timeframe. Customers still received rule evaluation results, but with limited signals. * For approximately 14% of `/v1/customers` traffic, requests returned HTTP 500 errors. The incident was resolved at approximately 17:57 UTC after our team rerouted database traffic to a healthy replicas. **Why It Happened** The root cause was an infrastructure failure in our cloud provider's \(Google Cloud\) database service in the US-central1 region. Internal resource shortage for certain instance types caused a routine automatic update operation on our primary read replica to fail. While Google Cloud UI and CLI reported instance to be healthy, database instances were not properly handling incoming queries. Our team performed a manual failover to redirect traffic to a healthy replicas, which restored service. **What We're Doing About It** We are taking the following actions to reduce the likelihood and impact of similar incidents: * Improved monitoring: We are adding alerts for failed database update operations and for "zombie" database states where an instance appears healthy but is not accepting queries. * Failover runbook: We are formalizing a documented procedure for read replica failover, including how to identify a failover target, resize a replica, update service configuration, and restart affected services. * Graceful degradation: We are investigating how to maintain partial service when a read replica is unavailable, rather than surfacing timeouts to customers. * Application timeout enforcement: We are reviewing and correcting how our services enforce database query timeouts to ensure failures surface quickly rather than hanging. * Database architecture review: We have scheduled a review with our cloud provider to evaluate high-availability configuration improvements and reduce our exposure to single-replica failure modes. We are also requiring a full root cause analysis from Google Cloud within 3 business days. We take the reliability of our platform seriously and apologize again for the impact this incident had on your operations. Please reach out to your account team or [[email protected]](mailto:[email protected]) if you have questions.

Read the full incident report →

Started Apr 22, 2026, 12:59 AM UTC· resolved Apr 22, 2026, 12:59 AM UTC

1. monitoring Apr 22, 2026, 12:14 AM UTC

   Apr 22, 00:14 UTC Investigating - We are currently investigating this issue.

Read the full incident report →

Started Mar 31, 2026, 04:31 PM UTC· resolved Mar 31, 2026, 04:31 PM UTC

1. monitoring Mar 31, 2026, 04:31 PM UTC

   Mar 31, 16:31 UTC Resolved - The issue has been resolved by the upstream provider, and operations have now returned to normal for Sanctions, PEP, and Adverse Media services. Mar 31, 15:41 UTC Investigating - We are currently investigating this issue

Read the full incident report →

Started Mar 17, 2026, 09:00 PM UTC· resolved Mar 17, 2026, 09:00 PM UTC

1. monitoring Mar 17, 2026, 09:00 PM UTC

   Mar 17, 21:00 UTC Completed - The scheduled maintenance has been completed. Mar 17, 20:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar 11, 13:04 UTC Scheduled - Upcoming Maintenance – No Expected Impact We will be performing scheduled database configuration changes during the following window. All services are expected to remain fully operational throughout the maintenance period. In the unlikely event of any disruption, we will post updates here immediately. EU: Mar 17th 20.00 UTC

Read the full incident report →

Started Mar 17, 2026, 11:08 AM UTC· resolved Mar 17, 2026, 11:08 AM UTC

1. monitoring Mar 17, 2026, 11:08 AM UTC

   Mar 17, 11:08 UTC Resolved - Issue has been resolved on March 16th 22:20 UTC Mar 16, 16:30 UTC Identified - The issue has been identified and the engineering team is working on a fix. A workaround fix is to clear cache and cookies.

Read the full incident report →

Started Mar 12, 2026, 01:14 AM UTC· resolved Mar 12, 2026, 01:14 AM UTC

1. monitoring Mar 12, 2026, 01:14 AM UTC

   Mar 12, 01:14 UTC Resolved - This incident has been resolved. Mar 11, 22:01 UTC Identified - Starting Mar 11, 2026 — 20:20:27.906 (UTC) we are facing an issue that doesn't allow the creation of Persistent Tags. Requests will go through fine but tags won't be created. Team identified root cause already and it's working on solving it. Persistent tags are also known as customer tags: https://docs.sardine.ai/guides/integration/developerresources/tags#using-customer-tags-in-the-dashboard This ONLY affects Production US, no other environment.

Read the full incident report →

Started Mar 04, 2026, 10:40 PM UTC· resolved Mar 04, 2026, 10:40 PM UTC

1. monitoring Mar 04, 2026, 10:40 PM UTC

   Mar 4, 22:40 UTC Resolved - This incident has been resolved. Mar 4, 22:15 UTC Identified - The issue has been identified and a fix is being implemented. Mar 4, 22:15 UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar 4, 22:09 UTC Investigating - Team is already working on mitigation and resolution

Read the full incident report →

Started Mar 02, 2026, 07:13 PM UTC· resolved Mar 02, 2026, 07:13 PM UTC

1. monitoring Mar 02, 2026, 07:13 PM UTC

   Mar 2, 19:13 UTC Resolved - This incident has been resolved. Mar 2, 18:29 UTC Update - We are continuing to work on a fix for this issue. Mar 2, 18:25 UTC Identified - The issue has been identified and a fix is being implemented.

Read the full incident report →

Started Feb 23, 2026, 09:00 PM UTC· resolved Feb 24, 2026, 12:23 AM UTC

1. resolved Feb 24, 2026, 01:13 AM UTC

   Some TaxID enrichments might have gotten an error while trying to perform them. You'll see STERR error code in the Risk Reason Codes widget of the session. Incident started 21:20 UTC and ended 00:23 UTC.

Read the full incident report →

Started Feb 19, 2026, 08:59 PM UTC· resolved Feb 19, 2026, 08:59 PM UTC

1. monitoring Feb 19, 2026, 08:59 PM UTC

   Feb 19, 20:59 UTC Resolved - This incident has been resolved. Feb 19, 16:50 UTC Identified - We are currently experiencing increased latency and timeouts due to service instability with our AI model provider (Google Gemini). Our team is monitoring the situation and working to restore normal performance.

Read the full incident report →

Started Feb 19, 2026, 04:57 PM UTC· resolved Feb 19, 2026, 04:57 PM UTC

1. monitoring Feb 19, 2026, 04:57 PM UTC

   Feb 19, 16:57 UTC Resolved - This incident has been resolved. Feb 19, 16:24 UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb 19, 14:56 UTC Identified - We are facing an issue that might cause sessions to NOT load in the dashboard. This does NOT affect risk assessment capabilities, it's simply a visual issue and our team is working on fixing it.

Read the full incident report →