Sapling Outage History
Sapling is up right nowPingoru has recorded 46 incidents for Sapling: 10 in the last 12 months and 2 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 0 | 0 of 720 (0%) |
| May 2026 | 0 | 0 of 744 (0%) |
| June 2026 | 1 | 4 of 720 (1%) |
| July 2026 | 1 | 0 of 744 (0%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 1 | 0 of 720 (0%) |
| October 2026 | 1 | 0 of 744 (0%) |
| November 2026 | 0 | 0 of 25 (0%) |
Incidents, newest first
Major October 8, 2026 Ongoing cluster upgrade 4m API, Backend, Website, Frontend Integration (Extensions)
Started Oct 08, 2026, 01:04 AM UTC· resolved Oct 08, 2026, 01:09 AM UTC
- investigating Oct 08, 2026, 01:04 AM UTC
We are currently investigating this issue.
- investigating Oct 08, 2026, 01:08 AM UTC
We are continuing to investigate this issue.
- resolved Oct 08, 2026, 01:09 AM UTC
This incident has been resolved.
Notice September 10, 2026 Sporadic downtime in previous hour due to database migration 13h API, Backend, Website, Frontend Integration (Extensions)
Started Sep 10, 2026, 11:30 AM UTC· resolved Sep 11, 2026, 12:30 AM UTC
- resolved Sep 11, 2026, 12:39 AM UTC
From 4:30 to 5:30pm PDT on Sep. 10, 2026, a database migration overloaded the production database with two other read-heavy jobs running at the same time. The issue has been identified and resolved.
Major July 6, 2026 Website unexpected instance reboot 6m Website
Started Jul 06, 2026, 05:18 PM UTC· resolved Jul 06, 2026, 05:25 PM UTC
- identified Jul 06, 2026, 05:18 PM UTC
Website server needed unexpected reboot
- resolved Jul 06, 2026, 05:25 PM UTC
This incident has been resolved.
Major June 15, 2026 /edits model downtime 3h 35m API, Backend, Frontend Integration (Extensions)
Started Jun 15, 2026, 01:47 AM UTC· resolved Jun 15, 2026, 05:23 AM UTC
- monitoring Jun 15, 2026, 01:47 AM UTC
Upgrade of model resulted in unexpected downtime due to mis-specified model ID
- monitoring Jun 15, 2026, 01:58 AM UTC
We are continuing to monitor for any further issues.
- resolved Jun 15, 2026, 05:23 AM UTC
This incident has been resolved.
Critical January 25, 2026 linkerd expired anchors led to cluster downtime 3h
Started Jan 25, 2026, 12:40 PM UTC· resolved Jan 25, 2026, 03:40 PM UTC
- resolved Jan 25, 2026, 12:30 PM UTC
From 4:40am to 7:40am PST on Jan 25, 2026, the cluster experienced downtime due to expired trust anchors. New certs have been generated.
Major January 16, 2026 Chrome 144 breaking change for runtime.onMessage 8d 16h Frontend Integration (Extensions)
Started Jan 16, 2026, 11:24 PM UTC· resolved Jan 25, 2026, 03:46 PM UTC
- investigating Jan 16, 2026, 11:24 PM UTC
We are currently patching this issue -- the extension will not work in Chrome v144
- investigating Jan 16, 2026, 11:38 PM UTC
We are continuing to investigate this issue.
- monitoring Jan 17, 2026, 06:53 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jan 19, 2026, 07:20 AM UTC
We are continuing to monitor for any further issues.
- resolved Jan 25, 2026, 03:46 PM UTC
This has been resolved for over 6 days with the new integration. We left the message so those with the older integration would update.
Major December 30, 2025 /edits and /spellcheck APIs team_id argument error –
Started Dec 30, 2025, 11:00 PM UTC· resolved Dec 30, 2025, 11:00 PM UTC
- resolved Dec 30, 2025, 11:00 PM UTC
This incident has been resolved.
Major November 26, 2025 AI detect downtime –
Started Nov 26, 2025, 11:00 AM UTC· resolved Nov 26, 2025, 11:00 AM UTC
- resolved Nov 26, 2025, 11:00 AM UTC
AWS cluster upgrades led to insufficient AI detection pod assignment, leading to latency and then downtime starting from 10:30am PST.
Critical November 11, 2025 Downtime due to new AWS AMI upgrade 35m
Started Nov 11, 2025, 03:30 PM UTC· resolved Nov 11, 2025, 04:05 PM UTC
- resolved Nov 11, 2025, 10:00 AM UTC
AWS no longer supported older AMI, hence forcing upgrade. However, this resulted in certain nodegroups changing and pods not beeing assigned.
- postmortem Nov 11, 2025, 06:35 PM UTC
AWS announced end of support for AMIs that Sapling used in its EKS cluster. ``` Amazon EKS will end support for EKS optimized AL2 AMIs on November 26, 2025. To avoid any disruptions, please migrate your workloads to alternative AMI types before November 26, 2025. ``` Around midnight Pacific Time on November 10, 2025, Sapling’s Helm configuration was updated with the updated AMIs. No disruption was observed. However, around 2am Pacific Time on November 11, 2025, the main backend K8S pods on Sapling were unable to be assigned to new nodes due to node affinity settings. This resulted in requests timing out and failing, affecting API, extension, and website users. Around 7:30am Pacific Time, the AMI update was fixed and the instances were updated. With the redeployment of the existing pods as well as the Linkerd mesh, most services were restored around 7:45am Pacific Time, with the remaining services up and running around 8:05am PT. Future updates of the AMI will be performed shortly before midnight Friday time, with monitoring to see if pods are assigned to new instances without any node affinity issues.
Minor November 7, 2025 AI detector pods overloaded 4h 36m API, Backend, Website
Started Nov 07, 2025, 11:24 PM UTC· resolved Nov 08, 2025, 04:01 AM UTC
- monitoring Nov 07, 2025, 11:24 PM UTC
Limited resources in AWS region have led to insufficient AI detector pods since 11/5
- resolved Nov 08, 2025, 04:01 AM UTC
This incident has been resolved.
Minor October 8, 2025 Spellcheck latency 14m API, Backend
Started Oct 08, 2025, 06:33 PM UTC· resolved Oct 08, 2025, 06:47 PM UTC
- monitoring Oct 08, 2025, 06:33 PM UTC
We have seen a surge in spellcheck usage in the past day resulting in some scaling of the cluster components needed.
- resolved Oct 08, 2025, 06:47 PM UTC
This incident has been resolved.
Notice October 7, 2025 Spellcheck API / service request failures – API, Backend
Started Oct 07, 2025, 05:15 PM UTC· resolved Oct 07, 2025, 05:15 PM UTC
- resolved Oct 07, 2025, 05:15 PM UTC
Subset of pods failing to connect to caching system after caching system upgrade led to degraded services
Major September 27, 2025 High latency – API, Backend, Website
Started Sep 27, 2025, 03:45 PM UTC· resolved Sep 27, 2025, 03:45 PM UTC
- resolved Sep 27, 2025, 03:45 PM UTC
Delays when writing to S3 resulted in latency spikes across endpoints.
Major April 30, 2025 Increased multilingual usage leading to some timeouts –
Started Apr 30, 2025, 02:32 AM UTC· resolved Apr 30, 2025, 02:32 AM UTC
- resolved Apr 30, 2025, 02:32 AM UTC
A fix has been implemented and we are monitoring the results.
- postmortem Apr 30, 2025, 05:38 AM UTC
Increased load to multilingual handling as well as increased latency when sending Amplitude events led to degraded latency for other endpoints.
Major April 16, 2025 Latency spike –
Started Apr 16, 2025, 05:30 AM UTC· resolved Apr 16, 2025, 05:30 AM UTC
- resolved Apr 16, 2025, 05:30 AM UTC
A shortage of available pods resulted in latency spike starting around 10:20 PDT
Major February 20, 2025 Increased latency –
Started Feb 20, 2025, 03:00 PM UTC· resolved Feb 20, 2025, 03:00 PM UTC
- resolved Feb 20, 2025, 03:00 PM UTC
Increase in latency due to upgrades to libraries for error classification pipeline.
Minor November 20, 2024 Cluster capacity reached 33m API, Backend, Website
Started Nov 20, 2024, 08:03 PM UTC· resolved Nov 20, 2024, 08:36 PM UTC
- monitoring Nov 20, 2024, 08:03 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 20, 2024, 08:36 PM UTC
This incident has been resolved.