Cycode Outage History

Started Nov 04, 2026, 11:38 PM UTC· resolved Nov 04, 2026, 11:41 PM UTC

1. investigating Nov 04, 2026, 10:38 PM UTC

   Some customers experienced delayed pull request scans between 22:10 and 22:32 UTC on November 4. Services have been restored and the scan backlog has cleared; we are investigating the underlying database connectivity issue.

2. resolved Nov 04, 2026, 11:24 PM UTC

   The issue has been fully resolved and all pending scans have completed. We are investigating the underlying database connectivity issue and will publish a root cause analysis on this page once it’s complete.

Read the full incident report →

Started Oct 27, 2026, 10:06 AM UTC· resolved Oct 27, 2026, 12:51 PM UTC

1. investigating Oct 27, 2026, 10:06 AM UTC

   Summary Some Bitbucket Cloud integrations that use OAuth disconnected after Bitbucket’s authentication service returned errors on October 27, 2026, at 03:45 UTC. Impact For affected integrations, Cycode cannot access Bitbucket. Scans, repository syncs, and commit statuses for those workspaces stop until the integration is authorized again. Integrations that use app passwords, API tokens, or workspace access tokens are not affected. Latest update Our investigation shows that Bitbucket issued new credentials for these integrations but did not deliver them to Cycode, because its service returned an error. Bitbucket no longer accepts the previous credentials, so the integrations cannot recover automatically. We are reporting the issue to Atlassian. Action required If your Bitbucket integration shows as disconnected, authorize it again in the Cycode console under [Settings > Integrations > Bitbucket]. Cycode continues from where it stopped, and no other change is necessary. We will post an update when we have more information.

2. monitoring Oct 27, 2026, 10:46 AM UTC

   We identified every affected integration and are now contacting their users directly. An integration starts to work again as soon as its user authorizes it again in the Cycode console under [Settings > Integrations > Bitbucket]. We reported the Bitbucket error to Atlassian and will post an update when we have more information.

3. resolved Oct 27, 2026, 12:51 PM UTC

   All affected Bitbucket Cloud integrations were identified, and their users received instructions to authorize them again. Integrations that were authorized again work normally. The root cause was a Bitbucket service error on October 27, 2026, at 03:45 UTC, which invalidated the credentials of some integrations. We shared the details with Atlassian for their investigation.

Read the full incident report →

Started Oct 20, 2026, 09:28 AM UTC· resolved Oct 20, 2026, 10:04 AM UTC

1. investigating Oct 20, 2026, 09:28 AM UTC

   We’ve identified a potential cause of the login flow issues and are investigating further.

2. investigating Oct 20, 2026, 09:54 AM UTC

   We’ve resolved the login flow issue in the EU region.

3. monitoring Oct 20, 2026, 10:04 AM UTC

   The issue has been resolved, and we’re continuing to monitor the system to ensure stability.

4. resolved Oct 20, 2026, 03:43 PM UTC

   The incident has been fully resolved, and we see no further issues.

Read the full incident report →

Started Oct 18, 2026, 01:10 PM UTC· resolved Oct 18, 2026, 02:40 PM UTC

1. investigating Oct 18, 2026, 01:10 PM UTC

   We are investigating errors in the application related to PR scanning and violation load failures. We will keep posting further updates once we found the cause.

2. identified Oct 18, 2026, 01:26 PM UTC

   We had an issue with one of our application databases. The database is back to normal operations, but is still degraded, and we are actively working on fixing it. The platform is back to normal operations, and violations and PR scans should work normally.

3. monitoring Oct 18, 2026, 01:34 PM UTC

   The database is back to normal operations. We are actively monitoring to ensure there are no remaining issues.

4. resolved Oct 18, 2026, 02:40 PM UTC

   The incident has been fully resolved, and we see no further issues.

Read the full incident report →

Started Oct 13, 2026, 08:30 AM UTC· resolved Oct 13, 2026, 11:27 AM UTC

1. investigating Oct 13, 2026, 07:20 AM UTC

   GitLab is currently experiencing a service disruption that may impact Cycode PR scanning and integration connectivity. The issue originates from GitLab's infrastructure and is outside of Cycode's control. We are monitoring the situation and will provide updates as GitLab works to restore normal service. For more information, please refer to GitLab's status page: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/6acdd94de3aa7005ae52c614

2. monitoring Oct 13, 2026, 08:47 AM UTC

   GitLab has moved the incident affecting PR scanning and integration connectivity to the monitoring phase. GitLab has implemented a fix and is monitoring the results. We are continuing to monitor the impact on Cycode integrations and will provide further updates if needed. For more information, please refer to GitLab's status page: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/6acdd94de3aa7005ae52c614

3. resolved Oct 13, 2026, 11:27 AM UTC

   GitLab has resolved the incident affecting PR scanning and integration connectivity

Read the full incident report →

Started Oct 13, 2026, 07:58 AM UTC· resolved Oct 13, 2026, 08:28 AM UTC

1. monitoring Oct 13, 2026, 08:30 AM UTC

   Monitoring: scans API (US region) Earlier today our alerting system detected that the service behind the scans API was restarting more often than expected, which caused some requests to fail for a short period. We increased the memory available to the service, and it has been stable since. We are keeping the service under close observation while we finalize a permanent improvement.

2. resolved Oct 13, 2026, 09:00 AM UTC

   Resolved: Brief slowdown of the scans API (US region) On October 13, 2026, between 07:58 and 08:28 UTC, some requests to the scans API in the US region failed or were slow. Scan pages in the Cycode UI and the Cycode CLI may have shown temporary errors, and some pull request scans arrived a few minutes later than usual. We increased the memory available to the service, which fully resolved the issue at 08:34 UTC. All queued scans completed shortly after, and no data was lost. We have identified the cause and a permanent improvement is in review.

Read the full incident report →

Started Oct 05, 2026, 01:24 PM UTC· resolved Oct 05, 2026, 08:24 PM UTC

1. investigating Oct 05, 2026, 01:24 PM UTC

   We have noticed degraded performance in prod-us Flows like Pull Request or CLI scanning can be slower than usual

2. investigating Oct 05, 2026, 02:17 PM UTC

   We’re continuing to investigate a major degradation in the US platform caused by PostgreSQL connection exhaustion. Customers may experience slow response times, elevated 500 errors, and slower PR workflows and CLI scans. We’ve reduced database load across several services and initiated an RDS failover. The team is actively working to restore normal performance.

3. investigating Oct 05, 2026, 02:29 PM UTC

   The team is actively working with AWS and investigating what appears to be a database failover issue affecting the US environment. We’ll share another update as soon as we have more information.

4. investigating Oct 05, 2026, 03:17 PM UTC

   The team continues to work with AWS, which escalated the issue internally. In parallel, we are working to restore functionality by moving to another database. Any blocked pull requests waiting for Cycode status can be unblocked by removing Cycode scans from expected checks directly in your SCM (Source Code Management). Reach out to Your Cycode representative to help with streamlining the process.

5. investigating Oct 05, 2026, 04:10 PM UTC

   AWS has confirmed that the affected PostgreSQL instance is currently in an internal recovery state. The expected Multi-AZ failover did not occur as designed, and AWS is actively investigating why. In parallel, we are continuing work to restore the affected services using an alternative database path to accelerate recovery

6. investigating Oct 05, 2026, 04:58 PM UTC

   We are seeing database is restoring its operational capabities. We see first Pull Request scans being processed. We are monitoring the system and increasing the throughput.

7. investigating Oct 05, 2026, 05:31 PM UTC

   Database remains fully operational. We are working on increasing processing speed beyond regular to process all backlogs as soon as possible.

8. investigating Oct 05, 2026, 06:17 PM UTC

   The US platform is improving following the earlier RDS/PostgreSQL connection exhaustion. AWS has confirmed that the original PostgreSQL instance has returned to an operational state, and we’re seeing PR scans being processed again. The main remaining degradation is delayed GitHub-driven processing due to a large webhook backlog, which is continuing to drain. We’ll continue monitoring recovery and provide another update as processing returns to normal

9. investigating Oct 05, 2026, 07:06 PM UTC

   Last remaining backlog continues being processed at a steady rate. Our estimations show that last parts backlog should be fully processed in about 60-90 minutes (20:30 UTC). We see the majority of pull request scans (60% at the moment of this update) are being completed in below 15 minutes. and this number is constantly growing at a steady rate. Other than that system looks operational. We believe application/UI are fully operational.

10. investigating Oct 05, 2026, 08:09 PM UTC

    The system is back to being fully operational

11. resolved Oct 05, 2026, 08:24 PM UTC

    (2/2) • Paused the workloads that were creating high volumes of short-lived data operations, to reduce database load while delayed events cleared. • Scanned the pull requests that were missed during the incident. 4b. Action Items • Change how large batches of data are written to reduce database load. • Move our highest-volume workload to its own dedicated database, so a problem there can't affect the rest of the platform. • Add alerts and deeper monitoring for database connection usage and database health. • Give each service its own dedicated database credentials. • Process incoming events more efficiently to speed up recovery after a disruption. • Improve monitoring of processing delays and of internal event alerts. • Review database capacity and performance settings to add more headroom.

12. resolved Oct 05, 2026, 08:24 PM UTC

    (1/2) 1. Summary On October 5, 2026, starting at 15:55 IDT, customers on our US platform saw slow responses, intermittent errors, and delays in pull request checks and CLI scans. Some customers whose repositories require Cycode pull request checks couldn't merge until scanning resumed. The issue started when a background maintenance process in our primary database fell behind and stopped completing, and the database stopped saving new data. Restoring service took several hours because the standby database first had to process a large backlog of changes. The database was back and the platform was available again by 19:43 IDT. Delayed processing then caught up, and the incident was resolved at 23:24 IDT. Pull requests that were missed during the incident were scanned. 2. Key Timeline (IDT) • October 5, 2026, 12:55 IDT: A routine database maintenance process starts running much slower than normal. No customer impact yet. • October 5, 2026, 14:53 IDT: The next maintenance cycle starts and doesn't complete. • October 5, 2026, 15:55 IDT: The database stops saving new data, and platform performance degrades. • October 5, 2026, 16:03 IDT: The database reaches its connection limit. Customers start seeing errors. • October 5, 2026, 16:21 IDT: Incident declared and response team engaged. • October 5, 2026, 16:24 IDT: First update posted to the status page. • October 5, 2026, 16:49 IDT: Switch to the standby database started. • October 5, 2026, 17:00 IDT: Support case opened with our cloud infrastructure provider. • October 5, 2026, 18:24 IDT: Alternative recovery path started in parallel. Workaround published for customers blocked by required pull request checks. • October 5, 2026, 19:43 IDT: Database available again, and the Cycode API and Cycode Platform are accessible. • October 5, 2026, 19:54 IDT: Pull request scanning resumes. • October 5, 2026, 20:20 IDT: Processing capacity increased to clear delayed events. • October 5, 2026, 20:45 IDT: High-volume workloads paused to reduce database load. • October 5, 2026, 21:56 IDT: Pull requests missed during the incident identified and scanned. • October 5, 2026, 23:09 IDT: All services fully operational. • October 5, 2026, 23:24 IDT: Incident resolved. 3. Root Cause Our database runs a routine background process that regularly saves recent changes to permanent storage. Two conditions contributed to that process slowing down: • High-volume short-lived data operations: some of our processing workloads were constantly creating and removing large numbers of short-lived data structures. • A delayed background sync: a data-sync process had fallen far behind, which stopped the database from cleaning up after itself. The maintenance process slowed down and eventually stopped completing. The database could then no longer save new data, and requests started to queue. Our services retried those requests and opened many new connections, which used up the database's connection capacity. Customers saw this as errors and slow responses. We switched to our standby database. Before it could serve traffic, it had to process all changes made since the last completed maintenance cycle, about four hours of activity. That extended the outage. 4. Actions Taken • Reduced load on the database by temporarily scaling down non-essential processing. • Started a switch to the standby database. • Opened a support case with our cloud infrastructure provider and worked with them on an escalated basis. • Prepared an alternative recovery path in parallel. • Shared a temporary workaround for customers blocked by required pull request checks. • Once the database recovered, increased processing capacity to clear delayed events.

Read the full incident report →

Started Sep 28, 2026, 07:08 PM UTC· resolved Sep 28, 2026, 08:45 PM UTC

1. monitoring Sep 28, 2026, 07:08 PM UTC

   At 2:00 PM EDT (18:00 UTC), we identified a significant load spike affecting the authentication database. To mitigate the issue, we performed an instance upgrade. A brief interruption to database connections may have occurred during the upgrade. The platform is stable, and we continue to closely monitor the situation.

2. resolved Sep 28, 2026, 08:45 PM UTC

   The system is fully operational, with no ongoing issues observed. This incident is now resolved.

Read the full incident report →

Started Sep 13, 2026, 10:36 AM UTC· resolved Sep 13, 2026, 06:53 PM UTC

1. investigating Sep 13, 2026, 10:36 AM UTC

   GitHub component: Pull Requests Original GitHub incident: https://www.githubstatus.com/incidents/0rn90wk115q9

2. resolved Sep 13, 2026, 06:53 PM UTC

   Resolved

Read the full incident report →

Started Sep 08, 2026, 04:53 PM UTC· resolved Sep 08, 2026, 05:26 PM UTC

1. investigating Sep 08, 2026, 04:53 PM UTC

   We have noticed degraded performance in prod-us Flows like Pull Request or CLI scanning can be slower than usual

2. monitoring Sep 08, 2026, 05:21 PM UTC

   We observed a sharp spike in MemoryDB memory utilization in prod-us, including active swap usage. Mitigation has been applied. MemoryDB metrics have remained stable since then, and the processing lag is decreasing quickly

3. resolved Sep 08, 2026, 05:26 PM UTC

   MemoryDB metrics are stable, the processing lag has recovered, and the system is now fully operational.

Read the full incident report →

Started Sep 08, 2026, 09:53 AM UTC· resolved Sep 08, 2026, 10:52 AM UTC

1. investigating Sep 08, 2026, 09:53 AM UTC

   We have noticed degraded performance in prod-eu. Performance degradation is affecting all application and all flows (like Pull Request or CLI scanning) can be slower than usual. Our initial investigation suggest excessive load on the shared database, making reading and writing to it slower. The team is continuing to investigate.

2. resolved Sep 08, 2026, 10:52 AM UTC

   The system is back to being fully operational.

Read the full incident report →

Started Sep 01, 2026, 03:10 PM UTC· resolved Sep 01, 2026, 04:05 PM UTC

1. investigating Sep 01, 2026, 03:10 PM UTC

   GitHub component: Pull Requests Original GitHub incident: https://stspg.io/0xbhhq84v4mt

2. resolved Sep 01, 2026, 04:05 PM UTC

   GitHub component: Pull Requests Original GitHub incident: https://stspg.io/0xbhhq84v4mt

Read the full incident report →

Started Aug 26, 2026, 11:04 PM UTC· resolved Aug 27, 2026, 12:30 AM UTC

1. investigating Aug 26, 2026, 11:04 PM UTC

   GitHub component: Pull Requests Original GitHub incident: https://stspg.io/nrbwjftcz72d

2. investigating Aug 26, 2026, 11:59 PM UTC

   GitHub component: Pull Requests Original GitHub incident: https://stspg.io/nrbwjftcz72d

3. resolved Aug 27, 2026, 12:30 AM UTC

   GitHub component: Pull Requests Original GitHub incident: https://stspg.io/nrbwjftcz72d

Read the full incident report →

Started Aug 20, 2026, 02:52 PM UTC· resolved Aug 24, 2026, 12:53 PM UTC

1. investigating Aug 20, 2026, 02:52 PM UTC

   We’re currently experiencing some delays in GitHub scan processing. Our team is investigating the issue and working to restore normal processing times.

2. identified Aug 20, 2026, 03:48 PM UTC

   GitHub scan processing and status-check updates remain delayed, affecting all scan types. We have made changes to improve processing capacity and are monitoring recovery as the backlog decreases.

3. monitoring Aug 20, 2026, 04:04 PM UTC

   Scan processing delays have been draining quickly, and most scans have not been affected. A small number of pull requests may still experience delayed scan processing. We’re continuing to monitor recovery and will share another update as we learn more

4. resolved Aug 20, 2026, 04:36 PM UTC

   Summary Problem: We observed a delay in processing status updates for GitHub checks due to a backlog in our event processing system. Impact: A small number of pull requests (PRs) experienced delays in status check updates. While the majority of the system remained functional, a specific segment of the processing queue was affected, causing a lag for those specific updates over several hours. Cause: The delay was caused by a high volume of retries generated by unrecoverable permission errors on a limited number of requests, which blocked a portion of our processing capacity. This occurred during a period of high event volume, which led to the database reaching maximum capacity and slowing down the overall processing workflow. Resolution: The issue has been resolved. We scaled up our database capacity, increased the number of processing units to handle the higher volume, and implemented logic to stop retrying updates that fail due to permanent permission errors. All backlogs have been cleared, and system performance has returned to normal for all pull requests. Key Timeline (IDT) • August 21, 2026, 06:30 IDT: An incident was declared as delays in status updates for a subset of requests became visible. • August 21, 2026, 06:50 IDT: A public status page was published to inform customers of the degradation. • August 21, 2026, 07:15 IDT: The database was scaled up to address processing bottlenecks. • August 21, 2026, 07:45 IDT: Processing capacity was doubled by increasing the number of active queue partitions. • August 21, 2026, 08:10 IDT: A corrective update was deployed to stop unnecessary retries on unrecoverable permission errors. • August 21, 2026, 08:45 IDT: The event backlog was fully drained, and processing returned to real-time. • August 21, 2026, 09:00 IDT: The incident was officially resolved. Root Cause The incident was caused by a combination of high event volume and a lack of "fail-fast" logic for permanent errors. Specific requests with incorrect permission settings generated a massive volume of retry attempts. Because the system treated these permanent permission errors as temporary, it repeatedly retried them, which blocked a specific processing partition and prevented updates for a small number of pull requests from being processed. This was further exacerbated by the database reaching its resource limits, creating a temporary slowdown in the update workflow. Actions Taken • Scaled Database Resources: Increased the capacity of the primary database to handle the increased load and improve processing speed. • Increased Processing Parallelism: Doubled the number of active partitions in the event queue to ensure all available processing units were utilized. • Deployed Retry Exclusion Logic: Updated the system to identify unrecoverable permission errors and stop retrying them, preventing queue blockages.

5. investigating Aug 24, 2026, 11:30 AM UTC

   We’re currently experiencing some delays in GitHub scan processing. Our team is investigating the issue and working to restore normal processing times.

6. resolved Aug 24, 2026, 12:53 PM UTC

   The system is back to being fully operational.

Read the full incident report →

Started Aug 19, 2026, 01:18 PM UTC· resolved Aug 19, 2026, 06:06 PM UTC

1. investigating Aug 19, 2026, 01:18 PM UTC

   Cycode experiences increased response time for selected requests. We're investigating the issue.

2. monitoring Aug 19, 2026, 01:28 PM UTC

   System response times are back to normal. We're monitoring system state.

3. investigating Aug 19, 2026, 02:16 PM UTC

   We're investigating potential issue with one of the infrastructural services with infra provider support team.

4. identified Aug 19, 2026, 03:54 PM UTC

   Some customers may continue to experience slower response times across the Cycode application. This can include delays loading pages, viewing data, and starting or completing pull request scans. We have identified instability in an underlying data service that is affecting request processing. Our infrastructure provider is actively working to restore the affected service; recovery has not yet completed successfully. We have applied temporary changes to reduce demand on the service while we monitor its stability and work with the provider on a permanent resolution. We will share another update as soon as we have confirmed service stability or have additional information to report.

5. monitoring Aug 19, 2026, 04:02 PM UTC

   We are monitoring stability of the affected memory database service. It currently appears stable, but our provider has not yet confirmed full recovery. We will share another update once recovery is confirmed.

6. monitoring Aug 19, 2026, 04:25 PM UTC

   The Cycode application is stable and page loading is operating normally. Some pull request scans are still experiencing delays. We are continuing to monitor the service and work to clear the remaining scan backlog.

7. monitoring Aug 19, 2026, 05:00 PM UTC

   The Cycode application remains stable and page loading is operating normally. Some pull request scans may still be delayed while we continue clearing the remaining backlog and monitor recovery.

8. monitoring Aug 19, 2026, 05:32 PM UTC

   The remaining processing delays have cleared, and the Cycode application is operating normally. We are continuing to monitor the service.

9. resolved Aug 19, 2026, 06:06 PM UTC

   1. Summary On August 19, 2026, some customers experienced slower page loading and data display, as well as delays when starting or completing pull request scans. The disruption was caused by a managed memory database entering a repeated restart cycle and not recovering automatically. This created a temporary slowdown in processing. Expected service behavior was restored after corrective actions, including replacement and increased capacity for the affected service. The provider is continuing to investigate why the service was unable to return to normal operation automatically. 2. Key Timeline (IDT) • August 19, 2026, 16:10 IDT: We identified performance degradation affecting parts of the platform and pull request scanning. • August 19, 2026, 16:30 IDT: An initial corrective change was applied and service behavior was monitored. • August 19, 2026, 16:46 IDT: We confirmed that some delays were continuing and expanded the investigation. • August 19, 2026, 17:14 IDT: We began working with our service provider to investigate instability in the managed memory database. • August 19, 2026, 18:46 IDT: Recovery actions were in progress, with temporary mitigations in place to reduce customer impact. • August 19, 2026, 19:06 IDT: The managed memory database capacity update completed, and platform responsiveness and pull request scanning returned to expected behavior. 3. Root Cause The managed memory database entered a repeated restart cycle and was unable to recover automatically. This caused delays in the processing systems that support platform responsiveness and pull request scans. The affected service underwent a routine replacement process, followed by a capacity increase during the recovery effort. While the capacity increase completed successfully and restored expected behavior, AWS is still researching why the service did not come back up normally after the restart cycle. Their detailed root-cause analysis is pending. 4. Actions Taken • Applied corrective updates to reduce immediate processing demand. • Temporarily adjusted the status update process to reduce reliance on the affected service. • Restarted affected processing components to restore responsiveness. • Worked with the service provider to replace affected capacity and complete a capacity increase. • Monitored platform performance and scan processing until expected behavior was confirmed. 4b. Action Items • Review the service provider’s detailed root-cause analysis when available. • Improve monitoring to detect similar recovery delays earlier. • Add safeguards to reduce the impact of temporary processing slowdowns. • Introduce separate Redis capacity for each service to isolate workloads and reduce the impact of an issue in one service on others.

Read the full incident report →

Started Aug 18, 2026, 07:35 AM UTC· resolved Aug 18, 2026, 08:31 AM UTC

1. investigating Aug 18, 2026, 07:35 AM UTC

   Some GitHub pull request status checks in the EU region were delayed by up to 30 minutes before completing. No checks were lost and no action is needed - delayed checks completed automatically.

2. identified Aug 18, 2026, 08:10 AM UTC

   Root cause identified.

3. monitoring Aug 18, 2026, 08:11 AM UTC

   Issue was fixed. Monitoring closely.

4. resolved Aug 18, 2026, 08:31 AM UTC

   Some GitHub pull request status checks in the EU region were delayed by up to 30 minutes before completing. No checks were lost and no action is needed - delayed checks completed automatically. We identified the cause, and fixed it. We're monitoring closely.

Read the full incident report →

Started Aug 17, 2026, 01:44 PM UTC· resolved Aug 17, 2026, 09:21 PM UTC

1. investigating Aug 17, 2026, 01:44 PM UTC

   GitHub component: Webhooks, API Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

2. investigating Aug 17, 2026, 01:45 PM UTC

   GitHub component: Webhooks, API Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

3. investigating Aug 17, 2026, 01:47 PM UTC

   GitHub component: Webhooks, API Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

4. investigating Aug 17, 2026, 01:50 PM UTC

   GitHub component: Webhooks, API Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

5. investigating Aug 17, 2026, 01:51 PM UTC

   GitHub component: Webhooks, API Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

6. investigating Aug 17, 2026, 02:28 PM UTC

   GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

7. investigating Aug 17, 2026, 02:34 PM UTC

   GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

8. investigating Aug 17, 2026, 02:35 PM UTC

   GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

9. investigating Aug 17, 2026, 02:50 PM UTC

   GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

10. investigating Aug 17, 2026, 02:57 PM UTC

    GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

11. investigating Aug 17, 2026, 02:59 PM UTC

    GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

12. investigating Aug 17, 2026, 03:04 PM UTC

    GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

13. investigating Aug 17, 2026, 03:04 PM UTC

    GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

14. investigating Aug 17, 2026, 03:06 PM UTC

    GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

15. investigating Aug 17, 2026, 03:07 PM UTC

    GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

16. investigating Aug 17, 2026, 03:14 PM UTC

    GitHub component: Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

17. investigating Aug 17, 2026, 03:27 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

18. investigating Aug 17, 2026, 03:29 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

19. investigating Aug 17, 2026, 03:31 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

20. investigating Aug 17, 2026, 03:47 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

21. investigating Aug 17, 2026, 04:04 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

22. investigating Aug 17, 2026, 04:41 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

23. investigating Aug 17, 2026, 04:53 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

24. investigating Aug 17, 2026, 05:05 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

25. investigating Aug 17, 2026, 06:00 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

26. investigating Aug 17, 2026, 06:21 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

27. investigating Aug 17, 2026, 06:25 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

28. investigating Aug 17, 2026, 06:40 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

29. investigating Aug 17, 2026, 06:55 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

30. investigating Aug 17, 2026, 07:11 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

31. investigating Aug 17, 2026, 07:22 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

32. investigating Aug 17, 2026, 07:42 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

33. investigating Aug 17, 2026, 08:16 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

34. investigating Aug 17, 2026, 08:31 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

35. investigating Aug 17, 2026, 08:51 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

36. resolved Aug 17, 2026, 09:21 PM UTC

    GitHub component: Git Operations, Webhooks, API Requests, Pull Requests Original GitHub incident: https://stspg.io/y1fl26l6wpzr

Read the full incident report →

Started Aug 14, 2026, 01:41 PM UTC· resolved Aug 14, 2026, 04:14 PM UTC

1. investigating Aug 14, 2026, 01:41 PM UTC

   We are investigating degraded performance affecting Secrets scanning for Pull Requests in the US environment. Our team is actively working to identify the root cause.

2. identified Aug 14, 2026, 02:26 PM UTC

   We have identified the root cause of the issue and are actively working on a fix. In the meantime, affected PR Secret detections will not be marked as stuck.

3. monitoring Aug 14, 2026, 03:02 PM UTC

   The backlog of PR Secrets scans has been fully processed, and scan latency has returned to normal levels. The issue was caused by elevated latency in our AI model-serving infrastructure. We have applied a mitigation and are continuing to monitor the system. We temporarily disabled PR Secrets scanning between 4:24 PM and 4:50 PM CEST and allowed the affected checks to pass to prevent PRs from being blocked. During this period, PR Secrets scans were not created and therefore will not appear on the PR Scans page. PR Secrets scanning has now been re-enabled.

4. monitoring Aug 14, 2026, 03:38 PM UTC

   PR Secrets scanning is operating normally, the backlog has been fully processed, and scan latency has returned to expected levels. The incident was caused by elevated latency in our AI model-serving infrastructure. A mitigation has been applied, and we will continue monitoring the system to ensure stability.

5. resolved Aug 14, 2026, 04:14 PM UTC

   (2/2) • Add a graceful degradation path for AI validation. The system currently has a kill switch for PR secret scanning, but does not have a graceful degradation mechanism for the AI validation step. We are working on a mechanism that can disable AI model calls specifically for PR scans, allowing scans to continue while avoiding the dependency on a degraded model-serving layer. The goal is to allow scans to continue with reduced functionality rather than stopping PR secret scanning entirely. • Improve alerting and incident communication. The alerting infrastructure detected PR scan lag, but the status page was not updated promptly. We are improving the process for escalating service degradation and communicating customer-facing impact through the status page. • Improve model-serving infrastructure resilience. The underlying infrastructure issue occurred because nodes were scheduled on a subnet with no available IPs. The immediate issue was resolved by moving nodes to subnets with free IPs. We are continuing to improve the resilience of the model-serving infrastructure to reduce the likelihood that infrastructure capacity issues can affect PR scan processing. Conclusion The incident was caused by degradation in the AI model validation service due to an infrastructure issue where nodes were scheduled on a subnet with exhausted IP capacity. The resulting model validation timeouts slowed PR secret scan processing and caused consumer lag to build. We mitigated the incident by temporarily stopping new PR secret scans, allowing the backlog to drain, and restoring model-serving capacity by moving nodes to subnets with available free IPs. We are following up with improvements to model-serving resilience, timeout and retry behavior, runtime configuration, graceful degradation, alerting, and customer communication.

6. resolved Aug 14, 2026, 04:14 PM UTC

   (1/2) Summary PR secret scans experienced processing lag, causing delays in processing pull requests. The issue was caused by timeouts when calling the model validation service. The model validation service was experiencing elevated latency and errors because compute nodes were scheduled on a subnet that had exhausted its available IPs. This slowed PR secret scan processing and caused consumer lag to build up. To resolve the issue, we temporarily disabled the feature that produces new PR secret scans so the queue could drain. We also resolved the underlying infrastructure issue by moving nodes to subnets with available free IPs. Customer Impact Customers experienced delayed PR secret scans. Other scan types remained operational. Root Cause The secret scanner calls the AI model validation service to classify detections during PR scans. The model validation call had a 30-second timeout configured. When the model validation service became slow or unresponsive, each failed request could block a scan consumer for the full 30 seconds before timing out. The underlying issue was in the model-serving infrastructure. One of the private subnets had run out of available IPs, and new nodes were created in that exhausted subnet, leaving model-serving pods in a pending state. This resulted in elevated latency and errors in the model validation service and reduced the processing capacity of PR secret scans. Contributing Factors • Model validation timeouts. The model validation service timeout was 30 seconds. When the service was degraded, scan consumers waited for the full timeout before the request failed, which created a processing bottleneck. • No fallback to bypass AI validation for PR scans. There was no mechanism to skip AI model validation specifically for PR secret scans while allowing the rest of the scan to continue. The available kill switch stopped producing PR secret scans entirely. This meant the mitigation required a tradeoff between allowing processing lag to continue or temporarily stopping PR secret scans. • Customer communication gap. Internal alerts for PR scan lag did fire, but the status page was not updated promptly to reflect the degradation. As a result, there was a period in which we had internal visibility into the issue while customers did not have the same visibility through the status page. Mitigation and Recovery A pre-existing feature flag was used to stop new PR secret scans from being produced. This stopped new messages from being added to the PR secret scan queue and allowed the existing backlog to drain. The backlog was cleared shortly after the mitigation was enabled. In parallel, the underlying model-serving infrastructure issue was resolved by moving nodes to subnets with available free IPs, restoring model-serving capacity. Once the infrastructure issue was resolved and the backlog had drained, PR secret scanning was restored. Learnings and Corrective Actions • Review the AI model validation timeout. The 30-second model validation timeout meant that when the model-serving layer degraded, each scan consumer could remain blocked for the full timeout before failing. We are reviewing the timeout and retry behavior. This work must also account for the risk that shorter timeouts and faster retries could increase request volume against an already degraded model-serving service.

Read the full incident report →

Started Aug 13, 2026, 02:53 PM UTC· resolved Aug 13, 2026, 03:39 PM UTC

1. investigating Aug 13, 2026, 02:53 PM UTC

   GitHub component: Git Operations, Webhooks, Pull Requests Original GitHub incident: https://stspg.io/f0cgtdc0kqxg

2. investigating Aug 13, 2026, 02:55 PM UTC

   GitHub component: Git Operations, Webhooks, Pull Requests Original GitHub incident: https://stspg.io/f0cgtdc0kqxg

3. investigating Aug 13, 2026, 02:57 PM UTC

   GitHub component: Git Operations, Webhooks, Pull Requests Original GitHub incident: https://stspg.io/f0cgtdc0kqxg

4. investigating Aug 13, 2026, 02:57 PM UTC

   GitHub component: Git Operations, Webhooks, Pull Requests Original GitHub incident: https://stspg.io/f0cgtdc0kqxg

5. investigating Aug 13, 2026, 03:30 PM UTC

   GitHub component: Git Operations, Webhooks, Pull Requests Original GitHub incident: https://stspg.io/f0cgtdc0kqxg

6. investigating Aug 13, 2026, 03:32 PM UTC

   GitHub component: Git Operations, Webhooks, Pull Requests Original GitHub incident: https://stspg.io/f0cgtdc0kqxg

7. resolved Aug 13, 2026, 03:39 PM UTC

   GitHub component: Git Operations, Webhooks, Pull Requests Original GitHub incident: https://stspg.io/f0cgtdc0kqxg

Read the full incident report →

Started Aug 12, 2026, 04:25 PM UTC· resolved Aug 12, 2026, 04:42 PM UTC

1. investigating Aug 12, 2026, 04:25 PM UTC

   GitHub component: Pull Requests Original GitHub incident: https://stspg.io/ssd9z8l2g46v

2. investigating Aug 12, 2026, 04:28 PM UTC

   GitHub component: Pull Requests Original GitHub incident: https://stspg.io/ssd9z8l2g46v

3. resolved Aug 12, 2026, 04:42 PM UTC

   GitHub component: Pull Requests Original GitHub incident: https://stspg.io/ssd9z8l2g46v

Read the full incident report →

Started Aug 11, 2026, 02:55 PM UTC· resolved Aug 11, 2026, 08:10 PM UTC

1. investigating Aug 11, 2026, 02:55 PM UTC

   GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

2. investigating Aug 11, 2026, 03:36 PM UTC

   GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

3. monitoring Aug 11, 2026, 05:09 PM UTC

   GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

4. monitoring Aug 11, 2026, 05:13 PM UTC

   GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

5. resolved Aug 11, 2026, 08:10 PM UTC

   GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

Read the full incident report →

Started Aug 10, 2026, 02:06 PM UTC· resolved Aug 11, 2026, 03:59 PM UTC

1. investigating Aug 10, 2026, 02:06 PM UTC

   Customers may experience degraded performance in scans. Pull request and CLI scans may be affected.

2. investigating Aug 10, 2026, 02:21 PM UTC

   The team has identified the root cause of the issue and is working on the solution.

3. investigating Aug 10, 2026, 02:47 PM UTC

   The root cause has been resolved. The system began to stabilize itself, and all the scans are starting to get processed with regular performance.

4. investigating Aug 10, 2026, 03:27 PM UTC

   All scan types except for SAST are fully operational. SAST continues to stabilize and will soon be fully stable.

5. resolved Aug 10, 2026, 03:45 PM UTC

   System should be back to being fully operational.

6. resolved Aug 11, 2026, 03:59 PM UTC

   Root Cause The incident was caused by a deployment of one service that ran a database index creation. The team has identified an issue with the way we perform index creations as database migrations. During a deployment the pods with newest image of the service attempted to create an index on a big table. The team has identified that the index creation took 7 minutes. However, during index creation pods were not responsive, and as a result, Kubernetes deemed them as unhealthy pods and attempted to retry those pods after 5 minutes. As a result, because the pod got killed before the index creation was fully completed, the database transaction was rolled back. Then, subsequent pods attempted to create the index again, dying after 5 minutes. This lead to the database being in unhealthy state, and the service was down. The team has rolled back the deployment, and killed all replicas that attempted to create the index. Thanks to that, the service and the database was in healthy state again. Why safety measures did not help Cycode provides a safety mechanism that unblocks all Pull Request scans after a specific period of time, giving each scan a maximum duration before the Pull Request is unblocked. However, because the service that is responsible for triggering and completing scans, as well as this safety net, was down, the process couldn't behave as expected. We acknowledge this gap and are working on strengthening this area of our system. Action items • The team is actively investigating enhancements and new safety protocols that can be put in place in order to have another safety net preventing Pull Request scans being stuck in case of any incident. • The team is investigating changes to the index creation process.

Read the full incident report →

Started Aug 02, 2026, 02:08 PM UTC· resolved Aug 04, 2026, 06:08 AM UTC

1. investigating Aug 02, 2026, 02:08 PM UTC

   We’re investigating an issue causing older events to be reprocessed. Impact: Some workflows may run again, which could result in duplicate alerts. PR scans may also be delayed.

2. identified Aug 02, 2026, 06:12 PM UTC

   We have resolved the processing backlog caused by an issue during a Kafka migration. The issue resulted in older events being processed alongside new events, which caused delays and may have updated some violations with an outdated status. Normal processing has been restored. However, customers may still see some violations with an outdated status while we identify and correct the affected records. PR scans were not delayed or reprocessed, and workflows were not affected. We are continuing remediation and closely monitoring the system.

3. monitoring Aug 02, 2026, 06:59 PM UTC

   Normal processing has been restored, and the incident is now in monitoring. A small number of customers may still see a limited number of violations with an outdated status as a result of the incident. We have identified the potentially affected environments and are working to correct the impacted records.

4. monitoring Aug 03, 2026, 03:25 PM UTC

   The system is fully operational. A small number of customers may still see a limited number of violations with an outdated status as a result of the incident. We have identified the potentially affected environments and are working to correct the impacted records.

5. resolved Aug 04, 2026, 06:08 AM UTC

   The platform is now fully operational and processing normally

Read the full incident report →

Started Jul 27, 2026, 10:31 AM UTC· resolved Jul 31, 2026, 07:43 AM UTC

1. investigating Jul 27, 2026, 10:31 AM UTC

   The team identified an issue with degraded performance with scanning. There can be delays in starting, running, and completing scans. All scan types may be affected (Pull request and CLI scans as well). The team has identified an issue with a malfunction in deployment, and the issue should be resolved shortly.

2. monitoring Jul 27, 2026, 11:27 AM UTC

   The team has identified the root cause and resolved it. We are seeing the system getting back to stability.

3. resolved Jul 27, 2026, 11:57 AM UTC

   The system is now back to being fully operational.

4. resolved Jul 31, 2026, 07:43 AM UTC

   Root Cause The incident was caused by a deployment of one service that ran a database migration. The team has identified that this migration contained faulty code and as a result lead to database overload when attempting to deploy the service. As a result, the service was partially down until the deployment was reverted. During this time, all scans were processed with lower than expected performance.

Read the full incident report →

Started Jul 27, 2026, 03:56 AM UTC· resolved Jul 27, 2026, 04:12 AM UTC

1. investigating Jul 27, 2026, 03:56 AM UTC

   GitHub component: API Requests Original GitHub incident: https://stspg.io/vr201n49yl53

2. resolved Jul 27, 2026, 04:12 AM UTC

   GitHub component: API Requests Original GitHub incident: https://stspg.io/vr201n49yl53

Read the full incident report →