Fluid Attacks experienced a notice incident on July 21, 2026 affecting Platform, lasting 1d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 21, 2026, 05:13 PM UTC
The analytics charts (by group, by organization, and portfolio views) are currently displaying outdated information. The data behind these charts last refreshed on 2026-06-24 and has not updated since, so recent activity, including groups and repositories created after that date, is not yet reflected. This issue is limited to the charts only: the underlying data remains complete, intact, and unaffected, and there is no impact to security or to any other part of the platform. Our team is actively investigating the cause and working to restore regular chart updates. Further updates will be posted as we make progress.
- resolved Jul 22, 2026, 07:30 PM UTC
This incident has been resolved.
- postmortem Jul 24, 2026, 01:51 AM UTC
### Postmortem ### Impact Users experienced stale analytics charts across the platform: the documents powering the "by group", "by organization", and "portfolio" views stopped being refreshed, leaving the vast majority of organizations displaying outdated data as of June 24. Groups and repositories created after that date did not appear in the charts even when data was available. The issue started on June 23, 2026 and was discovered approximately 20 days later by a staff member who noticed the charts were not up to date. Once identified, the problem was fully resolved within 1 day. ### Cause A scheduled background job responsible for regenerating analytics charts was failing silently. When processing a large number of organizations, a single error during the generation of one organization's charts caused the entire job to stop, preventing all remaining organizations from being updated. Over time, increased load on the platform's data layer began triggering these errors more frequently and across more organizations, amplifying the impact. Because no alerts were configured to notify the team when this job failed, the issue went undetected for several weeks. ### Solution The chart generation process was updated to handle errors on a per-organization basis: if one organization's charts fail to generate, it is logged and skipped, and processing continues for all others. Additional safeguards were introduced to reduce pressure on the underlying data layer during high-load periods. Alerting was also configured so that failures in this job are now immediately surfaced to the engineering team, preventing similar silent failures in the future. ### Conclusion A background job with no error isolation and no failure alerting can silently affect customer-facing data for an extended period before anyone notices. The key improvements, isolating errors per subject and adding alerts on job failures, ensure this class of issue will be detected and addressed much faster going forward. We apologize for the disruption and are committed to maintaining the accuracy and freshness of your analytics data.