Scheduler Incorrectly Opening Events Across Multiple Groups

Update timeline

1. investigating Jul 09, 2026, 03:55 PM UTC

   Some groups are receiving unexpected open events related to environment URL availability. These events were generated incorrectly and do not reflect actual issues in the affected groups. Our team is investigating and working on a fix.

2. resolved Jul 09, 2026, 05:50 PM UTC

   This incident has been resolved.

3. postmortem Jul 10, 2026, 04:51 AM UTC

   ### Postmortem #### Impact At least one user experienced a flood of false-positive `ENVIRONMENT_ISSUES` events on their production groups, each dispatching an automatic email notification, for URL environments that were in fact reachable. The issue started on 2026-07-08 at 20:00 \(UTC-5\) and was reactively discovered 14.9 hours later by a staff member who noticed an unusually high volume of events of this type opened simultaneously across multiple production groups, all sharing an identical "URL is not reachable" description. The problem was resolved within an additional 2.6 hours, resulting in a total window of exposure of 17.5 hours. #### Cause A new URL environment connectivity check ran for the first time in production on its automated schedule. The logic that determines which URLs are eligible for the check contained a flaw: instead of correctly excluding URLs that require authentication or that are only reachable through special network channels, it included them in an unauthenticated probe. Upon receiving unauthorized-access responses, such as redirects to login portals, the system incorrectly classified those URLs as unreachable and opened an incident event for each one, also dispatching a notification email to the group's stakeholders. The affected URLs were fully functional; they simply required credentials that the automated probe never supplied. Approximately 850 spurious events were created across production groups. #### Solution The approximately 850 incorrectly generated events were closed through an automated process that identified them by type, description, and creation time, and closed them directly without emitting further notification emails to users. The automated scheduler was disabled to prevent the issue from recurring. As a permanent fix, the connectivity check will no longer run automatically and will instead become an on-demand action that internal users can trigger manually from the interface. Additionally, the eligibility logic will be corrected to properly exclude authenticated environments and those operating through special network channels, and an exhaustive dry-run against real production data will be required before any re-enablement. #### Conclusion This incident revealed that the URL eligibility logic and the handling of authenticated environments were never validated end-to-end before the first production run, allowing an unauthenticated probe to be misread as a connectivity failure at scale. Tests covering the eligibility logic and the authenticated-URL flow, along with a dry-run against real production data before enabling any automated task, would have prevented these events from being created. **MISSING\_TEST < INCOMPLETE\_PERSPECTIVE**