Visitor - S3 - Multiple Daily Summary Emails Being Sent

Update timeline

1. identified Feb 10, 2026, 03:29 PM UTC

   Some customers may receive multiple daily summary emails for the same location. The issue does not impact application functionality or data. Our engineering team has identified the cause, and a fix is in progress. As a temporary workaround, daily summary emails can be disabled from profile notification settings. Next Update: 9:00 AM UTC

2. identified Feb 11, 2026, 08:59 AM UTC

   Our engineering team has deployed a fix and is in the process of validating it. We are closely monitoring the system to confirm that the issue is fully addressed. We will continue to observe performance and share further updates as needed. Next scheduled update: 12 February 2025

3. identified Feb 11, 2026, 04:32 PM UTC

   We have been informed that some customers are not receiving daily summary emails. Our Engineering team has deployed another fix and is actively monitoring the current state. We will continue to observe performance and provide further updates as needed. Next scheduled update: 12 February 2025

4. monitoring Feb 12, 2026, 04:38 PM UTC

   The issue causing multiple daily summary emails has been identified, and a resolution has been applied. Duplicate scheduler entries created during a previous clustering change have been cleaned up, and new duplicates are no longer being generated. Email delivery is now functioning as expected. We will continue to monitor to ensure ongoing stability. Thank you for your patience.

5. monitoring Feb 16, 2026, 02:21 PM UTC

   Following the recent fix for duplicate emails, we are currently observing delays in email delivery for some customers. Emails remain accurate, but delivery may occur with a delay after the scheduled time. Our Engineering team is actively monitoring the system. We will continue to share updates as needed.

6. resolved Feb 18, 2026, 09:36 AM UTC

   The issue causing delays in email delivery has been resolved. Daily summary emails have been sent at their scheduled execution times. The earlier delay was caused by a temporary backlog of scheduled events, which has now been fully processed. No configuration changes were required, and no further customer impact is expected. We will continue to monitor internally for the next 24 hours. A Root Cause Analysis (RCA) will be published within 10 days. Your patience is appreciated.

7. postmortem Mar 13, 2026, 03:50 PM UTC

   We appreciate your continued support and patience while we investigated and resolved this incident. ### Type of Event S2 – Visitor email and notification delivery defect ### Services / Modules Impacted * Daily Summary Email Notifications \(Admin Digest\) * Background Task Scheduler / Notification Engine ### Issue Summary / Background Beginning in early February 2026, some Visitor administrators received multiple copies of the Daily Summary \(Admin Digest\) email within the same day, instead of the expected single daily notification. While the core Visitor application remained fully operational and accessible throughout the incident, the unexpected duplicate messages caused unnecessary inbox noise and created confusion for affected administrators. Additionally, some Daily Summary emails were delivered several hours later than their normal delivery time. Our investigation found that the system responsible for generating these emails had inadvertently created duplicate scheduled tasks for a subset of organizations. When these tasks ran, each one triggered its own email, resulting in repeated Daily Summary messages for the same day. The additional workload created by these duplicate tasks also slowed down the task processing queue, which led to delays in sending some emails. Our Engineering and Cloud Operations teams worked together to identify and remove the duplicate scheduler entries, adjust task server configurations, and closely monitor the system. Normal email delivery patterns were fully restored once these corrections were applied. ### Root Cause The incident was caused by duplicate background scheduler entries responsible for generating the Daily Summary \(Admin Digest\) emails. These duplicate entries resulted in both repeated email notifications and delays in their delivery. #### **Duplicate scheduler entries triggering multiple emails** For a subset of organizations, the task scheduler contained more than one entry for the same Daily Summary event. When the scheduler ran, each duplicate entry was processed independently, causing multiple Daily Summary emails to be generated and delivered to the same administrators on the same day. #### **Task processing backlog causing delivery delays** The extra workload created by these duplicate tasks significantly increased the number of background jobs waiting to be processed. Because the system processes scheduled events sequentially, this backlog slowed down the queue. As a result, some Daily Summary emails were delivered **7–9 hours later** than their usual time. #### **No impact to core application data or check‑in workflows** This issue was limited to email notifications only. No customer data was lost, and all core Visitor check-in and operational workflows continued to function normally throughout the incident. ### Remediation The following actions were taken to resolve the incident and restore normal system behavior. 1. **Incident validation and scope review** Support and engineering teams reviewed reports to confirm the issue was limited to Daily Summary emails and did not affect core Visitor functionality, such as check-in workflows. 2. **Scheduler data analysis** Engineering analyzed scheduler records to identify organizations with duplicate Daily Summary events and reviewed other scheduled event types to detect additional anomalies. 3. **Removal of duplicate scheduled tasks** Duplicate scheduler entries were removed for Daily Summary and other related scheduled events. This reduced the overall task queue and prevented further duplicate email generation. 4. **Task server configuration adjustments** The Visitor task server configuration was adjusted to stabilize task processing and prevent uncontrolled parallel execution. After the changes were validated and behavior was confirmed, the server was returned to a stable configuration. 5. **Monitoring and validation** Email delivery patterns and scheduler activity were monitored over multiple days to confirm that: * Only one Daily Summary email per organization per day was being generated * The scheduler backlog was cleared * Email delivery times returned to their normal schedule 6. **Incident closure** After confirming stable system behavior and the absence of new duplicate emails, the incident was formally resolved. ### **Timeline \(UTC\)** ‌ **February 6, 2026 – 03:50 PM** Initial customer reports received that some Visitor administrators were getting multiple Daily Summary \(Admin Digest\) emails on the same day. Core Visitor application functionality and data were not impacted. **February 10, 2026 – 03:29 PM** Public status page incident opened at S3 to communicate that some customers were receiving multiple Daily Summary \(Admin Digest\) emails. Engineering identified the cause and began work on a fix. Customers were advised they could temporarily disable Daily Summary emails in their profile notification settings if needed. **February 11, 2026 – 08:59 AM** A first fix was deployed to address the duplicate Daily Summary emails. Engineering monitored the system to validate that the change was effective. **February 11, 2026 – 04:32 PM** Status page update posted noting that, while duplicate emails were being addressed, some customers were not receiving Daily Summary emails as expected. An additional fix was deployed, and delivery behavior continued to be monitored. **February 12, 2026 – 04:38 PM** Status page update posted confirming that the issue causing multiple Daily Summary emails had been identified and resolved. Duplicate scheduler entries were cleaned up so new duplicates would no longer be generated. Daily Summary \(Admin Digest\) emails were delivered successfully, with some delays while the remaining backlog of previously scheduled tasks was processed. **February 13–15, 2026** Engineering and Operations teams closely monitored the delivery of the Daily Summary \(Admin Digest\) email. No new duplicate emails were observed. Some customers continued to see delays in email delivery as the system worked through the backlog of earlier scheduled events. **February 16, 2026 – 02:21 PM** Status page update posted noting that, following the fix for duplicate emails, some customers were seeing Daily Summary \(Admin Digest\) emails delivered later than scheduled. Emails themselves remained accurate, and teams continued actively monitoring and tuning processing to reduce delays. **February 17, 2026** Monitoring confirmed that the backlog was clearing, and Daily Summary \(Admin Digest\) emails were executing closer to their scheduled times. No new duplicate emails or new customer cases were observed. **February 18, 2026 – 09:36 AM** Status page update posted, marking the incident as resolved. The backlog of scheduled events had been fully processed, and Daily Summary \(Admin Digest\) emails were once again being sent at their scheduled times. No further configuration changes were required, and internal monitoring continued for an additional 24 hours to ensure stability. **February 19–27, 2026** Ongoing monitoring confirmed stable Daily Summary \(Admin Digest\) email delivery with no recurrence of duplicate emails or unusual delays. The incident was considered fully resolved. ### Total Duration of Event 504 hours \(22 days\) ### Preventive Actions To reduce the likelihood of similar issues in the future, the following improvements are being implemented. 1. **Duplicate scheduler safeguards** Additional safeguards will ensure that only one Daily Summary task can exist per organization for a given day. 2. **Improved monitoring and alerting** Monitoring will be enhanced to detect unusual increases in email volume, scheduler backlog size, or deviations between scheduled and actual execution times. 3. **Task server capacity and configuration review** Capacity planning and configuration management procedures for the Visitor task server will be refined to ensure stable processing during operational changes. 4. **Expanded testing for scheduled notifications** Testing coverage will be extended to simulate higher scheduling loads and verify that only a single Daily Summary email is generated per day. 5. **Updated operational runbooks** Internal documentation will be updated with clear procedures for identifying duplicate scheduler entries and safely performing cleanup operations. 6. **Ongoing monitoring** Engineering and operations teams will continue monitoring scheduler health and notification delivery patterns to detect anomalies early.