Speaker & Attendee Profile Visibility Issue

Update timeline

1. resolved Mar 11, 2026, 01:30 PM UTC

   Status: Resolved This incident has been resolved.

2. resolved Mar 13, 2026, 02:02 AM UTC

   Status: Resolved Summary On March 11, 2026, we experienced a service disruption affecting the visibility of speaker and attendee profiles across multiple events. The incident was triggered by a timing issue during a scheduled infrastructure update — a data refresh was initiated before new field definitions had fully propagated to our Search Engine, causing the indexation system to be built from an outdated configuration. Our Search Engine relies on an indexation system to serve profile data quickly across events. When the refresh completed against the stale configuration, new fields were rejected due to strict schema enforcement, leaving profiles absent from search results and listings. The goal of this post-mortem is to share our assessment and the steps taken to resolve the issue, while providing full transparency to our customers. Timeline Time \(UTC\) Event Mar 11 – 11:04 Infrastructure update deployed to production. Changes confirmed as applied. Mar 11 – 13:21 Data refresh initiated as part of the planned release process. Previous indexation remained fully functional during this period. Mar 11 – 13:42 Release step complete. Data refresh running, expected duration ~3–4 hours. Mar 11 – 14:49 Configuration mismatch errors first logged internally. Mar 11 – 15:33 Data refresh reported as complete. Spot checks passed. Final release step initiated. Mar 11 – 16:52 Reports of missing profiles across multiple events surfaced. Incident response team immediately activated. Mar 11 – 17:29 Root cause identified in under few minutes: indexation system had been built before the new fields were fully available. Internal alerts corroborated the finding. Mar 11 – 18:08 Analysis completed confirming the refresh did not produce an indexation with the expected new fields. Mar 11 – 18:39 Corrective data refresh triggered with the correct configuration fully in place, within 30 minutes of initial reports. Mar 11 – 18:43 New indexation confirmed to contain the correct fields. Full refresh in progress. Resolution Once the root cause was identified, the team immediately initiated a new data refresh against the correct, fully propagated configuration. Profile visibility and search behavior were monitored throughout the process. Once the refresh completed, all profiles returned to normal search and listing behavior. Root Cause Analysis The root cause was a race condition between the deployment of new Search Engine field definitions and the triggering of a data refresh. The refresh started before the new configuration had fully propagated, causing the indexation system to be built from a stale schema. Due to strict schema enforcement, subsequent write operations referencing the new fields were rejected, leaving profiles absent from search results and listings. Several factors contributed to this incident: • Ambiguous deployment signal: A pipeline warning created uncertainty around whether the deployment had completed cleanly. • Insufficient spot checks: Post-refresh verification passed at a surface level, masking the underlying schema mismatch until live traffic exposed it. Short-term improvements: • Add a mandatory pre-refresh validation step to confirm the Search Engine schema matches the expected configuration before any refresh is initiated. • Update the release process to include explicit schema validation checkpoints for sub field, as a required step. Conclusion This incident stemmed from a timing edge case in our release process. The improvements listed above reflect our commitment to preventing similar incidents and maintaining a reliable platform for all our customers. For any questions or concerns regarding this incident, please reach out to our support team.