Expel experienced a minor incident on June 23, 2025 affecting Assembler connectivity and Alert ingestion, lasting 10h 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 23, 2025, 09:51 AM UTC
At approximately 4am EDT, we began experiencing a problem with our assembler VPN system that is preventing alerts from being ingested through assemblers. Alert ingestion is delayed for some devices behind an assembler. We will provide an update by 6:30am EDT.
- identified Jun 23, 2025, 10:30 AM UTC
We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 7:00 EDT.
- identified Jun 23, 2025, 11:00 AM UTC
We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 7:30 EDT.
- identified Jun 23, 2025, 11:47 AM UTC
We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 8:00 EDT.
- identified Jun 23, 2025, 12:07 PM UTC
We are continuing to work on a fix for this issue.
- identified Jun 23, 2025, 12:08 PM UTC
We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 8:30 EDT.
- identified Jun 23, 2025, 12:42 PM UTC
We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 9:30 EDT.
- identified Jun 23, 2025, 01:30 PM UTC
We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 10:30 EDT.
- identified Jun 23, 2025, 02:31 PM UTC
We have identified a further cause and are working on a remediation. Individual assemblers are coming back online. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 11:30 EDT.
- identified Jun 23, 2025, 03:32 PM UTC
We have identified a further cause and are working on a remediation. Individual assemblers are coming back online. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 12:30 EDT.
- monitoring Jun 23, 2025, 07:50 PM UTC
We have implemented a fix and believe the incident to be resolved. We are monitoring the health of the infrastructure as individual assemblers come back online, and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems. We will provide another update by 4:30pm EDT.
- resolved Jun 23, 2025, 08:04 PM UTC
The incident has been resolved, and all previously-available assemblers are once again fully available.