Obsidian Security incident
[Activity] Delay in processing of Activity Data
Obsidian Security experienced a notice incident on February 28, 2024 affecting Threat Management and Connections (SaaS Applications), lasting 3d 21h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Feb 28, 2024, 07:44 PM UTC
Our systems detected a delay in fetching activity information for connections to Salesforce. Our engineering teams have since identified and resolved the underlying issue and are currently monitoring. Due to the nature of the fix deployed, it might take a few hours for the data under the 'Activity' section to become latest on our systems for a few tenants. This might also result in some delay in regards to processing of a limited set of Alerts. Please open a support ticket via the normal channels in case more information is required and we would be happy to assist you.
- monitoring Feb 29, 2024, 09:17 PM UTC
We are continuing to monitor this issue on our side. There are still a few tenants where the activity data is not latest and is recovering swiftly. We expect the data to be fully latest by end of today PST. Please contact our support team for any further information that you may require.
- monitoring Mar 01, 2024, 10:51 PM UTC
We are aware of a few instances in which some tenants are still seeing delays with Activity data not being the latest. During our efforts to clear the delay in processing the events, we encountered an infrastructure issue which has caused the recovery process to not scale to our expectations. Our teams are continuing to work on optimizing the recovery process to clear the backlog. Next update with more concrete timelines on full recovery will be shared in next 3 hours. Please contact our support team in case you have any questions.
- monitoring Mar 02, 2024, 02:09 AM UTC
Our teams have put measures in place to normalize the infrastructure responsible for processing activity data. We will be monitoring overnight and expect to make significant progress to reduce the overall delay and backlog. We will share next updates on the overall progress of recovery in the PST morning hours. Please contact our support team in case you have any questions.
- monitoring Mar 02, 2024, 04:26 PM UTC
We have made significant progress in reducing the backlog overnight and the delay remains only in a small number of tenants which are recovering at a good pace. We expect the delay to be fully recovered in the next 6-8 hours. We will send an update once the activity data is latest across all tenants. Please contact our support team in case of any questions.
- resolved Mar 03, 2024, 04:46 PM UTC
All tenants are now fully up to date with the latest Activity Data as of 6 hours ago. Our teams will continue to monitor this from our side, however, this incident is now being marked as resolved. Please reach out to our support teams in case you have any questions.