Cribl incident

Splunk Single Instance and LB Unexpectedly Closing Connections

Critical Resolved View vendor source →

Cribl experienced a critical incident on August 15, 2024 affecting Data Plane (Workers), lasting 14h 25m. The incident has been resolved; the full update timeline is below.

Started
Aug 15, 2024, 01:31 AM UTC
Resolved
Aug 15, 2024, 03:56 PM UTC
Duration
14h 25m
Detected by Pingoru
Aug 15, 2024, 01:31 AM UTC

Affected components

Data Plane (Workers)

Update timeline

  1. identified Aug 15, 2024, 01:31 AM UTC

    The issue has been identified and a fix is being implemented.

  2. identified Aug 15, 2024, 01:33 AM UTC

    Status: We have identified an issue in Stream version 4.8.0, where Splunk Single Instance and Splunk Load Balanced Destinations mistakenly detected that the downstream Splunk instance is frequently shutting down. Customer Impact: Stream and Edge Nodes are not able to connect to the Splunk destination. This incident affects: Cribl.Cloud Stream and Edge. Customer Splunk destination data streams are affected by this incident. Next Update: We will provide updates every 60 minutes. Cribl Support is available to answer questions via email at [email protected]. Thank You for being a Cribl.Cloud customer.

  3. identified Aug 15, 2024, 03:25 AM UTC

    Status: We're continuing to review the issue and are working to address it as soon as possible. To mitigate the issue in the meantime, set S2S version to v4. Customer Impact: Stream and Edge Nodes are not able to connect to the Splunk destination. This incident affects: Cribl.Cloud Stream and Edge. Customer Splunk destination data streams are affected by this incident. Next Update: We will provide an update tomorrow morning at 8 AM EDT (Aug 15). Cribl Support is available to answer questions via email at [email protected].

  4. identified Aug 15, 2024, 12:16 PM UTC

    Status: We have identified a fix and plan on making it available today, within the next few hours. To mitigate the issue in the meantime, set S2S version to v4. Customer Impact: Stream and Edge Nodes are not able to connect to the Splunk destination. This incident affects: Cribl.Cloud Stream and Edge. Customer Splunk destination data streams are affected by this incident. Next Update: We will provide updates every 60 mins. Cribl Support is available to answer questions via email at [email protected].

  5. identified Aug 15, 2024, 01:20 PM UTC

    Status: We have started to roll out the fix to addresss the issue. Customer Impact: Stream and Edge Nodes are not able to connect to the Splunk destination. This incident affects: Cribl.Cloud Stream and Edge. Customer Splunk destination data streams are affected by this incident. Next Update: We will provide updates every 60 mins. Cribl Support is available to answer questions via email at [email protected].

  6. identified Aug 15, 2024, 03:12 PM UTC

    Status: We're continuing to roll out the fix to addresss the issue. Customer Impact: Stream and Edge Nodes are not able to connect to the Splunk destination. This incident affects: Cribl.Cloud Stream and Edge. Customer Splunk destination data streams are affected by this incident. Next Update: We will provide updates every 60 mins. Cribl Support is available to answer questions via email at [email protected].

  7. resolved Aug 15, 2024, 03:56 PM UTC

    This incident has been resolved.