Splunk Observability Cloud JP0 incident

Some sf.org metrics are incorrectly reporting drops - no data is being dropped

Notice Resolved View vendor source →

Splunk Observability Cloud JP0 experienced a notice incident on May 10, 2023, lasting 2h 15m. The incident has been resolved; the full update timeline is below.

Started
May 10, 2023, 03:00 PM UTC
Resolved
May 10, 2023, 05:15 PM UTC
Duration
2h 15m
Detected by Pingoru
May 10, 2023, 03:00 PM UTC

Update timeline

  1. resolved May 10, 2023, 04:59 PM UTC

    A recent code push resulted in some sf.org metrics reporting incorrectly that data was being dropped. No data drops actually occurred. The code change has been rolled back and the metrics are now being correctly reported. This issue started around 8:10 AM PT and was resolved by 10:15 AM PT. Again, no production outage actually happened only sf.org metrics are reporting incorrect information. We apologize for this inconvenience. Impacted metrics: - sf.org.numDatapointsDroppedBatchSizeByToken - sf.org.numDatapointsDroppedBatchSize