Highlight.io Outage History

Highlight.io is up right now

Pingoru has recorded 33 incidents for Highlight.io: 0 in the last 12 months and 0 in the last 90 days.

Source: status.highlight.io

33incidents recorded
0last 12 months
0last 90 days
–typical duration

Incidents, newest first

Minor March 4, 2025 Delayed session and error ingest. 7d 16h

Started Mar 04, 2025, 10:39 PM UTC· resolved Mar 12, 2025, 03:37 PM UTC

  1. investigating Mar 04, 2025, 10:39 PM UTC

    We are still investigating an ingest delay related to our kafka cluster that affected data ingest since yesterday. Sessions and errors are affected.

  2. investigating Mar 04, 2025, 10:39 PM UTC

    We are a couple of hours behind on processing right now - latest processed sessions are from around 10am PST; we are catching up at the moment and will probably be up to date in around 4 hours.

  3. resolved Mar 12, 2025, 03:37 PM UTC

    This was resolved as of March 5th.

Read the full incident report →

Minor February 28, 2025 Delayed traces ingest 12h 17m

Started Feb 28, 2025, 03:25 PM UTC· resolved Mar 01, 2025, 03:42 AM UTC

  1. investigating Feb 28, 2025, 03:25 PM UTC

    Traces are currently delayed 20 minutes. We're investigating.

  2. identified Feb 28, 2025, 03:51 PM UTC

    We've identified the root cause in our ClickHouse database cluster that we're working on addressing.

  3. identified Feb 28, 2025, 04:07 PM UTC

    We're catching up on the trace backlog. ETA 30 minutes.

  4. monitoring Feb 28, 2025, 08:59 PM UTC

    We had another related incident in the trace ingest which created an additional ingest blip. We've tuned the insert concurrency and the backlog is decreasing (ETA 30 minutes).

  5. resolved Mar 01, 2025, 03:42 AM UTC

    All data caught up.

Read the full incident report →

Minor February 24, 2025 Sessions stuck in live mode. 4h 11m

Started Feb 24, 2025, 09:18 PM UTC· resolved Feb 25, 2025, 01:29 AM UTC

  1. investigating Feb 24, 2025, 09:18 PM UTC

    We're investigating a report of sessions stuck in live mode as of 5:00 AM PT today.

  2. investigating Feb 24, 2025, 10:24 PM UTC

    New sessions are processing correctly. We're working on restoring sessions that are missing as a result of the incident.

  3. monitoring Feb 24, 2025, 11:30 PM UTC

    We have caught up on processing 80% of sessions with an ETA of 2 hours to completion.

  4. resolved Feb 25, 2025, 01:29 AM UTC

    All sessions have now been processed.

Read the full incident report →

Minor February 13, 2025 Delayed traces ingest 21h 58m

Started Feb 13, 2025, 06:31 PM UTC· resolved Feb 14, 2025, 04:30 PM UTC

  1. investigating Feb 13, 2025, 06:31 PM UTC

    We're investigating a delay in ingesting traces.

  2. investigating Feb 13, 2025, 07:04 PM UTC

    We've resolved the issue and are catching up on the delay.

  3. resolved Feb 14, 2025, 04:30 PM UTC

    We're all caught up on traces.

Read the full incident report →

Minor January 29, 2025 Frontend page hanging 10h 17m Frontend

Started Jan 29, 2025, 06:02 PM UTC· resolved Jan 30, 2025, 04:19 AM UTC

  1. investigating Jan 29, 2025, 06:02 PM UTC

    We're investigating situations causing app.highlight.io to crash in the browser (when multiple table are open).

  2. resolved Jan 30, 2025, 04:19 AM UTC

    The issue has been resolved.

Read the full incident report →

Minor January 28, 2025 Delayed session and error ingest 4h 46m

Started Jan 28, 2025, 05:41 PM UTC· resolved Jan 28, 2025, 10:27 PM UTC

  1. investigating Jan 28, 2025, 05:41 PM UTC

    We are investigating a delay in session and error processing due to blocking in our error ingest path.

  2. identified Jan 28, 2025, 06:19 PM UTC

    We're experiencing an ingest delay of roughly 6 hours of sessions / errors. A fix is in progress.

  3. identified Jan 28, 2025, 06:35 PM UTC

    We are deploying a fix for the ingest bottleneck.

  4. monitoring Jan 28, 2025, 07:55 PM UTC

    We are catching up on the data backlog.

  5. resolved Jan 28, 2025, 10:27 PM UTC

    The incident is resolved. See #incidents in Discord for an analysis.

Read the full incident report →

Minor January 21, 2025 Delayed session and error ingest 6h 34m

Started Jan 21, 2025, 06:53 PM UTC· resolved Jan 22, 2025, 01:28 AM UTC

  1. investigating Jan 21, 2025, 06:53 PM UTC

    We're investigating a ~2 hour delay in the sessions / errors appearing in the highlight UI.

  2. identified Jan 21, 2025, 11:09 PM UTC

    We have identified ClickHouse async inserts to be the bottleneck in our data sync queue. We are working on a fix.

  3. identified Jan 21, 2025, 11:57 PM UTC

    We've shipped a fix to the underlying issue and are starting to catch up on the data backlog.

  4. monitoring Jan 22, 2025, 12:17 AM UTC

    We're vaught up on all delayed data and are monitoring the current ingest rate.

  5. resolved Jan 22, 2025, 01:28 AM UTC

    The issue is resolved.

Read the full incident report →

Minor December 23, 2024 Delayed session and error ingest 16h 20m

Started Dec 23, 2024, 11:18 PM UTC· resolved Dec 24, 2024, 03:38 PM UTC

  1. investigating Dec 23, 2024, 11:18 PM UTC

    We're experiencing a data ingest delay for sessions and errors due to Kafka maintenance. We're working on a fix. Traces and logs are currently ingested with no delay.

  2. investigating Dec 24, 2024, 03:06 PM UTC

    We are investigating a disk space issue in the cluster.

  3. resolved Dec 24, 2024, 03:38 PM UTC

    We have resolved a disk space issue that caused our kafka cluster to become unhealthy.

Read the full incident report →

Minor November 19, 2024 Delayed traces ingest 1d 2h

Started Nov 19, 2024, 05:15 PM UTC· resolved Nov 20, 2024, 07:18 PM UTC

  1. investigating Nov 19, 2024, 05:15 PM UTC

    We're investigating a delay in ingesting traces.

  2. investigating Nov 19, 2024, 08:55 PM UTC

    We've addressed the issue and are catching up on the ingest delay.

  3. monitoring Nov 19, 2024, 11:28 PM UTC

    ETA to be caught up is about 3 hours from now, 6:30pm PT.

  4. resolved Nov 20, 2024, 07:18 PM UTC

    All delayed data has been ingested.

Read the full incident report →