CCLI incident

Generating PDF sheet music is failing intermittently in SongSelect and Partner API

Minor Resolved View vendor source →

CCLI experienced a minor incident on February 10, 2025, lasting 1d 10h. The incident has been resolved; the full update timeline is below.

Started
Feb 10, 2025, 04:30 AM UTC
Resolved
Feb 11, 2025, 03:16 PM UTC
Duration
1d 10h
Detected by Pingoru
Feb 10, 2025, 04:30 AM UTC

Update timeline

  1. investigating Feb 10, 2025, 04:30 AM UTC

    We are currently investigating this incident..

  2. identified Feb 10, 2025, 07:11 AM UTC

    We have identified the problem area and are working on a fix for this incident..

  3. monitoring Feb 10, 2025, 02:01 PM UTC

    We implemented a fix and are currently monitoring the result..

  4. monitoring Feb 10, 2025, 02:25 PM UTC

    After significant investigation, it appears the partner API was NOT negatively affected by this. It only affected our SongSelect application..

  5. identified Feb 10, 2025, 03:24 PM UTC

    Althought the partner API was not affected from the prior issue, we are now seeing some new issues with the partner API. We are working on it..

  6. monitoring Feb 10, 2025, 04:29 PM UTC

    At the moment, the system appears to be operating normal again. We will continue to monitor..

  7. resolved Feb 10, 2025, 04:50 PM UTC

    We believe this incident is resolved, but we will continue to monitor over the coming days..

  8. identified Feb 10, 2025, 07:06 PM UTC

    As we are monitoring, the problem has returned so we are reopening this incident. We are working on a solution. This is now likely also affecting the ability for Partners to get chord sheets, and it is impacting SongSelect ability to view Lead and Vocal sheets. .

  9. monitoring Feb 10, 2025, 09:56 PM UTC

    Things seem to have stabilized since 11:20am Pacific Time and things have been operational since then. We are continuing to monitor closely..

  10. resolved Feb 11, 2025, 03:16 PM UTC

    The system is stable. This incident has been marked resolved, but we will continue to monitor closely over the coming weeks..