Placements incident

Placements platform experiencing degraded service due to third party infrastructure provider issue

Notice Resolved View vendor source →

Placements experienced a notice incident on June 10, 2025 affecting app.placements.io, lasting 7h 30m. The incident has been resolved; the full update timeline is below.

Started
Jun 10, 2025, 02:29 PM UTC
Resolved
Jun 10, 2025, 10:00 PM UTC
Duration
7h 30m
Detected by Pingoru
Jun 10, 2025, 02:29 PM UTC

Affected components

app.placements.io

Update timeline

  1. investigating Jun 10, 2025, 02:29 PM UTC

    An ongoing incident with an infrastructure provider is causing degraded services on the platform. Specifically, we are unable to initiate scheduled jobs or spin up new machines to process queues. The app generally functions fine but scheduled ad server and CRM imports are paused and scheduled billing jobs can't be run. We are monitoring the incident with our infrastructure provider.

  2. investigating Jun 10, 2025, 02:31 PM UTC

    We are continuing to investigate this issue.

  3. investigating Jun 10, 2025, 04:21 PM UTC

    Our third party service provider continues to investigate and remediate the issue. Billing reports and hour imports, along with other backend jobs (like email notifications) continue to be backed up.

  4. investigating Jun 10, 2025, 05:48 PM UTC

    We received the following update from our infrastructure provider: "We continue to work on the permanent fix and explore alternate paths to resolution. Once the fix is developed, an estimated time to resolution will be shared."

  5. investigating Jun 10, 2025, 06:55 PM UTC

    We have received the following update from our infrastructure provider: "We’ve identified the likely trigger of the issue and are making strong progress toward a resolution. A recent automated update disrupted network connectivity on affected instances. We're working on a fix to prevent this from recurring and will provide an ETA for resolution when it's available."

  6. investigating Jun 10, 2025, 07:21 PM UTC

    We have recovered some ability to scale up workers manually and are now able to process billing reports. CRM imports are now processing faster as well after scaling up workers.

  7. monitoring Jun 10, 2025, 09:54 PM UTC

    Placements is fully operational at the moment - our third party infrastructure provider has rolled out a fix. The platform has caught up on most backend tasks and CRM and ad server imports should be functioning normally again.

  8. resolved Jun 10, 2025, 10:00 PM UTC

    This incident has been resolved.