Panoply incident

Data sources failures

Critical Resolved View vendor source →

Panoply experienced a critical incident on April 28, 2022 affecting Data Sources, lasting 3h 54m. The incident has been resolved; the full update timeline is below.

Started
Apr 28, 2022, 02:18 PM UTC
Resolved
Apr 28, 2022, 06:13 PM UTC
Duration
3h 54m
Detected by Pingoru
Apr 28, 2022, 02:18 PM UTC

Affected components

Data Sources

Update timeline

  1. identified Apr 28, 2022, 02:18 PM UTC

    We've identified an issue that is affecting multiple different data sources. Our team is currently working on solving this issue and getting all the failed data sources to work.

  2. identified Apr 28, 2022, 02:39 PM UTC

    Update: This issue is currently affecting the following data sources: Postgres Mongo MsSQL ElasticSearch Jira BigQuery Hubspot MySQL SFTP WooCommerce Netsuite Google Sheets BigCommerce Shopify Google Analytics Google Cloud Storage

  3. identified Apr 28, 2022, 03:33 PM UTC

    The root reason for this issue can be found here: https://github.com/paramiko/paramiko/issues/2038 The team is still working on a solution for that on our end.

  4. monitoring Apr 28, 2022, 05:21 PM UTC

    We've deployed a fix to the issue and monitoring the status of all the different data sources

  5. resolved Apr 28, 2022, 06:13 PM UTC

    The issue was resolved and every failing data source should complete successfully the next time it will run.