Hedera experienced a minor incident on March 23, 2023, lasting 4d 22h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Mar 23, 2023, 09:57 PM UTC
This update is meant for testnet mirrornode operators using the GCP bucket. In an effort to address technical debt from the last testnet reset, we cycled a credential on GCP. Unfortunately, we inadvertently deactivated a key, which led to the temporary disabling of uploaders. We have since re-enabled the uploaders. However, backfilling the record stream is taking longer than expected, and estimate it will be fully resolved within the next 4-5 days. During this period testnet mirrornode operators relying solely on the GCP bucket may experience halted operations. As a temporary workaround, we recommend using the AWS bucket in the meantime. Please note that this issue does not impact the Hedera-managed testnet mirrornode; it only affects operators running their own mirrornodes. We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve this issue. We will provide updates on the status page as the situation progresses.
- monitoring Mar 23, 2023, 11:00 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 28, 2023, 08:43 PM UTC
Backfill of the GCP bucket has been completed. Both the GCP and AWS buckets for testnet mirror nodes are at parity.