Is Depict down?

For about five minutes (17:45–17:50 UTC), a subset of API requests for listings and recommendations served from one of our two regions returned errors (HTTP 500/504) while a database migration waited on a lock. The other region, search results and the merchant portal's data were not affected; no data was lost. The lock was released at 17:49 UTC and all requests recovered by 17:50 UTC. We are moving schema changes on these tables to a low-traffic window with a lock check before each step.

Resolved: the lock was released at 17:49 UTC and all requests had recovered by 17:50 UTC.

For about five minutes (17:45–17:50 UTC), a subset of API requests for listings and recommendations served from one of our two regions returned errors (HTTP 500/504) while a database migration waited on a lock. The other region, search results and the merchant portal's data were not affected; no data was lost. The lock was released at 17:49 UTC and all requests recovered by 17:50 UTC. We are moving schema changes on these tables to a low-traffic window with a lock check before each step..

Resolved: the lock was released at 17:49 UTC and all requests had recovered by 17:50 UTC..

Search requests for stores served by our runway search API are returning an error instead of results. We are investigating.

Between 18:00 and 18:05 UTC, search requests for stores served by our runway search API returned an error instead of results. Cause: during clean-up of a decommissioned cluster, a DNS record that the search service still used to reach its search index was removed automatically by an operator on the old cluster. The record was re-created at 18:05 UTC and search recovered immediately. Stores on the realtime API were not affected. We are moving the search service onto an internal path so a DNS change cannot affect it again, and adding paging to the synthetic check that detected this.

Search requests for stores served by our runway search API are returning an error instead of results. We are investigating..

Between 18:00 and 18:05 UTC, search requests for stores served by our runway search API returned an error instead of results. Cause: during clean-up of a decommissioned cluster, a DNS record that the search service still used to reach its search index was removed automatically by an operator on the old cluster. The record was re-created at 18:05 UTC and search recovered immediately. Stores on the realtime API were not affected. We are moving the search service onto an internal path so a DNS change cannot affect it again, and adding paging to the synthetic check that detected this..

We are currently investigating this incident.

We are continuing to work on a fix for this incident.