Is Typesense down?

We are observing upstream network connectivity issues in one of the three data centers in the Bahrain region. Typesense Cloud clusters with High Availability enabled remain unaffected, as traffic is being routed away from the affected data center. Typesense Cloud clusters that do not have High Availability enabled will remain unavailable until network connectivity is restored.

Our infrastructure provider has completed their initial assessment, and confirmed that one of the three data centers in the Bahrain region have sustained structural damage due to drone strikes in the area. They estimate full recovery to take a prolonged amount of time given the extraordinary nature of the situation. HA clusters in this region continue to operate fine with two out of three nodes still functioning. However, given the volatility of the conflicts in the area, we recommend that users with clusters in this region plan to provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions.

The situation has not improved in the Middle East regions affected by the recent drone strikes. Our infra provider does not have an ETA on when they will be able to restore these data center facilities, due to the extraordinary nature and fluidity of the situation. So we recommend that users with clusters in the UAE and Bahrain regions provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. Since our infra provider's control plane is also down in these regions, it is not possible to access or export the existing data out of these regions. So it is not possible to clone or modify cluster configuration (like RAM, CPU, enabling HA, enabling SDN, etc) in these two regions. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of these two affected regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster.

A 2nd data center in the Bahrain region has been affected due to the ongoing conflict. We recommend customers provision a new cluster in a different region, and re-index their data from their primary database, and then update their applications to use the new cluster. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of the Bahrain and UAE regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster.

Our infrastructure provider is experiencing a power outage in one of the three data centers in the UAE region. Customers who have enabled High Availability for their Typesense Cloud Cluster continue to operate normally even with one node in the affected data center being down, as traffic is being re-routed away from the affected data center to the other two nodes in the other data centers in the region. Customers who do not have High Availability enabled for their Typesense Cloud Cluster and have their node running in the affected data center are affected by this issue, and will remain unavailable until power is restored.

A second data center in the UAE region has lost power, due to active military conflict in the region. An HA Typesense Cluster with 3 nodes can survive a maximum of 1 node failure, and with 2 data centers in the region now down, 2 nodes in HA clusters in the region are now inaccessible, which makes the 3rd node stop serving traffic to avoid split brain scenarios. Our infrastructure provider estimates that it might take at least a day to evaluate critical systems and their safety, before providing an ETA for full restoration. Given the severity and uncertainty of this event, we strongly recommend: 1. Provisioning a new cluster in a different region outside the Middle East. 2. Re-indexing data from your primary database. 3. Updating your application configuration to point to the new cluster. Please note: the cluster cloning feature is unavailable because the regional control plane is also affected.

Our infrastructure provider has completed their initial assessment, and confirmed that the two data centers in the UAE region have sustained structural damage due to drone strikes. They estimate full recovery to take a prolonged amount of time given the extraordinary nature of the situation. Like mentioned earlier, we recommend that users with clusters in this region provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions.

The situation has not improved in the Middle East regions affected by the recent drone strikes. Our infra provider does not have an ETA on when they will be able to restore these data center facilities, due to the extraordinary nature and fluidity of the situation. So we recommend that users with clusters in the UAE and Bahrain regions provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. Since our infra provider's control plane is also down in these regions, it is not possible to access or export the existing data out of these regions. So it is not possible to clone or modify cluster configuration (like RAM, CPU, enabling HA, enabling SDN, etc) in these two regions. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of these two affected regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster.