Xray experienced a major incident on April 29, 2023 affecting Xray App and Xray API, lasting 2d 12h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 29, 2023, 11:04 PM UTC
Xray cloud is currently experiencing an outage. We are working hard to resolve the problem and bring all services back online. Thank you for your patience.
- monitoring Apr 30, 2023, 02:11 AM UTC
Hi everyone, The outage has been resolved but we are not yet back to 100% capacity. So there might be some longer response times. Thank you for your patience.
- monitoring May 01, 2023, 07:44 AM UTC
Hi team, We understand the implications this is having on your work and sincerely apologies for the inconvenience caused, we are actively working on getting everything to 100% as soon as possible. Thank you for your patience.
- identified May 01, 2023, 11:40 AM UTC
Hi everyone, We are aware that Xray is still not working properly for some users and are actively working on resolving this with utmost priority. Thank you for your patience.
- identified May 01, 2023, 05:27 PM UTC
Hi team, We are aware of the ongoing issue, where users are not able to load data pertaining to Xray. We are doing all we can to resolve the situation. This might be in the form of screens not loading or status code 504 being returned. Thank you for your understanding and we will follow up with updates as soon as possible.
- monitoring May 02, 2023, 01:48 AM UTC
We have identified the issue with our DB cluster and completed the necessary resolution steps. However, index rebuilding is still in progress and may take approximately three more hours. During this time, a subset of users may experience slow performance. We will continue to monitor this, and provide another update once the index rebuild is complete. If you encounter any issues now that the problem has been addressed, please contact Support. Thank you for your patience and understanding.
- resolved May 02, 2023, 11:25 AM UTC
We are pleased to announce that the intervention to our systems has been successfully completed and the index rebuild is done. All systems should be functioning as expected. However, if you encounter any issues now that the problem has been resolved, please do not hesitate to contact Support. We apologize for any inconvenience caused by this intervention and we appreciate your patience and understanding.