LIKE MAGIC experienced a critical incident on November 17, 2024 affecting Opera Cloud, lasting 3h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 17, 2024, 08:11 AM UTC
We are currently investigating this issue.
- resolved Nov 17, 2024, 11:14 AM UTC
This incident has been resolved.
- postmortem Nov 19, 2024, 11:51 AM UTC
# Post-Mortem Report: Configuration Change Incident ## Incident Summary On Saturday, November 16, 2024, at 14:02 UTC, a configuration change was made while investigating an issue related to the room upgrade functionality. This configuration change contained an error that had the following impact: * Any changes initiated by LM in the PMS were no longer possible, including: * Changing guest information * Posting payments * Changing reservation statuses * However, any changes made directly within the PMS were properly received by LM. The issue was not immediately detected, and tickets reporting the problem were received later that day. The incident was actioned on Sunday, November 17, 2024. ## Timeline * Saturday, 16.11.2024, 14:02 UTC: Configuration change made containing the error * Saturday, 16.11.2024, 20:39 UTC: Tickets received reporting the issue * Sunday, 17.11.2024: Issue investigation began * Sunday, 17.11.2024, 09:28 UTC: Resolution implemented and issue resolved ## Root Cause The root cause of the incident was a misconfiguration in the PMS settings that prevented LM from making necessary changes to guest information, payments, and reservation statuses. ## Resolution Once the issue was identified, the configuration was corrected, and the system was restored to normal functionality. ## Lessons Learned and Action Items To prevent similar incidents from occurring in the future, we will: 1. Implement stricter controls and approval processes for critical configuration changes, such as PMS configs. 2. Improve testing procedures to catch potential misconfigurations before they are deployed to production. 3. Enhance monitoring and alerting systems to detect issues more quickly and minimize the impact on users. 4. Conduct thorough post-mortem reviews to identify areas for improvement in our processes and systems. ## Conclusion We apologize for any inconvenience caused by this incident. We are committed to learning from this experience and implementing the necessary changes to prevent similar issues from occurring in the future. If you have any further questions or concerns, please don't hesitate to reach out to our support team.