MyVenue experienced a critical incident on February 6, 2025 affecting Integrations and Real Time Updates, lasting 9h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 06, 2025, 11:48 PM UTC
Update as at 6:24pm 6 Feb 2025 EST Google Firebase is experiencing continued issues which is causing the following: Issues with remote printing, tables, KDS, Firebase updates and Pos online status on dashboard. MyVenue are continuing to investigate the cause of the issue and are working towards the restoration all services.
- investigating Feb 07, 2025, 02:21 AM UTC
Issue still under investigation and has been escalated to Google. Working: 1. Concessions 2. Pinpad & tips 3. Local printing 4. Table charging including credit cards & tips where a single POS terminal is used 5. Suites Event Day Orders Shared services still experiencing issues with: 6. Remote printing generally unless on the same Vlan 7. Kitchen Display Systems 8. Account charging 9. Loaded tickets and gift cards 10. Dashboards & event day reporting 11. Real time updates 12. Processing tokenized credit cards (credit cards on file) 13. Remote printing in Suites for Event Day Orders 14. Suites credit cards auth adjust MyVenue are continuing to investigate the cause of the issue and are working towards the restoration all services.
- investigating Feb 07, 2025, 04:05 AM UTC
SuitePOS payments have now been restored. (Credit Card tokenization, authorization, adjustments and payment completion) Google Firebase issue remains unresolved and has been escalated to the highest Google Firebase Support level. MyVenue are continuing to work towards the restoration of all services.
- monitoring Feb 07, 2025, 05:04 AM UTC
Google has made changes to the backoffice database. Services have started to be restored with time allocated to working through backlog. More details will be provided in a timely manner
- resolved Feb 07, 2025, 09:21 AM UTC
All MyVenue systems and processes have now been fully restored. The incident appears to have been caused by a Google server entering an unusual state, which occurred at approximately 5:30 PM ET. None of the available MyVenue or Google tools were able to identify or resolve the issue. Once the development team successfully engaged with the appropriate personnel at Google, the affected server was identified and replaced. The server replacement took place at 10:45 PM ET, with full operational capacity restored shortly thereafter. The MyVenue Risk Team has initiated an investigation into the root cause of the incident and will provide a detailed report in due course. We fully recognize that incidents of this nature impact venue operations and personnel at all levels. We deeply regret any disruption this may have caused and sincerely apologize for any inconvenience.