POS Portal experienced a critical incident on January 31, 2020 affecting Portal Access and POS Portal API and 1 more component, lasting 2h 14m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 31, 2020, 12:47 PM UTC
The issue appears to be with multiple internet upstream providers. An emergency maintenance is being performed until 0500 PT, as well as splicing is underway. A Major Incident has been declared and Senior Engineers are actively working to re-mediate the issue as soon as possible. We will continue to provide updates as they are available.
- identified Jan 31, 2020, 12:47 PM UTC
We are continuing to work on a fix for this issue.
- identified Jan 31, 2020, 01:36 PM UTC
Systems are back up and running at this time. We are still leaving this as Identified as we have not received final update that maintenance has been completed. Until this the final update has been provided, systems may experience additional downtime.
- monitoring Jan 31, 2020, 02:10 PM UTC
Current state is all systems are operational. The status of all applications is being changed to Degraded Performance until all maintenance is completed. The issue has been addressed at the data center. The carriers are still have maintenance ongoing. Until all maintenance has been complete, the incident will not be resolved.
- resolved Jan 31, 2020, 03:01 PM UTC
Current state is all systems are fully operational. Carrier maintenance has been completed and the incident is now resolved.