POS Portal experienced a notice incident on February 9, 2020 affecting POS Portal API, lasting 33s. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 09, 2020, 08:39 PM UTC
We are currently investigating this issue.
- resolved Feb 09, 2020, 08:40 PM UTC
This incident has been resolved.
- postmortem Feb 11, 2020, 11:49 PM UTC
**Issue Summary** On February 9, 2020 all API requests to POS Portal API Web services began receiving error response message "The provided Timestamp header is outside of the acceptable bounds. Please retry using the current time.”. This impacted API requests from Portal Cloud, Portal Access, Direct Merchant eCommerce, and Re-seller eCommerce. **Timeline \(all times Pacific Standard Time\)** * 11:35 AM: Outage reported * 12:02 PM: Notified IT On-Call support * 12:11 PM: Issue was identified * 12:15 PM: Issue was resolved * 12:17 PM: API service restored **Root Cause** The issue was identified that one of the API back-end system clock was out of synchronization with official time sources and caused a time drift of more than 3 minutes. This difference in time between client and server caused API authentication errors and rejected all API requests. **Resolution and recovery** System clocks were manually synchronized with official time sources **Corrective and Preventative Measures** · Automatic clock synchronization tool installed to prevent further time drifts on the identified servers. A full review of all Production servers will be completed and Hardening Policies will be reviewed and updated as needed. · Monitoring and alerting of clock synchronization and drift is being implemented.