POS Portal experienced a notice incident on March 16, 2020, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Mar 16, 2020, 10:06 PM UTC
On February 13, 2020 all API requests to POS Portal API Web services began receiving error response message "The provided Timestamp header is outside of the acceptable bounds. Please retry using the current time.”. This impacted API requests from Portal Cloud, Portal Access, Direct Merchant eCommerce, and Re-seller eCommerce.
- postmortem Mar 16, 2020, 10:06 PM UTC
**Issue Summary** On February 13, 2020 all API requests to POS Portal API Web services began receiving error response message "The provided Timestamp header is outside of the acceptable bounds. Please retry using the current time.”. This impacted API requests from Portal Cloud, Portal Access, Direct Merchant eCommerce, and Re-seller eCommerce. **Timeline \(all times Pacific Standard Time\)** * 2/13/2020 - 22:42 AM: Outage started * 2/14/2020 – 20:53 PM: Issue resolved **Root Cause** The issue was identified that one of the API back-end system clock was out of synchronization with official time sources and caused a time drift 182 seconds. This difference in time between client and server caused API authentication errors and rejected all API requests. **Resolution and recovery** System clocks were manually synchronized with official time sources. **Corrective and Preventative Measures** * Implemented a control that periodically checks for time drift and adjusts accordingly to official time sources. * A review of all system configurations will be completed. Additionally, a review of all Production environments will be completed to ensure all systems are in compliance standard and monitoring.