Resolved after 44 minutes, affecting Hire Central, on April 15, 2025.
### Summary Between **15:40 PM and 16:43 PM \+08:00**, Hire Central experienced service disruption due to an AWS infrastructure issue. Specifically, AWS reported a power loss event in a single Availability Zone \(`apne1-az4`\) in the Tokyo \(AP-NORTHEAST-1\) region, which impacted connectivity to a subset of EC2 instances. This resulted in: * Temporary unavailability of services hosted in the affected zone * Increased error rates and response time delays for certain API calls and features * Short-term degradation in user experience for some of our customers in the region ### Root Cause According to AWS, the issue stemmed from a **simultaneous interruption of primary and secondary power** to EC2 instances within the impacted zone. This hardware-level event caused affected instances to become unreachable and led to downstream issues in dependent AWS services. ### Mitigation & Resolution AWS engineers were engaged within minutes and worked to restore power and connectivity. By **4:43 AM SGT**, the issue was fully resolved, and our services were stabilized. Our engineering team monitored recovery closely and ensured: * Automated failover and retries were effective for most traffic * Manual restarts were initiated for impacted workloads where necessary * No data loss occurred, and all systems returned to normal performance levels post-recovery ### Preventive Measures While this incident was caused by a rare AWS infrastructure-level fault, we are taking the following steps to strengthen our platform's resilience: * Reviewing and rebalancing workloads across multiple Availability Zones * Increasing failover testing frequency for single-zone failure scenarios * Working with AWS support to assess early warning and hardware monitoring options ### Final Note We sincerely apologize for the inconvenience this incident may have caused. Ensuring high availability and reliability of our services remains our top priority. We appreciate your patience and understanding. For any follow-up concerns or assistance, please feel free to contact our support team. — **Company Central Operations Team**
Affected components
Update timeline
- investigating Apr 15, 2025, 07:56 AM UTC
We are currently experiencing connection timeouts in our infrastructure. Our engineering team is investigating the root cause and working to restore full functionality as soon as possible. We'll keep you updated on progress.
- monitoring Apr 15, 2025, 08:11 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 15, 2025, 08:41 AM UTC
This incident has been resolved.
- postmortem Apr 16, 2025, 04:03 PM UTC
### Summary Between **15:40 PM and 16:43 PM \+08:00**, Hire Central experienced service disruption due to an AWS infrastructure issue. Specifically, AWS reported a power loss event in a single Availability Zone \(`apne1-az4`\) in the Tokyo \(AP-NORTHEAST-1\) region, which impacted connectivity to a subset of EC2 instances. This resulted in: * Temporary unavailability of services hosted in the affected zone * Increased error rates and response time delays for certain API calls and features * Short-term degradation in user experience for some of our customers in the region ### Root Cause According to AWS, the issue stemmed from a **simultaneous interruption of primary and secondary power** to EC2 instances within the impacted zone. This hardware-level event caused affected instances to become unreachable and led to downstream issues in dependent AWS services. ### Mitigation & Resolution AWS engineers were engaged within minutes and worked to restore power and connectivity. By **4:43 AM SGT**, the issue was fully resolved, and our services were stabilized. Our engineering team monitored recovery closely and ensured: * Automated failover and retries were effective for most traffic * Manual restarts were initiated for impacted workloads where necessary * No data loss occurred, and all systems returned to normal performance levels post-recovery ### Preventive Measures While this incident was caused by a rare AWS infrastructure-level fault, we are taking the following steps to strengthen our platform's resilience: * Reviewing and rebalancing workloads across multiple Availability Zones * Increasing failover testing frequency for single-zone failure scenarios * Working with AWS support to assess early warning and hardware monitoring options ### Final Note We sincerely apologize for the inconvenience this incident may have caused. Ensuring high availability and reliability of our services remains our top priority. We appreciate your patience and understanding. For any follow-up concerns or assistance, please feel free to contact our support team. — **Company Central Operations Team**
Your product runs on more than Company Central
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services