Aptible experienced a minor incident on September 26, 2024 affecting Aptible Deploy, lasting 3h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 26, 2024, 04:15 PM UTC
We are investigating abnormally long registration times for new targets with AWS Load Balancers. This may be causing extended operation times for releases (Deploy, Scale, Restart) for services that have Endpoints.
- identified Sep 26, 2024, 04:24 PM UTC
This service impact only applies to resources hosted in the `us-east-1` region. Customers may notice operations reaching timeout, but at this point all operations are rolling back successfully to the previous state.
- identified Sep 26, 2024, 04:37 PM UTC
AWS has acknowledged the impact we are seeing and opened an incident: > We are investigating increased load balancer back-end instance registration times in the us-east-1 Region. September 26, 2024 at 16:21:43 UTC Since 16:05 UTC, Aptible is observing some recovery, about half of endpoint target registrations are succeeding at this time.
- monitoring Sep 26, 2024, 04:57 PM UTC
Loadbalancer registration appears to be working as expected at this time. We will continue to monitor operations until AWS resolves their service degradation notice.
- identified Sep 26, 2024, 05:52 PM UTC
We're again seeing degradation and failure to register new load balancer targets in about about 10% of running operations.
- identified Sep 26, 2024, 06:06 PM UTC
The latest update from AWS indicates that operations created around 17:10 through 17:20 UTC were impacted, which matches our internal metrics. AWS has promised another update by 18:00 UTC, and we will continue to monitor the situation until we are satisfied that it is resolved.
- resolved Sep 26, 2024, 07:26 PM UTC
AWS has marked this issue RESOLVED as of 19:19 UTC, and we have not observed any issues in the last hour. The issue has been resolved and all services are operating normally.