Felix incident

Loss of Service Due to AWS Outage

Critical Resolved View vendor source →

Felix experienced a critical incident on January 23, 2020 affecting Contractor Portal and Vendor Portal, lasting 2h 31m. The incident has been resolved; the full update timeline is below.

Started
Jan 23, 2020, 05:20 AM UTC
Resolved
Jan 23, 2020, 07:52 AM UTC
Duration
2h 31m
Detected by Pingoru
Jan 23, 2020, 05:20 AM UTC

Affected components

Contractor PortalVendor Portal

Update timeline

  1. investigating Jan 23, 2020, 04:00 AM UTC

    We are currently experiencing reduced service levels across the Felix platform due to an issue with an external service provider. Some users may experience 502 errors or occasional issues across the platform. Refreshing the page may resolve some of these issues.

  2. investigating Jan 23, 2020, 04:50 AM UTC

    We are still experiencing reduced service levels across all Felix applications due to issues with an external service provider. We will continue to provide updates as they come to light.

  3. investigating Jan 23, 2020, 04:53 AM UTC

    We are still awaiting full service restoration from an external provider

  4. investigating Jan 23, 2020, 05:22 AM UTC

    Please see https://status.aws.amazon.com/#AP_block for further information

  5. identified Jan 23, 2020, 05:30 AM UTC

    Due to an outage with Amazon AWS, parts of the Felix application are currently unavailable. We will provide updates as soon as they are received.

  6. monitoring Jan 23, 2020, 06:09 AM UTC

    Felix application accessibility is still intermittent. We are awaiting a further update from Amazon AWS.

  7. monitoring Jan 23, 2020, 06:12 AM UTC

    We will continue to monitor application uptime and provide updates where possible.

  8. monitoring Jan 23, 2020, 07:34 AM UTC

    Amazon AWS have indicated that services are beginning to come back online. Felix apps are available and all services appear to be recovering.

  9. resolved Jan 23, 2020, 07:52 AM UTC

    AWS have notified us that the issues have been resolved. Service has been restored to normal.