Brillium incident

Intermittent Performance Issues

Minor Resolved View vendor source →

Brillium experienced a minor incident on August 22, 2024 affecting Assessment Authoring, lasting 1h 5m. The incident has been resolved; the full update timeline is below.

Started
Aug 22, 2024, 01:55 PM UTC
Resolved
Aug 22, 2024, 03:01 PM UTC
Duration
1h 5m
Detected by Pingoru
Aug 22, 2024, 01:55 PM UTC

Affected components

Assessment Authoring

Update timeline

  1. investigating Aug 22, 2024, 01:55 PM UTC

    We are currently experiencing intermittent issues that may result in a 503 error being displayed. Refreshing the browser will resolve the issue and display the expected screen properly. The systems operations team is currently working to resolve the issue as quickly as possible.

  2. investigating Aug 22, 2024, 02:00 PM UTC

    The source of the issue has been identified. Root cause analysis is currently underway.

  3. monitoring Aug 22, 2024, 02:32 PM UTC

    The issue has been identified and a fix has been implemented. We are currently monitoring systems to ensure all issues are fully resolved.

  4. resolved Aug 22, 2024, 03:01 PM UTC

    Monitoring has confirmed the issue has been resolved.

  5. postmortem Aug 22, 2024, 03:14 PM UTC

    **ISSUE** After a security update, monitoring services reported that some customers may be experiencing intermittent 503 errors. This issue impacted a small number of customers. **ROOT CAUSE ANALYSIS** Investigation by our DevOps team identified the root cause as a configuration issue that affected an underlying systems application web service. **RESOLUTION** A revised web service configuration was deployed to resolve the issue. Further monitoring indicated that the service was functioning normally and as expected. **NEXT STEPS** The DevOps team is conducting further analysis to prevent the issue from resulting in any future service degradation.