Housecall Pro incident

Intermittent Server Load Issues

Minor Resolved View vendor source →

Housecall Pro experienced a minor incident on August 21, 2020 affecting Pro web and Text notifications and 1 more component, lasting 4h 55m. The incident has been resolved; the full update timeline is below.

Started
Aug 21, 2020, 03:03 PM UTC
Resolved
Aug 21, 2020, 07:59 PM UTC
Duration
4h 55m
Detected by Pingoru
Aug 21, 2020, 03:03 PM UTC

Affected components

Pro webText notificationsEmail notificationsOnline bookingAdd a job APIiOS Mobile AppAndroid mobile appCustomer job preview

Update timeline

  1. investigating Aug 21, 2020, 12:41 PM UTC

    We are currently investigating this issue.

  2. identified Aug 21, 2020, 12:59 PM UTC

    The issue has been identified and a fix is rolling out to our servers. You should start to see improved performance on both web and mobile.

  3. monitoring Aug 21, 2020, 01:02 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. investigating Aug 21, 2020, 01:16 PM UTC

    We are investigating follow on issues after our initial fix. Web and mobile apps may see degraded performance.

  5. investigating Aug 21, 2020, 02:03 PM UTC

    We are continuing to investigate server load issues. Web and mobile apps may see degraded performance.

  6. investigating Aug 21, 2020, 03:03 PM UTC

    We are experiencing an unforeseen conflict between software versions on Amazon Web Services that's causing a performance issue for some customers. All data is safe and backed-up, this is a load issue with the new version of the infrastructure, and the full engineering team is working directly with Amazon to resolve this issue.

  7. investigating Aug 21, 2020, 05:32 PM UTC

    We are continuing to investigate an unforeseen software versioning conflict with Amazon Web Services. All data is safe and backed-up, this is a load issue with the new version of the infrastructure, and the full engineering team is working directly with Amazon to resolve this issue.

  8. monitoring Aug 21, 2020, 07:19 PM UTC

    Performance has improved over the past hour as we continue to monitor the situation. Web and mobile apps should start to show improvement and begin to communicate with the system normally.

  9. resolved Aug 21, 2020, 07:59 PM UTC

    Web and mobile apps have resumed normal functionality.