3stepIT incident

3CheckIT and Profiler Client data handling degraded

Major Resolved View vendor source →

3stepIT experienced a major incident on March 11, 2019, lasting 27d 22h. The incident has been resolved; the full update timeline is below.

Started
Mar 11, 2019, 08:55 AM UTC
Resolved
Apr 08, 2019, 07:36 AM UTC
Duration
27d 22h
Detected by Pingoru
Mar 11, 2019, 08:55 AM UTC

Update timeline

  1. monitoring Mar 11, 2019, 08:55 AM UTC

    The service is experiencing problems in handling device data reports sent by the Clients. Data is being taken in with a delay. Oldest reports yet to be handled are from 25.2. A fix has been attempted, performance is being monitored. Update: Server version of Profiler is also affected. This causes heartbeat reports to come in late.

  2. identified Mar 11, 2019, 08:57 AM UTC

    The issue has been identified and a fix is being implemented.

  3. identified Mar 11, 2019, 09:11 AM UTC

    We are continuing to work on a fix for this issue.

  4. identified Mar 11, 2019, 10:08 AM UTC

    We are continuing to work on a fix for this issue.

  5. monitoring Mar 11, 2019, 02:35 PM UTC

    Fixes are applied and we are monitoring the performance. Normal data handling performance has been regained and the service is now handling the queued data. More status update will be provided in Tue 12.3. Update to original message: Server version of Profiler is also affected. This causes heartbeat reports to come in late.

  6. monitoring Mar 12, 2019, 01:06 PM UTC

    Device data report queue is still being handled. We are hoping the queue to be emptied before Wednesday morning.

  7. monitoring Mar 13, 2019, 10:05 AM UTC

    Device data queue is now handling current days reports. We are still monitoring the data handling for exceptions.

  8. identified Mar 18, 2019, 02:28 PM UTC

    A new issue has been identified in data reader that currently causes problems in handling of data sent by the Clients if same customer has Profiler in use. A fix is being worked on, next update will be informed here on Tue 19.3. For Profiler, data reader problems are solved at this point.

  9. identified Mar 19, 2019, 02:31 PM UTC

    A fix will be attempted today outside of business hours.

  10. monitoring Mar 26, 2019, 06:39 AM UTC

    A fix has been implemented and we are monitoring the results.

  11. resolved Apr 08, 2019, 07:36 AM UTC

    Problem has been fixed.