Dualog experienced a major incident on February 21, 2023 affecting Email and Automatic File Transfer, lasting 1h 20m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 21, 2023, 02:54 PM UTC
We are currently experiencing some issues with incoming emails to vessels. All emails are stored and will be delivered once the issue is rectified.
- investigating Feb 21, 2023, 03:00 PM UTC
We are currently experiencing some issues with incoming emails to vessels. All emails are stored and will be delivered once the issue is rectified. We are currently investigating and will revert when we have more information.
- identified Feb 21, 2023, 03:35 PM UTC
The root cause of the ongoing lack of mail traffic to and from vessels has been identified, and we are working to implement a fix to resume mail flow to and from vessels. At the moment there is no estimated resolution time, but the status page will be updated as the case progresses.
- monitoring Feb 21, 2023, 04:01 PM UTC
The ongoing issue has been solved and we have entered a monitoring phase. Once we are satisfied that mail is flowing without issues in both directions, we will issue a resolution message. Please allow some time for all ships to receive mail as the queue on our shore mail servers is actively being cleared.
- resolved Feb 21, 2023, 04:14 PM UTC
Mail traffic has resumed in both directions, and the queue on our mail servers is being expedited at a steady pace. The incident is to be considered resolved.
- postmortem Feb 27, 2023, 01:08 AM UTC
On February 21st, 2023, at 13:58 UTC, the Dualog cloud environment encountered an issue that caused a delay in email delivery. All ships were affected. The mail flow was restored on the same day at 16:01 UTC, and all messages that were sent during the outage window were processed and delivered accordingly. **Incident summary** On February 21st, 2023, at 13:58 UTC, we detected a performance issue that affected the shore-to-ship and ship-to-shore mail flow. The service outage caused a delay in email delivery, but no data was lost during the incident. All emails were stored in the system and queued for delivery once the backend was functioning. The IT Operations team investigated the issue and, after identifying the cause, deployed a solution and restored the mail flow at 16:01 UTC. All email traffic accumulated during the outage was processed accordingly, and the traffic flow quickly returned to normal shortly after. **Causes** The incident’s root cause was identified in a procedural oversight during system maintenance leading to an unexpected error. This resulted in an inconsistency that affected the system's stability, causing it to become degraded and consequently delaying emails. **Actions taken** Our team has since reviewed and updated our maintenance procedures to prevent similar incidents in the future.