Actito incident

(SOLVED) ENV-3 : Some test emails and campaigns are not sent

Notice Resolved View vendor source →

Actito experienced a notice incident on June 28, 2024 affecting Portal and API and 1 more component, lasting 1d 10h. The incident has been resolved; the full update timeline is below.

Started
Jun 28, 2024, 07:24 AM UTC
Resolved
Jun 29, 2024, 05:59 PM UTC
Duration
1d 10h
Detected by Pingoru
Jun 28, 2024, 07:24 AM UTC

Affected components

PortalAPIE-mailSMS

Update timeline

  1. investigating Jun 28, 2024, 07:24 AM UTC

    We currently have an issue on ENV-3, impacting campaigns send by API or scheduled in the UI. There is also errors in the UI while trying to manage campaigns. Our IT team is urgently working on fixing this problem.

  2. identified Jun 28, 2024, 07:52 AM UTC

    The issue has been identified and a fix is being implemented.

  3. identified Jun 28, 2024, 09:14 AM UTC

    We are continuing to work on a fix for this issue.

  4. identified Jun 28, 2024, 10:34 AM UTC

    We can confirm that the majority of campaigns are properly sent and delivered, We are still working on restoring the full service

  5. identified Jun 28, 2024, 11:50 AM UTC

    Sending services are working on most of licenses while some clients are still impacted Service restore is still in progress

  6. identified Jun 28, 2024, 01:08 PM UTC

    All technical teams are still working on the restoration of the service

  7. identified Jun 28, 2024, 03:24 PM UTC

    The incident cause is confirmed as a database cluster malfunction. And the restoration is in progress. The restoration process is taking longer than expected. Our team is actively working multiple plans in parallel to restore the service as soon as possible. Next update will be shared at 18:30.

  8. identified Jun 28, 2024, 04:40 PM UTC

    We are still working on the service restoration. Over night, the problem should have been resolved to come back to normal activities for tomorrow morning. Next update at 20:00.

  9. identified Jun 28, 2024, 06:03 PM UTC

    The process of restoring service is still underway. Next update at 9.30pm.

  10. identified Jun 28, 2024, 07:38 PM UTC

    Restore is still in progress. Next update at 11pm.

  11. identified Jun 28, 2024, 08:58 PM UTC

    We are continuing to work on a fix for this issue.

  12. identified Jun 28, 2024, 10:50 PM UTC

    We are continuing working on restoring the cluster.

  13. identified Jun 29, 2024, 06:07 AM UTC

    The team continues to troubleshoot to restore services as quickly as possible. We don't have ETA for final restore at the moment and will update you once we have additional information.

  14. identified Jun 29, 2024, 07:49 AM UTC

    Dear customers, 2 messages 1/ A majority of the licenses are capable of sending campaigns and test mails. Just the click and open rate reporting is currently affected, and showing 0% rate. Please check the following : - If you have sent campaigns since Friday morning, please check in the campaign reporting if you have delivered number above 0. If yes, it means that all the 100% emails have been delivered to end customers. - You can as well send you a test mail. If you receive it, it means that all campaigns have been and are well functioning. 2/ For Licenses that are currently unable to send email campaigns, we are continuing our efforts to restore the service as soon as possible.

  15. identified Jun 29, 2024, 11:56 AM UTC

    The restoration ended successfully. Service is recovering

  16. monitoring Jun 29, 2024, 01:20 PM UTC

    Version française ci-dessous Dear customers, Following the deterioration in service having affected a number of users, All services have been restored and are operational. 3 main messages 1/ You can send new campaigns and test campaigns. 2/ A number of campaigns scheduled to be sent on Friday have been blocked. These campaigns are now in "draft" status. We invite you to consult them on your "Manage email campaigns" page, to sort the most recent ones using the "Updated on" column and to relaunch them manually, if you still consider them appropriate. 3/ The campaigns scheduled for Saturday have been sent by the system. The campaigns scheduled for the future will also be sent normally. We sincerely apologise for any inconvenience caused. ----------- Chers clients, Suite a la dégradation de service ayant affecté un certain nombre d’utilisateurs, tous les services ont été rétablis et sont opérationnels. 3 messages principaux : 1/ Vous pouvez envoyer de nouvelles campagnes et des campagnes de test. 2/ Un certain nombre de campagnes prévues pour envoi vendredi ont été bloquées. Pour ces campagnes, elle sont à présent en statut "draft". Nous vous invitons à les consulter dans votre page "gérer les campagnes email", à trier les plus récentes avec la colonne "Mise à jour" et à les relancer manuellement, si vous les jugez encore appropriées. 3/ Les campagnes programmées pour samedi ont été envoyées par le système. Idem pour les campagnes programmées dans le futur, elles seront envoyées normalement. Nous nous excusons sincèrement pour la gêne occasionnée.

  17. resolved Jun 29, 2024, 05:59 PM UTC

    This incident has been resolved.