PayPlug incident

Incident terminaux de paiement NEXO / Payment terminal incident NEXO

Critical Resolved View vendor source →

PayPlug experienced a critical incident on February 3, 2024, lasting 1h 37m. The incident has been resolved; the full update timeline is below.

Started
Feb 03, 2024, 11:35 AM UTC
Resolved
Feb 03, 2024, 01:13 PM UTC
Duration
1h 37m
Detected by Pingoru
Feb 03, 2024, 11:35 AM UTC

Update timeline

  1. investigating Feb 03, 2024, 11:35 AM UTC

    ---------------------------------------------------------- FR Un incident impacte les transactions des terminaux de paiement opérant sur Nexo (matériel PAX), nous investiguons pour rétablir la situation au plus vite. EN An incident affects payment terminal transactions processed on Nexo protocol (usually PAX terminals) Our technical team is investigating to restore the situation as quickly as possible.

  2. investigating Feb 03, 2024, 12:04 PM UTC

    FR Les équipes techniques sont pleinement mobilisées, nous poursuivons l'investigation pour rétablir les services au plus vite. EN Our technical teams are fully mobilized and we are continuing our investigations to restore services as quickly as possible.

  3. investigating Feb 03, 2024, 12:25 PM UTC

    FR Les équipes techniques sont pleinement mobilisées, nous poursuivons l'investigation pour rétablir les services au plus vite. EN Our technical teams are fully mobilized and we are continuing our investigations to restore services as quickly as possible.

  4. identified Feb 03, 2024, 12:45 PM UTC

    FR La cause de l'incident a été identifiée, nos équipes travaillent à la pleine restauration des services. EN The incident cause has been identified, and our teams are working to restore full service.

  5. monitoring Feb 03, 2024, 12:46 PM UTC

    A fix has been implemented and we are monitoring the results.

  6. monitoring Feb 03, 2024, 01:01 PM UTC

    We are continuing to monitor for any further issues.

  7. resolved Feb 03, 2024, 01:13 PM UTC

    FR Les services sont pleinement restaurés depuis 13h41 UTC+1. Un post-mortem sera publié rapidement. EN Services have been fully restored since 13h41 (1:41pm) UTC+1. A post-mortem will be updated shortly.

  8. postmortem Feb 06, 2024, 05:10 PM UTC

    **Service concerné** Paiements de proximité NEXO \(terminaux PAX\) **Impact client** Interruption des paiements de proximité NEXO de 10h50 à 13h42 **Synthèse de l’incident** 10h50 : expiration d’un certificat lié au trafic NEXO et **début de l’incident** 11h46 : alertes marchand 11h50 : ouverture de la cellule de crise 13h30 : détection du certificat expiré 13h42 : solution temporaire permettant la **reprise du trafic** Nexo 14h06 : mise à jour du certificat et **fin de l’incident** **Root cause** Expiration d’un certificat lié au trafic NEXO **Contexte** Incident durant le weekend sans l’expertise de toute l'équipe technique **Actions à entreprendre par Payplug** | **Symptôme** | **Action** | | --- | --- | | Expiration du certificat | Inventaire des certificats répartis sur nos datacenters et sur GCP réalisé cette semaine par l’ensemble de l'équipe infrastructure \(plus d’une centaine de certificats\) Ajout de monitoring sur chaque certificat inventorié avant mise en place d’une solution de gestion des certificats. | | Non détection de la panne | Ajout d’un alerting sur les paiements NEXO | | Délai de résolution important | Revue de la documentation NEXO Formation des équipes d’astreinte | ‌‌ ==============ENGLISH VERSION============== ‌ **Affected service** NEXO proximity payments \(PAX terminals\) **Customer impact** NEXO in store payment outage from 10.50am until 1.42pm **Incident overview** 10\.50am : expiry of a certificate linked to NEXO trafic. **Incident starts** 11\.46am : merchant alerts 11\.50am : crisis unit set up 1\.30pm : detection of the expired certificate 1\.42pm : workaround setup. **Service resumes.** 2\.06pm : expired certificate update. **End of the incident** **Root cause** Expiry of a certificate linked to NEXO trafic **Background** Incident during the wekend without full technical team expertise **Actions to be taken by Payplug** | **Symptom** | **Action** | | --- | --- | | Certificate expiry | Inventory of certificates spread accross our datacenters and GCP is to be performed this week by the whole infrastructure team \(more than a hundred certificates\) Monitoring added for each listed certificate before deployment of a certificate management solution | | Undetected failure | Alerting added on NEXO in store payments | | Large resolution delay | NEXO documentation review On-duty team training |