Is Lettermint down?

Status: Investigating We are investigating intermittent server errors affecting the platform. We will keep you posted once we know more. Affected components Team API (Partial outage) Outgoing (Transactional) (Partial outage) Incoming (Partial outage) Sending API (Partial outage) SMTP Relay (Partial outage) Outgoing (Broadcast) (Partial outage) Webhooks (Partial outage) Click tracking (Partial outage) Open tracking (Partial outage) Custom domains (Partial outage)

Status: Resolved Between 18:26 and 18:30 UTC (20:26–20:30 CEST) on 15 October, some requests to the sending API and SMTP returned errors. What happened One connection in the pool that links our application to the primary database ended up in a read-only state. Requests that were given this connection could not write to the database, so the send requests among them failed with a server error. All other connections kept working normally, so only part of the traffic was affected during this window. Impact • Some /v1/send API requests and SMTP submissions were rejected with a temporary error. Messages from rejected requests were not accepted and not sent. Clients that retry on temporary errors will have resent them automatically. • Some webhook deliveries and status updates (such as delivery events) may have been delayed. • No data was lost or corrupted, and messages accepted before or after this window were not affected. Resolution We found the affected connection and removed it from the pool. Error rates were back to normal at 18:30 UTC. We are adding a safeguard so that connections in this state are detected and replaced automatically. If you saw failed sends during this window and your integration does not retry automatically, please resend those messages. Our sincere apologies for the inconvenience. Affected components Open tracking (Operational) Custom domains (Operational) Sending API (Operational) Outgoing (Transactional) (Operational) Incoming (Operational) SMTP Relay (Operational) Webhooks (Operational) Click tracking (Operational) Team API (Operational) Outgoing (Broadcast) (Operational)

Status: Identified We have identified the cause of delayed webhook delivery and are working on a fix. Webhook events are currently reaching your endpoints later than usual. No events are lost, they will be delivered once processing catches up. Sending emails through the API and SMTP is not affected, and incoming emails on inbound routes are processed normally. Affected components Webhooks (Degraded performance)

Status: Resolved This incident has been resolved. Webhooks are operating normally again. The backlog has also been fully processed and no more delays have been noticed. We are already actively engaged in working on a solution to prevent these delays in the future with the same cause. Apologies for the inconvenience this may have caused. Affected components Webhooks (Operational)

Status: Monitoring We have identified the cause of intermittent 500 errors affecting the SMTP relay and APIs and are working on a fix. A small percentage of requests and SMTP sessions were impacted by this. The issue is caused by a corrupted API pod which has now been disabled to mitigate the issue. There should be no more impact as of 8:07 PM CEST.

Status: Resolved This incident has been resolved. The API and SMTP relay are operating normally again. The incident affected a small percentage of requests between 7:56 PM CEST and 8:07 PM CEST. No messages have been lost that have been accepted. We are reviewing the incident and will take any appropriate follow-up measures. Affected components Open tracking (Operational) Sending API (Operational) Team API (Operational) Incoming (Operational) SMTP Relay (Operational) Click tracking (Operational)

Status: Investigating We are investigating service unavailable errors affecting multiple services. Our apologies for the inconvenience. We will share an update once we have more information. Affected components Sending API (Degraded performance) Team API (Degraded performance) Incoming (Degraded performance) SMTP Relay (Degraded performance) Webhooks (Degraded performance) Click tracking (Degraded performance) Open tracking (Degraded performance) Custom domains (Degraded performance)

Status: Identified We have identified an issue in the infrastructure that supports message processing. We are working to resolve it and will share another update when we have verified the result. Affected components Click tracking (Partial outage) Open tracking (Partial outage) Custom domains (Operational) Sending API (Full outage) Team API (Partial outage) Incoming (Full outage) SMTP Relay (Full outage) Webhooks (Full outage)

Status: Identified We are performing emergency maintenance to increase capacity. Brief service interruptions may occur during this work. We will provide another update when the maintenance is complete and service is stable. Affected components Team API (Degraded performance) Incoming (Partial outage) SMTP Relay (Partial outage) Webhooks (Partial outage) Click tracking (Partial outage) Open tracking (Partial outage) Custom domains (Operational) Sending API (Partial outage)

Status: Monitoring Email submissions and message processing have resumed. We confirmed two short interruptions: 18:51–18:57 and 19:02–19:04 CEST. Some tracking requests failed and webhook delivery was delayed during those periods. Later, some Sending API requests returned errors; the last such error we observed was around 20:04 CEST. We are monitoring sending and processing, and we are still checking incoming mail, before we resolve this incident. Affected components Incoming (Partial outage) SMTP Relay (Operational) Webhooks (Operational) Click tracking (Operational) Open tracking (Operational) Custom domains (Operational) Sending API (Operational) Team API (Operational)

Status: Resolved The affected services are operating normally, and we have completed the emergency maintenance. Message intake stopped during two short periods: 18:51–18:57 and 19:02–19:04 CEST. Some tracking requests failed, and webhook delivery was delayed during those periods. During the later emergency maintenance, some Sending API requests returned errors until about 20:04 CEST. We have monitored the services after the maintenance and are marking this incident resolved. We are sorry for the disruption. We will publish a public write-up after we confirm the full customer impact, cause, and follow-up actions. Affected components Open tracking (Operational) Custom domains (Operational) Sending API (Operational) Team API (Operational) Incoming (Operational) SMTP Relay (Operational) Webhooks (Operational) Click tracking (Operational)

Status: Investigating We are investigating failing requests and retries affecting SMTP relay. Affected components Click tracking (Degraded performance) Open tracking (Degraded performance) Sending API (Degraded performance) Team API (Degraded performance) Incoming (Degraded performance) SMTP Relay (Degraded performance)

Status: Resolved On September 1, between 04:01 and 04:06 UTC, one of our API instances returned errors for a small number of requests. Some customers sending email through our API or SMTP service might have seen a submission or authentication error. Other instances remained healthy, and most requests continued normally. No email already accepted by Lettermint was lost. Failed requests stopped before we accepted or created the message, so clients could retry them after service recovery. A small number of email-open events might not have been recorded. No stored customer data was changed or exposed. The cause was a defect in the PHP JIT. We already had a safeguard for this issue, but it did not cover this additional code path. We expanded the safeguard and removed the affected date calculation from the application startup process. This lets us keep the performance optimisation active for the rest of the application. We will also install the permanent PHP fix when it becomes available in our production runtime which at the time of writing is not the case. We continue to monitor the service. We are sorry for the disruption. Affected components Team API (Operational) Incoming (Operational) SMTP Relay (Operational) Click tracking (Operational) Open tracking (Operational) Sending API (Operational)

Status: Investigating We have been made aware of this issue reoccurring again and are currently investigating. The incident has already been mitigated by killing the affected pod. We will share an update once we know more. Affected components Click tracking (Degraded performance) Open tracking (Degraded performance) Sending API (Degraded performance) Team API (Degraded performance) Incoming (Degraded performance) SMTP Relay (Degraded performance)

Status: Resolved This incident has been resolved. SMTP relay and API are operating normally again. We have deployed further mitigation measures to prevent this incident from occurring again and are keeping a close eye. Apologies for the inconvenience this has caused. Affected components Sending API (Operational) Team API (Operational) Incoming (Operational) SMTP Relay (Operational) Click tracking (Operational) Open tracking (Operational)