Inbound emails not received by Graia email channels

Update timeline

1. resolved Oct 06, 2025, 07:00 AM UTC

   Type: Incident Affected Components: CCaaS International - Email Services Oct 6, 07:00:00 GMT+0 - Investigating - We are experiencing a problem receiving inbound emails at the moment. We are investigating this with high priority. Outbound emails are working. Nov 25, 09:30:00 GMT+0 - Investigating - Related DevOps workitems: Incident 138175: Inbound emails not arriving into Graia Bug 136453: emailbots pod using 3.3 GiB Oct 8, 09:42:00 GMT+0 - Investigating - Root Cause: With the release of a bugfix pertaining to the emailbots service, another issue was unfortunately introduced: the “Wait For Customer Reply” email workflow node works in a way that it periodically checks whether the awaited reply has arrived or the timeout reached. If it needs to wait some more, it signals this with delayed queue messages. The introduced bug was that in such cases, the service not only queued a new message but it also requeued the original one. This led to exponential growth of messages in the queue. Because the messages are drained sequentially, newer messages that got introduced after the exponentially duplicated ones were waiting for their turn to be processed. This meant the email conversations were stuck at a point in their workflow that required the processing of messages in the queue. Oct 6, 20:29:02 GMT+0 - Investigating - The incident has been resolved