Kalix EMR experienced a major incident on November 9, 2021 affecting Messaging, lasting 14h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 09, 2021, 08:05 PM UTC
We have identified the issue that is causing faxes to error with an unknown error. This is not related to the ongoing message status issue. We should hopefully have a patch for this issue soon, however we have to be careful when we make the release so this may not happen until this evening.
- monitoring Nov 09, 2021, 11:25 PM UTC
It looks like the fax failed issue was due to downtime in our provider, due to a DDOS attack on their servers which lasted a total of about 2 hours. It seems like the attack is ongoing, but they have mitigations in place. Faxes should send successfully if you resend them (there is a button in Kalix on the message that allows you to do this). We are monitoring this and will change this to Resolved once the attack has abated.
- resolved Nov 10, 2021, 10:10 AM UTC
Our provider has improved their network in order to mitigate DDOS attacks in the future. It looks like their api has stabilized so there should not be ongoing issues with the faxing in Kalix.