Cluvio incident

Sending schedules and exporting as PDF/PNG delayed

Cluvio experienced a minor incident on July 5, 2021 affecting Cluvio Web Application (EU) and Emails for Sql Alerts and Dashboard Schedules (EU), lasting 24m. The incident has been resolved; the full update timeline is below.

Started: Jul 05, 2021, 08:18 AM UTC
Resolved: Jul 05, 2021, 08:43 AM UTC
Duration: 24m
Detected by Pingoru: Jul 05, 2021, 08:18 AM UTC

Affected components

Cluvio Web Application (EU)Emails for Sql Alerts and Dashboard Schedules (EU)

Update timeline

investigating Jul 05, 2021, 08:18 AM UTC

We are investigating an issue with dashboard schedule emails being delayed and exporting PDF/PNG not completing
resolved Jul 05, 2021, 08:43 AM UTC

The email sending and dashboard exporting are working properly again. The slowdown was caused by clogging of the dashboard schedule job queue from an attempted abuse for distribution of spam. We have disabled the spamming account and will deploy better protection against this type of abuse later today.
postmortem Jul 12, 2021, 10:17 AM UTC

**What happened:** Sending of dashboard schedule and sql alert emails was delayed for about 2 hours. **Root cause:** An attacker created a trial account and by abusing our API triggered spam/phishing email sending. Our ops team was alerted about the abuse and blocked the account, and due to rate-limiting, the number of requests processed was limited, but there was still about 2 hour delay for the email sending due to the job processing queue being affected. **What we did to avoid these types of issues in the future:** To prevent this type of abuse, we tightened some aspects of the trial account usage \(which otherwise has full functionality without any restrictions\) regarding email sending for alerts, schedules and user invitations. As a result, abuse attempts like the one causing this issue will not be possible.