OpsGenie experienced a minor incident on November 18, 2023 affecting Reporting & Analytics and Reporting & Analytics, lasting 6d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 18, 2023, 11:41 AM UTC
Scheduled report functionality is disabled as we suspect a possible spam activity. Only the reports with a custom schedules are disabled, periodic emails are not impacted. The team is investigating the issue and will provide more update.
- identified Nov 18, 2023, 03:52 PM UTC
The cause of the issue is identified, and the team is working on the fix.
- identified Nov 18, 2023, 06:53 PM UTC
The root cause is identified and the spam activity is mitigated. The team is working on adding more controls to prevent further spam activities. The scheduled report feature will be kept disabled for a while until the further controls implemented. However, the reporting service is fully available and the reports can be downloaded manually via the reporting page.
- identified Nov 23, 2023, 03:05 PM UTC
We have implemented additional controls and are introducing domain restrictions as well as limitations to the number of recipients for mitigation purposes. Scheduled reports will be enabled for all customers on November 24th, PST. Users will begin receiving emails for the reports they have scheduled for themselves and they will also have the ability to create new tasks. We will share more updates when the changes are fully implemented.
- monitoring Nov 24, 2023, 08:32 AM UTC
The changes have been shipped to production, and scheduled reports are enabled now.
- resolved Nov 24, 2023, 02:23 PM UTC
We're excited to inform you that we've shipped upgrades to our production environment, enabling scheduled reports once again. What's Changed: To continuously improve and ensure the security of our services, we've implemented additional controls including domain restrictions and a limitation on the number of recipients per email. This is specifically for mitigation purposes. From now on, users will start receiving emails for the reports they've scheduled for themselves, and they will also have the ability to create new tasks. Impact: Please note that any existing scheduled jobs with external recipients will no longer be editable. However, users can delete these and create new jobs using their email IDs. Thank you for your patience during these changes. We want to assure you that future updates and communications will be shared promptly to keep you informed. We appreciate your understanding and continued support.