ViiBE Invoice Generation

Update timeline

1. resolved Mar 01, 2025, 07:23 PM UTC

   ViiBE invoice generation did not initiate at 9:30 AM EST as expected. The issue was resolved at 2:15 PM EST and all invoices that were slated to be generated were processed successfully.

2. postmortem Mar 01, 2025, 07:23 PM UTC

   ## **Summary** On March 1st 2025, the scheduled invoice generation process for ViiBE failed to initiate at 9:30 AM EST as expected. The issue was identified and resolved by 2:15 PM EST, after which all invoices were successfully generated. ## **Impact** The failure to trigger invoice generation at the scheduled time resulted in a delay of approximately five hours and forty-five minutes. No invoices were lost, and all were eventually processed successfully. However, customers and internal stakeholders may have experienced delays in receiving invoice notifications. ## **Root Cause** Recent optimizations were made to streamline developers' local build processes by producing a single container image containing build artifacts for all services. This allowed faster local deployments to Kubernetes clusters. However, in the production environment, services are deployed using separate images, ensuring that only modified services are restarted. The difference in image deployment strategies led to a misconfiguration in the cron job definitions. While the build artifacts remained consistent across environments, the production deployment expected separate images per service. This mismatch resulted in the invoice generation cron job failing to trigger as expected. ## **Resolution** Once the issue was identified: 1. The cron job misconfiguration was corrected to align with the production deployment structure. 2. The affected images were updated, and the necessary services were redeployed. 3. A manual execution of the invoice generation process was performed to ensure all scheduled invoices were processed. ## **Preventive Measures** To mitigate the risk of similar issues in the future, the following actions will be taken: * **Enhance Configuration Validation**: Implement automated checks to validate cron job configurations before deployment. * **Consistency Between Local and Production Environments**: Standardize deployment approaches to minimize discrepancies between local development and production. * **Monitoring and Alerts**: Improve monitoring for scheduled jobs with proactive alerts when a cron job does not execute as expected. * **Pre-Deployment Testing**: Expand integration tests to cover scheduled jobs in a pre-production environment. ## **Conclusion** The issue was a result of a configuration oversight due to differences in build and deployment strategies between local development and production. The problem was identified and resolved with minimal long-term impact. Future preventive measures will help ensure more robust deployment practices and minimize the risk of similar failures.