Lob incident
Lag in QR Code Analytics - Reporting Tools Affected
Lob experienced a minor incident on March 24, 2026 affecting API and Dashboard and 1 more component, lasting 2d 18h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 24, 2026, 07:42 PM UTC
Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. Lob's QR Code Engagement visualizations in the "Mail Analytics" and "Campaigns" sections of the dashboard are missing some data that was created between Monday, March 16 and today, March 24. Only the data processing system was impacted - no original QR code scan data has been lost. Customers can still access this data if they make a GET call to Lob's APIs, or if they have webhook ingestion already set up. We've identified the issue and are actively backfilling the QR Code Engagement Chart with the missing records. We anticipate completing this backfill no later than tomorrow morning - Wednesday, March 25.
- identified Mar 25, 2026, 01:28 PM UTC
Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. Lob's QR Code Engagement visualizations in the "Mail Analytics" and "Campaigns" sections of the dashboard are missing some data that was created between Monday, March 16 and Tuesday, March 24. Only the data processing system was impacted - no original QR code scan data has been lost. Customers can still access this data if they make a GET call to Lob's APIs, or if they have webhook ingestion already set up. We've identified the issue and new scan events are now visible in the QR Code Engagement Chart. We are actively backfilling scans from the outage period that are missing from the chart. We've updated our anticipated date of resolution for completing this backfill - this should be complete no later than the evening of Friday, March 27.
- resolved Mar 27, 2026, 01:51 PM UTC
Backfills are complete. Additionally, Lob's engineering team has put improvements in place to prevent similar issues in the future, and updated our monitoring and alerting to catch similar issues if they do recur. Please reach out to [email protected] if you are still experiencing issues!