Greenshades incident
Partial Service Outage Affecting Critical Products
Greenshades experienced a notice incident on August 12, 2025 affecting Tax Service and Employee Services and 1 more component, lasting 7h 38m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 12, 2025, 07:05 PM UTC
We are currently experiencing a service disruption impacting several critical products. Our teams are actively working to restore full functionality and minimize any impact to users. We appreciate your patience and will continue to provide updates as more information becomes available.
- identified Aug 12, 2025, 07:35 PM UTC
We’ve identified the root cause of the service disruption. An internal message queuing services is currently experiencing communication failures, which is affecting multiple connected components across our platform. Our engineering teams are actively working on a resolution and will continue to provide updates as progress is made. Thank you for your continued patience.
- identified Aug 12, 2025, 08:59 PM UTC
Remediation of underlying issues especially for users of Benefits, Timesheets, and/or Timesheets + Payroll product offerings. Known outstanding issues still actively being addressed for PTS products.
- identified Aug 12, 2025, 09:34 PM UTC
Upon further investigation, new issues have been discovered with Payroll products, specifically around payroll completion and Pay History (i.e. Check Printing). PTS issues are currently expected to be resolved.
- identified Aug 13, 2025, 01:05 AM UTC
To continue to address the issues and ensure full restoration of services, Greenshades Online will be under maintenance starting at 9:00 PM EDT and concluding by 12:00 AM EDT. We appreciate your patience and understanding during this time. Our team is actively working to resolve the issue, and we will continue to provide updates as they become available. Thank you for your continued support.
- monitoring Aug 13, 2025, 02:33 AM UTC
All services are up and running. Things are looking good. We are monitoring for any additional issues.
- resolved Aug 13, 2025, 02:44 AM UTC
This incident has been resolved. The root cause was identified as an issue with our event handler. After a full restart during the maintenance period, we are seeing proper operation on all products and services. We appreciate your patience and understanding during this time.