pingoru pingoru Green Security had 17 outages in the last 2 years totaling 263h 41m of downtime, averaging 0.7 incidents per month.
There were 17 Green Security outages since November 15, 2024 totaling 263h 41m of downtime. Each is summarised below. Incident details, duration, and resolution information.
We are investigating an issue with SSO SAML configurations returning 500 errors.
We have identified the issue and are rolling out a fix right now.
This incident has been resolved.
We are currently investigating an issue where an increased number of timeout responses to API calls.
We have identified and mitigated the issue. All requests are returning to normal now.
This incident has been resolved.
We are currently investigating an issue where the patient search integration with Epic/Cerner through the FHIR interface is experiencing an increase in `invalid_client` errors.
We have identified the issue and are working through a fix.
We have identified the issue as a rotated JWK Set key that was not picked up by Epic. Your CSM will reach out with a new per-tenant JWK Set URL (newly required best practice from Epic) to re-fetch the newest keys for your instance. These URLs properly handle key rotations.
We have restored the previous keys yesterday and via monitoring have found that all patient search functionality is working as it has previously. We encourage all customers to reach out to their CSM to plan for a migration to a tenant specific JWK Set URL.
We are currently investigating the issue. Once fixed, we will remove violations that should not have occurred for all affected vendors. In the meantime, please ensure that your Green Security for Vendors app is updated to the latest available version.
We have identified the source issues and will have a fix in the next 30 minutes.
A fix has been implemented and we are monitoring the results.
We are monitoring the fix as well as removing any erroneous vendor scan violations.
The issue has been resolved completely.
We are investigating reports of screening questions being required for some mobile check-ins where they should not be.
We have identified the source of the issue and are currently working on a fix.
We have identified the core issues. The fix should be live in the next 20 minutes. The following issues will be fixed in a follow up release later today: * A successful spot check appears in the log after a successful login in some situations * For facilities with RepScrubs SCRUBPORTS screening questions or full check-in questions will not result in an in-app notification (this is the same behavior as before today)
Check-ins are successful and we continue preparing fixes for the following: * A successful spot check appears in the log after a successful login in some situations * For facilities with RepScrubs SCRUBPORTS screening questions or full check-in questions will not result in an in-app notification (this is the same behavior as before today)
This issue has been resolved
We're experiencing an elevated level of Kiosk errors and are currently looking into the issue. Some kiosks are stuck at the "Checking for internet connection..." screen.
The issue has been identified and a fix is being rolled out soon.
A fix has been implemented and we are monitoring the system closely. Kiosks should come back up automatically. Restarting the Kiosk app may get your kiosk up faster.
This incident has been resolved.
We're experiencing an elevated level of system errors and are currently looking into the issue. We are investigating an issue with duplicated emails sent to vendors. We are investigating login/SSO issues for hospital employees.
We have resolved login/SSO issues as well as the duplicate emails being sent out to vendors. We are working with our email provider to fix email delivery issues.
This incident has been resolved
We are investigating a delay in email notifications. This could impact invitations, 2FA codes and reports. For accounts logging in with a confirmed phone number, it is possible to "resend" a code via phone number instead of email until the issue is resolved.
We have implemented a fix to catch up on communications. We anticipate all communication to be nearly instant within the next 30 minutes as we work through the backlog of pending emails.
This incident is resolved and all communication should be happening near instantly going forward.
The issue has been identified and a fix is being implemented.
We have fixed the visitor check-ins view and label printing issues and are monitoring the platform as the fix rolls out.
This incident has been resolved.
Salesforce/Heroku is experiencing intermittent outages which affects our platform. They are actively investigating and looking for a fix. In the meantime, please retry any operation that times out or is slow. https://status.heroku.com/incidents/2822
The issue has been identified and a fix is being implemented. Salesforce/Heroku is working to mitigate the issues. https://status.salesforce.com/generalmessages/10001540 Heroku is a platform as a service (PaaS) that helps developers deploy, manage, and scale their applications in the cloud. Because Green Security uses Heroku to manage its infrastructure, our platform is currently affected by this outage.
This incident has been resolved.
We are currently investigating an increase in undeliverable 2FA text messages. If you are having trouble logging in, please select "Resend Code" and select "Email" until the issue is resolved.
It appears the issue is localized to AT&T mobility customers. The issue is on the AT&T side. Please check https://www.att.com/outages/ to see if you are affected. If you are having trouble logging in, please select "Resend Code" and select "Email" until the issue is resolved.
We are monitoring deliverability.
We are seeing normal deliverability through AT&T Mobility network at this time.
We're experiencing an elevated level of SAML authentication errors. We have identified the issues and implemented a fix. We are monitoring the system.
Our provider for the 866 phone number is currently experiencing an outage. Support is available via 716-898-0615 or [email protected] until the 866 number is operational again.
The root cause has been identified and our upstream providers are working to restore service for the 866 number. Support is available via 716-898-0615 or [email protected] until the 866 number is operational again.
The issue is now resolved.
We're experiencing an elevated level of errors and are currently looking into the issue.
We have resolved most of the issues causing the partial outages. We continue to monitor to ensure that all issues are corrected.
This incident has been resolved.