GUIDEcx Outage History

Started Oct 09, 2026, 08:27 PM UTC· resolved Oct 09, 2026, 09:03 PM UTC

1. investigating Oct 09, 2026, 08:27 PM UTC

   We are currently investigating this incident..

2. investigating Oct 09, 2026, 08:27 PM UTC

   We are currently investigating this incident.

3. monitoring Oct 09, 2026, 08:35 PM UTC

   We implemented a fix and are currently monitoring the result..

4. monitoring Oct 09, 2026, 08:35 PM UTC

   We implemented a fix and are currently monitoring the result.

5. resolved Oct 09, 2026, 09:03 PM UTC

   This incident has been resolved. Some customers experienced errors and degraded performance in our web app. The issue was caused by a synchronization problem between two of our internal services, which prevented a translation-related request from completing. Our team identified the cause and restored full service. We apologize for the disruption and appreciate your patience. We’ll continue to monitor our systems closely..

6. resolved Oct 09, 2026, 09:03 PM UTC

   This incident has been resolved. Some customers experienced errors and degraded performance in our web app. The issue was caused by a synchronization problem between two of our internal services, which prevented a translation-related request from completing. Our team identified the cause and restored full service. We apologize for the disruption and appreciate your patience. We’ll continue to monitor our systems closely.

Read the full incident report →

Started Aug 31, 2026, 03:00 PM UTC· resolved Aug 31, 2026, 03:00 PM UTC

1. resolved Aug 31, 2026, 03:00 PM UTC

   This incident has been resolved. More details will follow in the RCA.

2. resolved Aug 31, 2026, 03:00 PM UTC

   .

3. investigating Aug 31, 2026, 05:17 PM UTC

   August 31, 2026 **Summary** On August 31, 2026, an issue occurred with our email notification queue that affected providers using GUIDEcx 1.0\. A legacy email service outage over the weekend caused outbound jobs to accumulate, resulting in duplicate email notifications being sent when the jobs were restarted. The duplicate sending was contained, and email services are operating normally. **Impact** Some providers associated with GUIDEcx 1.0 accounts received multiple duplicate email notifications during the morning hours. The incident was identified and contained earlier this morning. System monitoring confirms that no additional duplicate emails are being generated or sent. **Resolution** Our engineering team halted the queued outbound email jobs to stop additional duplicate messages from sending. Once contained, the team verified queue states and stabilized the email dispatch pipeline to ensure normal message flow resumed safely without further duplicate processing. **Root Cause** Our legacy email service went offline over the weekend, leading to a backlog of queued email jobs. When the service was restarted this morning, the backed-up jobs executed simultaneously, triggering duplicate email transmissions to affected providers on GUIDEcx 1.0. **Corrective and Preventive Actions** * Legacy email stack has been updated to alert if this issue arises again. * GUIDE 2.0 email service already contains duplicate email filtering logic. * Communicate improved stability benefits in GUIDE 2.0 to customers still using GUIDE 1.0 projects. We apologize for any confusion or inconvenience this may have caused your providers and appreciate your patience as our team worked to resolve the issue.

4. monitoring Aug 31, 2026, 05:25 PM UTC

   August 31, 2026 **Summary** On August 31, 2026, an issue occurred with our email notification queue that affected providers using GUIDEcx 1.0\. A legacy email service outage over the weekend caused outbound jobs to accumulate, resulting in duplicate email notifications being sent when the jobs were restarted. The duplicate sending was contained, and email services are operating normally. **Impact** Some providers associated with GUIDEcx 1.0 accounts received multiple duplicate email notifications during the morning hours. The incident was identified and contained earlier this morning. System monitoring confirms that no additional duplicate emails are being generated or sent. **Resolution** Our engineering team halted the queued outbound email jobs to stop additional duplicate messages from sending. Once contained, the team verified queue states and stabilized the email dispatch pipeline to ensure normal message flow resumed safely without further duplicate processing. **Root Cause** Our legacy email service went offline over the weekend, leading to a backlog of queued email jobs. When the service was restarted this morning, the backed-up jobs executed simultaneously, triggering duplicate email transmissions to affected providers on GUIDEcx 1.0. **Corrective and Preventive Actions** * Legacy email stack has been updated to alert if this issue arises again. * GUIDE 2.0 email service already contains duplicate email filtering logic. * Communicate improved stability benefits in GUIDE 2.0 to customers still using GUIDE 1.0 projects. We apologize for any confusion or inconvenience this may have caused your providers and appreciate your patience as our team worked to resolve the issue..

Read the full incident report →

Started Aug 12, 2026, 08:11 PM UTC· resolved Aug 12, 2026, 09:53 PM UTC

1. investigating Aug 12, 2026, 08:11 PM UTC

   We are currently investigating this incident..

2. investigating Aug 12, 2026, 08:11 PM UTC

   We are currently investigating this incident.

3. identified Aug 12, 2026, 08:24 PM UTC

   We have identified the issue and the remediation is in progress. .

4. identified Aug 12, 2026, 08:24 PM UTC

   We have identified the issue and the remediation is in progress.

5. identified Aug 12, 2026, 08:42 PM UTC

   We are continuing to work on a fix for this incident..

6. identified Aug 12, 2026, 08:42 PM UTC

   We are continuing to work on a fix for this incident.

7. identified Aug 12, 2026, 09:05 PM UTC

   While we aren't fully resolved, many users are now able to navigate the app. .

8. identified Aug 12, 2026, 09:05 PM UTC

   While we aren't fully resolved, many users are now able to navigate the app.

9. monitoring Aug 12, 2026, 09:31 PM UTC

   We implemented a fix and are currently monitoring the result..

10. monitoring Aug 12, 2026, 09:31 PM UTC

    We implemented a fix and are currently monitoring the result.

11. resolved Aug 12, 2026, 09:53 PM UTC

    This incident has been resolved..

12. resolved Aug 12, 2026, 09:53 PM UTC

    This incident has been resolved.

13. monitoring Aug 13, 2026, 05:46 PM UTC

    Post-Mortem: Production Availability Incident August 12, 2026 Summary On August 12, 2026, a production deployment caused an availability issue that affected portions of the application. A deployment configuration problem caused several services to become unavailable. The issue was corrected and services were restored progressively while system health was monitored. Impact Some customers experienced errors or were unable to load parts of the application. The incident lasted approximately 1 hour and 46 minutes, from 2:07 PM to 3:53 PM MDT. Service began recovering after approximately 50 minutes, with additional functionality returning progressively until full resolution. Resolution Our engineering team identified the deployment configuration issue and updated the production deployment to use the correct configuration and application versions. Services were restored gradually while we monitored application health to ensure the changes were stable before returning to full traffic. Root Cause A configuration difference between our deployment environments caused production services to receive an incorrect scaling configuration during a deployment. This resulted in several services becoming unavailable. The configuration worked correctly in our staging environment, which made the issue less apparent before the production deployment. Corrective and Preventive Actions • Standardize deployment configuration between staging and production. • Add automated checks to prevent unexpected service scaling during deployments. • Validate production deployment configuration before changes are released.. • Continue using progressive deployments and health monitoring to reduce risk during releases. We apologize for the disruption and appreciate your patience while our team worked to restore service..

14. investigating Aug 13, 2026, 05:46 PM UTC

    Post-Mortem: Production Availability Incident August 12, 2026 Summary On August 12, 2026, a production deployment caused an availability issue that affected portions of the application. A deployment configuration problem caused several services to become unavailable. The issue was corrected and services were restored progressively while system health was monitored. Impact Some customers experienced errors or were unable to load parts of the application. The incident lasted approximately 1 hour and 46 minutes, from 2:07 PM to 3:53 PM MDT. Service began recovering after approximately 50 minutes, with additional functionality returning progressively until full resolution. Resolution Our engineering team identified the deployment configuration issue and updated the production deployment to use the correct configuration and application versions. Services were restored gradually while we monitored application health to ensure the changes were stable before returning to full traffic. Root Cause A configuration difference between our deployment environments caused production services to receive an incorrect scaling configuration during a deployment. This resulted in several services becoming unavailable. The configuration worked correctly in our staging environment, which made the issue less apparent before the production deployment. Corrective and Preventive Actions • Standardize deployment configuration between staging and production. • Add automated checks to prevent unexpected service scaling during deployments. • Validate production deployment configuration before changes are released.. • Continue using progressive deployments and health monitoring to reduce risk during releases. We apologize for the disruption and appreciate your patience while our team worked to restore service.

Read the full incident report →

Started Jul 30, 2026, 10:28 PM UTC· resolved Jul 30, 2026, 11:26 PM UTC

1. investigating Jul 30, 2026, 10:28 PM UTC

   We are currently investigating this incident..

2. investigating Jul 30, 2026, 10:28 PM UTC

   We are currently investigating this incident.

3. monitoring Jul 30, 2026, 10:54 PM UTC

   Latency has improved and most users able to navigate the app without issues. We are continuing to work on identifying the root cause. .

4. investigating Jul 30, 2026, 10:54 PM UTC

   Latency has improved and most users able to navigate the app without issues. We are continuing to work on identifying the root cause.

5. resolved Jul 30, 2026, 11:26 PM UTC

   This incident has been resolved..

6. resolved Jul 30, 2026, 11:26 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 26, 2026, 01:00 AM UTC· resolved Mar 26, 2026, 01:47 AM UTC

1. scheduled Mar 26, 2026, 01:00 AM UTC

   We are planning for a scheduled maintenance during that time. We don't anticipate having any down time within the application, but wanted everyone to be made aware. .

2. in progress Mar 26, 2026, 01:00 AM UTC

   Maintenance is now in progress.

3. completed Mar 26, 2026, 01:47 AM UTC

   Maintenance has completed successfully. .

Read the full incident report →

Started Jan 30, 2026, 05:44 PM UTC· resolved Jan 30, 2026, 06:06 PM UTC

1. investigating Jan 30, 2026, 05:44 PM UTC

   We are currently investigating reports of timeout issues for SAML SSO impacting some customers.

2. monitoring Jan 30, 2026, 06:00 PM UTC

   We implemented a fix and are currently monitoring the result..

3. resolved Jan 30, 2026, 06:06 PM UTC

   This incident has been resolved..

4. investigating Jan 30, 2026, 06:28 PM UTC

   This morning, between approximately 10:00 and 10:45 MST, a small number of users experienced intermittent issues signing in using SAML SSO. Because the impact was limited and overall error rates remained within SLA thresholds, automated monitoring did not alert the operations team. After receiving customer reports, our team promptly investigated, identified a configuration issue affecting one of the services supporting SAML authentication, and resolved it. Full SAML SSO functionality was restored by 10:45 MST. We apologize for the inconvenience to the affected users..

Read the full incident report →

Started Jan 27, 2026, 05:05 PM UTC· resolved Jan 27, 2026, 05:55 PM UTC

1. investigating Jan 27, 2026, 05:05 PM UTC

   We’re aware of an issue affecting the legacy (1.0) platform and are actively investigating. The team is fully engaged and working to resolve this as quickly as possible. We’ll share updates as they become available..

2. monitoring Jan 27, 2026, 05:15 PM UTC

   We have identified the issue and it has been resolved. We'll continue to monitor the issue..

3. resolved Jan 27, 2026, 05:55 PM UTC

   **RCA – Root Cause Analysis: Latency Issues with 1.0 Platform** **Status: Resolved** **Date:** January 27, 2026 **Incident Window:** 9:06 AM – 10:15 AM MST **Summary** Between 9:06 AM and 10:15 AM MST, the 1.0 platform experienced elevated latency and intermittent API timeouts. The root cause was identified as storage exhaustion within the Redis cache cluster. This saturation forced the platform to fall back to primary database queries, leading to system congestion and slower response times. **Timeline** * **9:06 AM:** Incident start; Redis storage reaches capacity. * **9:26 AM:** **System alerts received by IT and Operations teams.** * **9:33 AM:** Formal incident ticket submitted and investigation intensified. * **9:50 AM:** Root cause confirmed as Redis cluster saturation. * **9:55 AM:** Cluster upgrade initiated to increase storage capacity. * **10:06 AM:** Upgrade complete; cache layer fully operational. * **10:15 AM:** **Resolution:** API latency returned to baseline. **Resolution & Prevention** To resolve the incident, the Redis cluster was vertically scaled to increase available storage. To prevent a recurrence, we are **recalibrating our monitoring and alerting thresholds.** These updates will ensure that alerts are triggered more appropriately and earlier in the utilization cycle, allowing our team to intervene before the system reaches saturation..

Read the full incident report →

Started Dec 01, 2025, 08:30 PM UTC· resolved Dec 01, 2025, 09:01 PM UTC

1. investigating Dec 01, 2025, 08:30 PM UTC

   We are currently investigating this incident..

2. monitoring Dec 01, 2025, 08:44 PM UTC

   We implemented a fix and are currently monitoring the result..

3. resolved Dec 01, 2025, 09:01 PM UTC

   This incident has been resolved. Thank you for your patience. .

4. investigating Dec 01, 2025, 10:28 PM UTC

   **Root Cause Analysis – Redis Cache Storage Exhaustion** Date of incident: 2025-12-01 Incident window: \~1:10 PM–1:45 PM MST Impact: Elevated latency and intermittent API timeouts on the 1.0 platform 1\. Summary At 1:10 PM MST, the Redis cache supporting the 1.0 platform reached 100% storage utilization, causing cache write failures and forcing the platform to fall back to slower database queries. This resulted in significant API latency increases, ultimately leading to timeouts for some users. An alert was triggered at 1:17 PM, the issue was identified as Redis storage exhaustion, and the Redis instance was resized. Normal operation resumed at 1:45 PM. 2\. Impact Affected system: 1.0 Platform APIs User impact: Some API requests experienced elevated latency A subset of users experienced full timeouts Business impact: Temporary performance degradation Increased system load during fallback operations Duration: Approximately 35 minutes 3\. Root Cause The Redis cache used to optimize 1.0 platform queries ran out of available storage. Once storage was exhausted: * Redis evictions and failures occurred * Systems reliant on cached query results began performing full database queries * Resulting load contributed to latency increases and some request timeouts _Contributing factor_: A faulty Redis storage monitoring alert failed to notify the operations team before the cache reached saturation. 4\. Timeline Time (MST) Event * 1:10 PM Redis cache reaches 100% storage capacity; platform begins falling back to database queries, increasing latency. * 1:17 PM Operations alert triggered due to API timeouts. * 1:20–1:40 PM Investigation identifies Redis storage exhaustion; Redis instance is resized. * 1:45 PM Resized Redis instance becomes fully operational; traffic and latency return to normal ranges. 5\. Resolution Redis instance was resized, increasing available storage capacity. Platform traffic and API latency returned to baseline levels after the upgrade was completed. 6\. Preventative Measures Completed Fixed the faulty Redis storage monitor, ensuring future alerts will correctly trigger before storage is fully consumed..

Read the full incident report →

Started Sep 16, 2025, 11:01 PM UTC· resolved Sep 17, 2025, 12:14 AM UTC

1. investigating Sep 16, 2025, 11:01 PM UTC

   We are currently investigating this incident. .

2. investigating Sep 16, 2025, 11:18 PM UTC

   We are currently investigating this incident. This is our top priority right now and we have our engineering team actively investigate potential solutions. A root cause has yet to be identified. .

3. investigating Sep 16, 2025, 11:34 PM UTC

   We are currently investigating this incident and login access has been restored, although it can take up to a minute or so to login in. Once a user is logged in they can navigate the app as expected and are experiencing normal navigation speeds. Users should be able to log in now, but again will experience a minor delay after entering your password. .

4. monitoring Sep 17, 2025, 12:08 AM UTC

   We implemented a fix and are currently monitoring the result. The fix has been deployed and users should be able to log in without any delay. Any users that were currently logged in during this degraded performance didn't experience any additional slowness while logged in. .

5. resolved Sep 17, 2025, 12:14 AM UTC

   This incident has been resolved. Thank you for your patience as we navigated restoring the log in flow. .

6. investigating Sep 17, 2025, 11:25 PM UTC

   ## **Post-Mortem: Service Provider (Atlas) MongoDB Connection Incident (September 16, 2025)** **Summary** On September 16, 2025, the access-audit service experienced widespread connection failures to MongoDB Atlas, causing major latency in the login flow and service disruptions. The MongoDB was taking too long to respond, resulting in timeouts for requests sent there. MongoDB was making some infrastructure changes that affected us and other clients. The issue was resolved through implementation of a workaround to handle MongoDB timeouts more gracefully. **Resolution** The issue was resolved by: Deploying a change to the access-audit service to Gracefully handle the timeouts. Incident Timeline | **Time (MDT)** | **Date** | **Status** | | -------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 4:34 PM | Sep 16 | Engineering was alerted through our automated alerts that we were seeing unexpected latency. Posts were also made to the engineering channel to alert other engineers. | | 4:47 PM | Sep 16 | Support raises alarm that they are seeing customer impact in the Product Support channel. | | 4:53 PM | Sep 16 | Engineering assembles in a war room and communicates to the org they are investigating. | | 5:01 PM | Sep 16 | The situation is deemed an incident and the status page is updated to indicate investigation is underway. | | 5:18 PM | Sep 16 | Engineering team actively investigating, root cause not yet identified | | 5:34 PM | Sep 16 | Login access restored with \~1 minute delay, users can navigate normally once logged in | | 6:08 PM | Sep 16 | Fix implemented and deployed, users should be able to log in without delay | | 6:14 PM | Sep 16 | Incident resolved, login flow restored to normal operation | **Root Causes** * Atlas experienced an issue when implementing a feature flag for serverless MongoDB databases, causing latency for us and others of their customers. **Observed Evidence**: **Contributing Factors:** * Service did not gracefully handle MongoDB connection timeouts, causing complete service failures instead of degraded operation. * Authentication endpoints were dependent on the response from audit calls, though successful completion or error did not impede user login. A fix was implemented to cease awaiting that response, thereby allowing the continued processing of login, logout, and other requests (e.g., SendEmail) irrespective of the audit's response. **Additional Notes** To prevent similar issues in the future, we will be implementing the following: * Implementing better timeout handling and retry logic for MongoDB connections * Adding in better logging to indicate connection issues. * Code adjustments to Mongo follow our existing database connection processes..

Read the full incident report →

Started Jun 22, 2025, 02:00 AM UTC· resolved Jun 22, 2025, 04:00 AM UTC

1. scheduled Jun 22, 2025, 02:00 AM UTC

   We are planning for a scheduled maintenance during that time..

2. in progress Jun 22, 2025, 02:00 AM UTC

   Maintenance is now in progress.

3. completed Jun 22, 2025, 04:00 AM UTC

   Maintenance has completed successfully.

Read the full incident report →

Started Jun 10, 2025, 08:59 PM UTC· resolved Jun 10, 2025, 09:48 PM UTC

1. investigating Jun 10, 2025, 08:59 PM UTC

   We are currently investigating this incident. Users are seeing slowness or errors when it comes to Custom Fields, Messages, and general navigation. Thank you for your patience and we will continue to update as more information becomes available. .

2. identified Jun 10, 2025, 09:13 PM UTC

   We are continuing to work on a fix for this incident. We believe we have identified the issue and are deploying a potential fix. If that works we will update to Monitoring status otherwise we will revert back to Investigating status. .

3. resolved Jun 10, 2025, 09:48 PM UTC

   The issue has been resolved and users should be able to navigate without significant lag or latency. .

4. investigating Jun 11, 2025, 03:57 PM UTC

   # **Post-Mortem: image limit (June 10th 2024)** ## **Summary** Following increased system load today, users experienced delays in messaging and email processing. Image registry rate limiting prevented services from restarting properly. This caused cascading dependent systems to fall behind in processing, resulting in delayed message and email delivery. ## **Resolution** The issues were resolved by taking two key actions: * Monitoring existing services - Monitored existing services to make sure that they were running as performant as possible while investigating the underlying image pull failures. * Image registry migration - Converted affected services to pull images from a more reliable image registry location that doesn't have the same rate limiting constraints. ## **Incident Timeline** | Time | Event | | ------- | -------------------------------------------------------------------- | | 1:15 PM | First error detected | | 1:17 PM | Engineering team notified of the incident | | 1:34 PM | Root cause identified as image registry rate limiting | | 2:50 PM | Fix deployed to production (services migrated to new image registry) | | 3:46 PM | All issues resolved, system returned to normal operation | ## **Root Causes** * Image Registry Rate Limiting: The primary image registry had restrictive rate limits that prevented services from pulling server images during restart events, blocking the successful restart of services that were failing health checks. * Service Health Check Failures: services were restarting due to health check failures, and the subsequent server image pull failures during restart created a cascading effect that impacted message and email processing across the platform. ## **Additional Notes** To prevent similar issues in the future, the following measures will be implemented: * Image pre-caching - Implement image pre-caching on nodes to reduce dependency on external registry pulls during scaling events. * Enhanced monitoring - Add specific monitoring for image pull failures and registry rate limit warnings. * Root Cause Analysis: Examination into the persistent health check failures experienced by the service..

Read the full incident report →

Started May 28, 2025, 03:54 PM UTC· resolved May 28, 2025, 06:50 PM UTC

1. investigating May 28, 2025, 03:54 PM UTC

   We are currently investigating this incident. We have already boosted resources to reduce lag, but still haven't found the root issue. We have a couple of leads we are still investigating. .

2. investigating May 28, 2025, 04:24 PM UTC

   We are currently investigating this incident. Again, we have a few ideas about a potential root cause, but haven't identified the culprit yet. .

3. investigating May 28, 2025, 05:16 PM UTC

   Our team is actively investigating the issue impacting performance. We've identified a likely root cause and are preparing a fix that will be deployed shortly. We’re closely monitoring the situation and will provide updates as soon as more information becomes available. Thank you for your patience as we work to resolve this quickly and thoroughly..

4. monitoring May 28, 2025, 06:04 PM UTC

   We implemented a fix and are currently monitoring the result. Typical navigation speed, project creation, task creation, etc. are all seeing improved speed. .

5. resolved May 28, 2025, 06:50 PM UTC

   This incident has been resolved. Thank you for your patience. We will be reviewing what caused the issue and share with you our learnings and the steps we have taken to help prevent future disruptions. Please reach out to Support if you experience any future latency. .

6. investigating May 28, 2025, 08:18 PM UTC

   **Post-Mortem: Messaging Service Incident (May 28, 2024)** # **Summary** Following the release of a new messaging feature early on May 28, 2024, users experienced general slowness in messaging related requests. The new messaging feature increased the number of requests hitting our gateway, triggering rate limit issues that hadn't been encountered before. Additionally, messaging failed to load in the task drawer due to invalid feature flag configurations related to customers, and channel loading was slow due to an expensive database query. # **Resolution** The issues were resolved by taking three key actions: 1. Increasing resources on affected services 2. Temporarily scaling gateway services to handle the increased request volume while implementing a permanent fix that adjusted rate limits to more reasonable values 3. Optimizing the expensive database query to retrieve only necessary data, reducing contention on the database # **Incident Timeline** | **Time** | **Event** | | -------------------- | -------------------------------------------------------- | | 2:30 AM MDT, May 28 | New messaging feature released | | 8:12 AM MDT, May 28 | Customer reports of issues received | | 8:30 AM MDT, May 28 | Resources adjusted on affected services | | 9:00 AM MDT, May 28 | War Room initiated for coordinated response | | 11:33 AM MDT, May 28 | Optimized query deployed | | 11:35 AM MDT, May 28 | Adjusted the feature flag configuration | | 12:34 PM MDT, May 28 | Adjusted rate limit configuration deployed | | 12:50 PM MDT, May 28 | All issues resolved, system returned to normal operation | # **Root Causes** * Resource Constraints: Insufficient resources allocated to select services to handle the increased load from the new messaging feature * Gateway Rate Limiting: Rate limits on gateway services were too restrictive, causing legitimate requests to be denied when traffic increased * Inefficient Database Queries: Certain queries were retrieving excessive data, causing database contention and slowing down channel loading # **Additional Notes** To prevent similar issues in the future, we will be looking into the following: * Proactive monitoring for rate limit thresholds, especially during new feature releases * Load testing with realistic traffic patterns prior to major feature releases * Database query optimization reviews as part of the deployment checklist * Automated scaling policies for critical gateway servicesEnhanced existing * monitoring to include coverage for areas primarily affected by the incident, such as messaging request latency and error rates..

Read the full incident report →

Started Apr 30, 2025, 08:55 PM UTC· resolved Apr 30, 2025, 09:32 PM UTC

1. investigating Apr 30, 2025, 08:55 PM UTC

   We are currently investigating this incident..

2. identified Apr 30, 2025, 08:56 PM UTC

   The root cause has been identified. An index is being applied that will help. .

3. monitoring Apr 30, 2025, 09:03 PM UTC

   The improvements have been applied and we are monitoring the impact to ensure all logins are being processed quickly. .

4. resolved Apr 30, 2025, 09:32 PM UTC

   The fix was successful. All logins are working well. .

Read the full incident report →

Started Jan 13, 2025, 10:45 PM UTC· resolved Jan 13, 2025, 10:53 PM UTC

1. identified Jan 13, 2025, 10:45 PM UTC

   We are continuing to work on a fix for this incident. We deployed an update and are seeing some odd behavior in some of our labels. Users should still be able to login and navigate, but may have some weirdness in the UI until some additional deployments finish. .

2. monitoring Jan 13, 2025, 10:49 PM UTC

   We implemented a fix and are currently monitoring the result..

3. resolved Jan 13, 2025, 10:53 PM UTC

   This incident has been resolved..

Read the full incident report →

Started Jan 13, 2025, 05:10 PM UTC· resolved Jan 13, 2025, 05:17 PM UTC

1. investigating Jan 13, 2025, 05:10 PM UTC

   We are currently investigating this incident. The system is still available, but login credential processing is slower than usual. .

2. identified Jan 13, 2025, 05:12 PM UTC

   The root cause has been diagnosed. The access service is being updated. This is expected to be resolved very soon..

3. monitoring Jan 13, 2025, 05:16 PM UTC

   The access service has been restarted and the pending backlog of requests are being processed. Authentication performance will be monitored for the next few minutes until login credentials are processed within 3-5 seconds as usual. .

4. resolved Jan 13, 2025, 05:17 PM UTC

   This issue has been fully resolved. Login should be fast and consistent now for all organizations..

Read the full incident report →

Started Nov 20, 2024, 04:13 PM UTC· resolved Nov 20, 2024, 06:54 PM UTC

1. investigating Nov 20, 2024, 04:13 PM UTC

   We are currently investigating this incident..

2. investigating Nov 20, 2024, 04:44 PM UTC

   We are still investigating this incident. We have found some insights from our monitoring tools and are chasing down a few theories. We also understand that some issues are leaking into the UI experience and that it isn't just related to work done via the API, including issues with custom fields, new Team page, etc. .

3. identified Nov 20, 2024, 06:08 PM UTC

   The team has identified a fix and has started working on implementing a fix. We will continue to update as progress is made. .

4. resolved Nov 20, 2024, 06:54 PM UTC

   This incident has been resolved. Our team will continue to monitor the situation but users are no longer receiving errors within the API and the Team page and Custom Fields are loading properly. .

5. resolved Nov 21, 2024, 12:26 AM UTC

   ### **RCA** **Summary:** The latest GUIDEcx API services run on AWS EKS Clusters. On the evening of November 19, 2024, a scheduled deployment introduced changes to a particular service related to new project and task-related features. This service also manages custom fields, certain application pages, and interfaces with Open API requests concerning custom fields. A specific database query, in certain instances, returned more data than necessary. Although these queries executed within acceptable time limits, the excessive amount of data returned created a bottleneck for other queries requesting similar data. While it can sometimes be acceptable for queries to return significant amounts of data, this query was intended to return smaller amounts, as it was being run frequently across multiple distributed services. This led to a backlog of requests, which in turn created pressure on messages, events, and other services reliant on the same data. This pressure caused the slowness and inconsistencies experienced during the incident. --- **Resolution:** The issue was resolved by identifying the cases where more data was being requested than intended and modifying those queries to only return the necessary data. Once this adjustment was made, queries executed efficiently, and the backlog of messages, events, and services was processed, allowing the system to return to normal operations. --- **Incident Timeline (in EST):** * **6:30 PM, November 19**: Code released for feature updates. Services appeared healthy and scaled appropriately. * **6:48 PM, November 19**: Automated monitoring detects minor instability, but the incident resolves itself before a support representative can investigate further. * **8:00 AM, November 20**: Engineering support discovers errors related to requests failing to connect to services. A backlog of requests causes some services to restart, resulting in intermittent failures. * **10:52 AM, November 20**: Support receives reports of API errors (later identified as timeouts), incomplete page loads, and missing custom fields on projects. Open API traffic spikes during this time (an expected occurrence, but it increases the frequency of the problematic query execution). * **12:24 PM, November 20**: The root cause is identified, and a resolution is implemented. * **1:54 PM, November 20**: The incident is fully resolved. System back pressure is processed successfully, and operations return to normal. --- **Additional Notes:** Three specific actions are being implemented to mitigate similar issues in the future: 1. Automated alerts will flag queries that return larger-than-expected payloads to clients. Use cases will define reasonable maximum thresholds for data loads. 2. Automated alerts will detect system backlogs related to messaging. Use cases will define reasonable maximum thresholds for such backlogs. 3. APIs associated with AWS EKS services will require filterable criteria for database queries unless explicitly defined otherwise..

Read the full incident report →