Entitle Outage History

Started Jul 05, 2026, 05:00 PM UTC· resolved Jul 06, 2026, 04:25 PM UTC

1. investigating Jul 05, 2026, 05:00 PM UTC

   Agents that are newly installed, upgraded, or restarted during this window may fail to start. The agent's container image cannot be pulled because the image-registry credentials issued with agent tokens have expired, causing image-pull (authentication) failures — the agent pod stays in a non-running state.

2. resolved Jul 06, 2026, 04:25 PM UTC

   This incident is resolved. We've re-enabled the connection to the agent image registry, so image pulls are working as before. Agents that failed to start during the incident just need a rollout — trigger a rollout (restart) of the agent and it will pull the image successfully, exactly as before. No token changes or reinstall are required. Agents that were already running were not impacted.

Read the full incident report →

Started Jun 16, 2026, 01:50 PM UTC· resolved Jun 16, 2026, 02:03 PM UTC

1. investigating Jun 16, 2026, 01:50 PM UTC

   We are currently investigating this issue.

2. resolved Jun 16, 2026, 02:03 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 16, 2026, 03:08 PM UTC· resolved Feb 16, 2026, 03:35 PM UTC

1. investigating Feb 16, 2026, 03:08 PM UTC

   We are currently investigating this issue.

2. investigating Feb 16, 2026, 03:08 PM UTC

   We are continuing to investigate this issue.

3. identified Feb 16, 2026, 03:26 PM UTC

   The issue has been traced to Cloudflare, and we’re currently working on a fix.

4. monitoring Feb 16, 2026, 03:28 PM UTC

   A fix has been implemented and we are monitoring the results.

5. monitoring Feb 16, 2026, 03:28 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Feb 16, 2026, 03:35 PM UTC

   This incident has been resolved.

7. postmortem Feb 19, 2026, 10:15 AM UTC

   **Service Incident Summary — February 16, 2026** On Monday, February 16, 2026, between approximately 16:45 and 17:30 IST, customers in EU region experienced connectivity issues to our portal due to a DNS disruption in our production environment. **What happened:** During routine infrastructure work to set up a new environment, a configuration file was not properly updated, causing it to inadvertently remove DNS records managed through Cloudflare in our EU region. This disrupted routing for services relying on those DNS records. **Impact:** Customers in the EU region have experienced service unavailability during this window. No data was lost, and no security breach occurred. Other regions were not affected. **How we resolved it:** Our team identified the misconfiguration within approximately 30 minutes of the initial alert, corrected it, and fully restored all affected DNS records shortly after. **What we're doing to prevent recurrence:** We've already implemented environment-specific Cloudflare API tokens to limit the blast radius of any future misconfiguration, improved our alerting, and are strengthening our code review process for infrastructure changes. We're also creating better internal documentation to guide environment setup going forward. We apologize for any disruption this caused. If you have questions or are experiencing any related issues, please reach out to our support team.

Read the full incident report →

Started Jan 22, 2026, 08:18 PM UTC· resolved Jan 22, 2026, 10:11 PM UTC

1. investigating Jan 22, 2026, 08:18 PM UTC

   We are currently investigating this issue.

2. identified Jan 22, 2026, 08:32 PM UTC

   The issue has been identified in one of our backend services ( pub sub module) and a fix is being implemented.

3. monitoring Jan 22, 2026, 08:32 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jan 22, 2026, 10:11 PM UTC

   This incident has been resolved.

5. postmortem Jan 25, 2026, 10:55 AM UTC

   # 22 January 2026 - Post-Incident Report: Delayed Permission Provisioning in US Region **Incident Date:** **22 January 2026** **Affected Region:** US **Status:** Resolved **Timezone:** UTC ## Summary On 22 January 2026, around 18:00 UTC some customers experienced delays in permission provisioning. At 20:00 UTC we validated the the issue is related to US region only. The underlying issue was resolved at approximately 20:30 UTC, restoring normal service. The incident remained open until approximately 22:00 UTC to allow time to validate recovery and collect confirmation from all affected customers. ## Impact * A limited number of customers experienced delays in permission grants * Permission requests were processed later than expected * No data loss occurred * No incorrect permissions were granted * No security impact was identified ## Root Cause The issue was caused by a temporary backlog in background job processing in the US region, which delayed execution of permission requests. ## Resolution Our engineering teams mitigated the issue by: * Releasing delayed background jobs * Scaling processing capacity to clear the backlog * Verifying successful permission grants Service functionality was restored around 20:30 UTC. ## Validation & Closure Following service restoration, we: * Actively monitored the system for stability * Confirmed successful processing with affected customers After completing validation with customers impacted by this issue, the incident was formally closed around **22:00 UTC**. ## Prevention To reduce the likelihood of similar incidents, we have: * Improved monitoring and alerting for delayed background processing * Enhanced visibility into regional job execution health * Updated system configurations to prevent queue backlogs ## Closing We apologize for the inconvenience and appreciate your patience. If you have any further questions or experience related issues, please contact BeyondTrust support team by [https://www.beyondtrust.com/customer-support-guide](https://www.beyondtrust.com/customer-support-guide).

Read the full incident report →

Started Dec 11, 2025, 11:30 PM UTC· resolved Dec 11, 2025, 11:30 PM UTC

1. resolved Dec 11, 2025, 11:30 PM UTC

   During routine work on expanding our deployment automation for new regions, an update script was executed in a way that unintentionally impacted an existing production region (EU). The script modified a secret metadata used by our slack service in that region, which caused downstream components to fail validation and resulted in service disruption.

Read the full incident report →

Started Nov 18, 2025, 11:50 AM UTC· resolved Nov 18, 2025, 04:29 PM UTC

1. identified Nov 18, 2025, 11:50 AM UTC

   Entitle had degraded performance due to Cloudflare maintenance.

2. identified Nov 18, 2025, 12:15 PM UTC

   We are continuing to work on a fix for this issue.

3. resolved Nov 18, 2025, 04:29 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 02:35 PM UTC· resolved Oct 20, 2025, 05:39 PM UTC

1. monitoring Oct 20, 2025, 02:35 PM UTC

   A fix has been implemented and we are monitoring the results.

2. resolved Oct 20, 2025, 05:39 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 06:45 AM UTC· resolved Oct 20, 2025, 11:00 AM UTC

1. monitoring Oct 20, 2025, 06:45 AM UTC

   We are currently experiencing a service disruption due to an ongoing outage in the AWS us-east-1 region. Impact: Access Change Requests Scope: Other regions and environments remain operational. Next Steps: Our engineering team is monitoring the AWS status page and will provide updates as soon as AWS confirms recovery or mitigation steps.

2. monitoring Oct 20, 2025, 10:48 AM UTC

   The platform is starting to recover, we will update again soon

3. resolved Oct 20, 2025, 11:00 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 11, 2025, 04:35 PM UTC· resolved Sep 11, 2025, 05:15 PM UTC

1. investigating Sep 11, 2025, 04:35 PM UTC

   We are currently investigating this issue.

2. resolved Sep 11, 2025, 05:15 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 25, 2025, 08:32 AM UTC· resolved Aug 25, 2025, 10:55 AM UTC

1. investigating Aug 25, 2025, 08:32 AM UTC

   We are currently investigating this issue.

2. identified Aug 25, 2025, 09:18 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 25, 2025, 09:55 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 25, 2025, 10:55 AM UTC

   This incident has been resolved.

5. postmortem Aug 28, 2025, 07:26 PM UTC

   # Service Degradation and Partial Downtime on August 25, 2025 ## Incident Overview On **August 25, 2025**, Entitle services experienced a major service disruption between **02:00 and 15:00 UTC**. During this period, customers encountered **JIT access requests processing failures** and **reduced web app frontend availability**. **Impact:** All customers were affected.` `**Duration:** ~14 hours of instability. Root Cause The disruption resulted from a **combination of two factors**: 1. **A new version of queue mechanism** introduced in the latest release was not fully resilient under failure conditions. 2. **Database resources utilization and configuration issues** led to instability, and restarting the system. Together, these factors created cascading failures that degraded core services. ## Response * Entitle engineering team mobilized and worked continuously to investigate and mitigate. * Temporary fixes improved stability, but a **full rollback to the prior version at 15:00 UTC** was ultimately required to restore full-service SLA. * Customers were kept informed through status page updates. ## Business Impact * Service availability was significantly reduced during the incident. * JIT access requests processing delays impacted customers operations. ## Preventive Actions We are implementing the following improvements to prevent recurrence: * **Safer Releases:** Automated rollback validation in every release cycle. * **Better Monitoring:** improving real-time alerts for JIT access requests backlogs and service degradation. * **Operational Resilience:** Expanded tests for third-party dependencies ## Closing Note We sincerely apologize for this disruption. We recognize the trust you place in us and are committed to learning from this incident. The actions outlined above are already in progress to ensure stronger reliability and resilience moving forward.

Read the full incident report →

Started Jul 15, 2025, 09:06 AM UTC· resolved Jul 15, 2025, 03:02 PM UTC

1. investigating Jul 15, 2025, 09:06 AM UTC

   Original Slack has issues : https://slack-status.com/2025-07/1ce16e396a56dcf8 So slack integration may not work

2. resolved Jul 15, 2025, 03:02 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 16, 2025, 06:48 PM UTC· resolved Jun 16, 2025, 07:00 PM UTC

1. investigating Jun 16, 2025, 06:48 PM UTC

   We are currently investigating this issue.

2. resolved Jun 16, 2025, 07:00 PM UTC

   This incident has been resolved.

3. postmortem Jun 16, 2025, 07:56 PM UTC

   ## Postmortem: Redis Outage — Entitle US Region **Date:** 2025-06-16 **Service Affected:** Entitle – US Region **Root Cause:** Redis pods CrashLoopBackOff ### Summary On **June 16, 2025**, the Entitle US region experienced a **partial service outage** due to a failure in the **Redis** backend service. The Redis pod entered a `CrashLoopBackOff` state, failing to restart successfully. This caused cascading issues in dependent services, including elevated Redis client errors, Pub/Sub disconnections, and connection retry exhaustion. The issue was resolved by performing a **manual hard reset** of the Redis pod. The Redis container has **crashed unexpectedly**, and Kubernetes was **unable to successfully restart it** due to repeated startup failures. This caused a `BackOff` state, where Kubernetes delays further restarts. The downstream Node.js services attempted to reconnect continuously, leading to: * **Redis connection timeouts** * **Pub/Sub disconnections \(**`EOF`\) * **Memory pressure due to too many event listeners** This condition persisted until the pod was **forcefully deleted**, which reset the backoff and allowed Redis to start cleanly. Action Items : A **new monitor was created** to track `CrashLoopBackOff` and `BackOff` events in critical infrastructure pods. This will allow us to detect and respond to container restart failures earlier — potentially preventing downtime through intervention.

Read the full incident report →

Started Jun 03, 2025, 01:37 PM UTC· resolved Jun 03, 2025, 01:55 PM UTC

1. investigating Jun 03, 2025, 01:37 PM UTC

   We are currently investigating this issue.

2. resolved Jun 03, 2025, 01:55 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 13, 2025, 01:54 PM UTC· resolved May 13, 2025, 02:40 PM UTC

1. investigating May 13, 2025, 01:54 PM UTC

   We are currently investigating this issue.

2. resolved May 13, 2025, 02:40 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 10, 2025, 03:09 PM UTC· resolved Apr 10, 2025, 03:09 PM UTC

1. resolved Apr 10, 2025, 03:09 PM UTC

   This incident has been resolved.

2. postmortem Apr 10, 2025, 03:09 PM UTC

   ### **Postmortem: Elastic Reindex Failure Impacting Sourcegraph Requests** **Date of Incident:** April 9, 2025 **Reported by:** Sourcegraph Integration **Impact Duration:** Until reindex was manually retriggered **Status:** Resolved ### **Summary** On April 9, 2025, a user from Sourcegraph encountered an error while attempting to create a new request. This was due to a failure in our daily Elasticsearch reindex process, which is responsible for refreshing role-based access data across integrations and bundles. ### **Root Cause** The reindex process failed due to an unusually large volume of roles on that day. This overwhelmed Elasticsearch, leading to internal failures and restarts. As a result: * Elasticsearch temporarily blocked access to the affected index. * Sourcegraph, which relies on this index to retrieve asset information and validate access, received errors when making requests. ### **Resolution** The issue was resolved by manually re-running the reindex operation, which successfully completed and restored access. ### **Impact** * Sourcegraph users were unable to create new requests during the failure window. * System stability and access to critical index data were temporarily compromised. ### **Action Items** 1. **Improve Monitoring on Elasticsearch Reindexing** * Add alerts for failed or unusually long reindexing jobs. 2. **Optimize the Reindex Process** * Investigate batching or parallelization strategies to better handle high-volume role data. 3. **Enhance Elasticsearch Stability and Fault Tolerance** *

Read the full incident report →

Started Mar 26, 2025, 03:01 PM UTC· resolved Mar 26, 2025, 03:03 PM UTC

1. investigating Mar 26, 2025, 03:01 PM UTC

   Application Not working

2. resolved Mar 26, 2025, 03:03 PM UTC

   This incident has been resolved.

3. postmortem Mar 27, 2025, 10:27 AM UTC

   **Post-Mortem Report: System Downtime on 03/27/2025** **Incident Summary:** During the deployment of a hotfix to the production environment, the system experienced unexpected downtime due to database locks. The downtime lasted for approximately 10 minutes, affecting all users attempting to access Entitle. **Root Cause Analysis:** The deployment process triggered database migrations, and during execution, certain queries caused locks on the database. These locks resulted in database transactions being blocked, leading to system unresponsiveness and eventual downtime. **Impact:** * Users were unable to access the system during the downtime window. **Resolution:** * The locks were cleared, and the system was restored from downtime. **Action Items:** 1. **Enforce a Deployment Check:** Ensure that deployments proceed only after all database migrations have been successfully executed to prevent locking issues. 2. **Implement a New Deployment Strategy:** Build and use an improved release deployment strategy to minimize the risk of downtime. 3. **Enhance Monitoring & Alerts:** Set up real-time monitoring and alerting mechanisms to detect database locks early and act proactively. 4. **Improve Rollback Mechanism:** Establish a seamless rollback mechanism to restore the previous stable state quickly in case of deployment issues. By implementing these action items, we aim to enhance deployment reliability and prevent similar incidents in the future.

Read the full incident report →

Started Mar 24, 2025, 09:12 PM UTC· resolved Mar 24, 2025, 09:12 PM UTC

1. resolved Mar 24, 2025, 09:12 PM UTC

   Some customers may experience delays in the ticket granting mechanism.

2. postmortem Mar 26, 2025, 02:50 PM UTC

   **Incident Postmortem: Ticket Processing Partial Downtime** **Incident Date:** Monday, 3.24.2025 **Impact:** Partial downtime affecting ticket processing**Summary** On Monday, we experienced a partial downtime where submitted tickets became stuck in processing. This issue was identified after noticing a backlog in the ticket queue. To resolve the problem, we rescaled our job handler to handle more parallel jobs, which successfully cleared the backlog and restored normal processing.**Root Cause** The issue was caused by a large number of requests overwhelming the job handler, which was unable to handle the increased load. Additionally, the jobs were in the same queue along with other missions, which prevented them from executing in a timely manner. This led to a buildup in the queue, delaying ticket processing.**Resolution** Once the issue was identified, we rescaled our job handler to allow for more parallel job executions. This enabled faster processing of the stuck jobs and resolved the queue backlog.**Action Items** 1. **Separate the ticket processing job into a new queue** - Done * This prevents ticket processing from being impacted by other jobs in the future. 2. **Extend the lock duration for the ‘give access’ job** - Done * Ensures better handling of multiple jobs running in parallel without conflicts. 3. **Create specific monitoring for the new ticket queue** - Done * Improves visibility and allows for proactive detection of similar issues in the future. **Lessons Learned** * Monitoring gaps delayed detection of the issue. * Job queue separation is critical for isolating failures and ensuring reliability. * Scaling parallel processing dynamically can be an effective quick-fix but needs to be complemented with structural changes. **Next Steps** * Continuously monitor the new ticket queue to ensure stability. * Evaluate job locking mechanisms for other critical processes. * Implement automated alerts for job queue delays.

Read the full incident report →

Started Mar 20, 2025, 06:00 PM UTC· resolved Mar 20, 2025, 06:00 PM UTC

1. resolved Mar 20, 2025, 06:00 PM UTC

   Between March 20–21, certain customer environments experienced a temporary degradation in sync reporting, where data generated by agents was not fully ingested by our backend systems. The issue was rooted in a recent credential rotation and was fully resolved within hours of identification. No impact was observed on access provisioning or enforcement, and no data was lost. Functionality has been fully restored.

2. postmortem Apr 06, 2025, 07:02 AM UTC

   # Sync Reporting Degradation **Date:** March 20–21, 2025 **Service Impacted:** Sync Data Reporting **Impact Duration:** Intermittent impact from March 20th to March 21st, 2005. ## Summary Between March 20–21, certain customer environments experienced a temporary degradation in sync reporting, where data generated by agents was not fully ingested by our backend systems. The issue was rooted in a recent credential rotation and was fully resolved within hours of identification. No impact was observed on access provisioning or enforcement, and no data was lost. Functionality has been fully restored. ## Background On March 15, a critical vulnerability in GitHub Actions was disclosed \(CVE-2025-30066\), highlighting a potential risk of secret leakage via CI logs. As part of our proactive security posture, Entitle initiated a comprehensive rotation of all production secrets stored in GitHub. Among the rotated credentials was a token used by Entitle’s agent software to transmit sync metadata to a backend storage system. While most GitHub secrets are associated with build and deployment pipelines, this specific token was also used in the runtime path of deployed agents — a distinction that wasn’t fully accounted for during the rotation process. ## Timeline ### March 20 The storage service token was rotated as part of a broader security effort. ### March 21 A subset of sync tasks across some customer tenants were not fully transmitted to Entitle's backend. Initial signals were not escalated immediately due to routing misconfigurations in alerting logic. Engineering was notified of anomalies and began an active investigation. An hour later, a fix was deployed, including token override distribution and a new agent image update. ## Root Cause The rotated credential was being used operationally by agents, not just by CI/CD processes. This use case was not fully scoped during the rotation, leading to unexpected communication issues between agents and backend services. Alerting for failed data transmissions was present but not configured to reach the relevant on-call personnel, delaying detection. ## Resolution * A new valid token was issued and rolled out using an agent override mechanism. * A fresh agent version containing the new token was published. * Affected tenants were monitored closely post-resolution to ensure full restoration of data flows. ## Mitigations & Next Steps ### Credential Management Enhancements * Segregate CI/CD and runtime credentials across dedicated secret stores. * Introduce tagging and classification for all credentials to indicate their operational usage. ### Monitoring and Alerting Improvements * Add high-sensitivity monitors for sync degradation and agent communication failures. * Ensure alert routing logic is codified and version-controlled \(via Terraform\). ### Operational Resilience * Enhance agent-side error handling for token-related failures. * Introduce rollback fallback logic in critical agent communication flows.

Read the full incident report →

Started Feb 26, 2025, 10:21 AM UTC· resolved Feb 26, 2025, 10:38 AM UTC

1. investigating Feb 26, 2025, 10:21 AM UTC

   We are currently investigating this issue.

2. resolved Feb 26, 2025, 10:38 AM UTC

   This incident has been resolved.

3. postmortem Mar 11, 2025, 11:08 AM UTC

   **Postmortem Report: Elastic Index Update Execution Failure When New Release Uploaded** **Date of Incident:** 2/26/25 **Summary:** When the release was uploaded, the indexing job took longer than expected to complete, preventing customers from requesting access to integrations. This delay led to inconsistencies in the system, including missing data and degraded performance. **Impact:** * Users were unable to access integrations. **Resolution:** The index job eventually completed after the release was uploaded. However, this delay is not a sustainable approach, as it can result in missing data and potential system disruptions. **Action Items:** 1. **Implement a Pre-Release Phase:** * Introduce a pre-release stage in the deployment pipeline where essential background jobs, including indexing, are executed before the full release upload. 2. **Automate Pre-Release Job Execution:** * Ensure that indexing jobs run automatically before the final release is pushed. * Implement monitoring/logging to verify job completion. 3. **Enhance Deployment Validation Checks:** * Define a checklist to confirm that all required jobs have been executed before proceeding with the release. 4. **Improve Alerts & Monitoring:** * Set up automated alerts if the indexing job is skipped or fails to execute. **Lessons Learned:** * Lack of a structured pre-release phase led to an avoidable issue. * Critical background jobs should be enforced as part of the deployment workflow. * Automation can help prevent manual errors and ensure smooth releases.

Read the full incident report →

Started Feb 26, 2025, 09:51 AM UTC· resolved Feb 26, 2025, 10:01 AM UTC

1. investigating Feb 26, 2025, 09:51 AM UTC

   We are investigating reports of degraded performance in our app. Our team is actively investigating and seeking a resolution, and we'll continue to update here.

2. resolved Feb 26, 2025, 10:01 AM UTC

   This incident has been resolved.

3. postmortem Mar 16, 2025, 10:15 AM UTC

   > **Root Cause:** > The deployment resulted in downtime due to a database migration step that took significantly longer than originally anticipated. As the deployment pipeline was configured to execute migrations synchronously prior to launching the new version, the prolonged migration caused the older version to terminate before the new version was fully ready to serve traffic, resulting in temporary unavailability. > **Resolution:** > To avoid recurrence, future deployments will separate database migrations from application releases, using incremental migration strategies or pre-migration scripts to ensure database readiness. Additionally, we've enhanced our monitoring to better estimate migration durations and introduced procedures to roll back or pause deployment if a migration exceeds expected durations. > **Follow-up Actions:** > > Implemented separate pre-deployment migration jobs with clearly defined thresholds for maximum allowable duration. Improved monitoring and alerting on migration performance and durations. Updated deployment procedures to allow manual approval for potentially long-running migrations.

Read the full incident report →

Started Dec 11, 2024, 08:07 AM UTC· resolved Dec 11, 2024, 08:11 AM UTC

1. investigating Dec 11, 2024, 08:07 AM UTC

   We are investigating reports of degraded performance in our app. Our team is actively investigating and seeking a resolution, and we'll continue to update here.

2. resolved Dec 11, 2024, 08:11 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 09, 2024, 09:25 AM UTC· resolved Dec 09, 2024, 09:46 AM UTC

1. investigating Dec 09, 2024, 09:25 AM UTC

   We are investigating reports of degraded performance in our app. Our team is actively investigating and seeking a resolution for the issue, and we'll continue to update here.

2. monitoring Dec 09, 2024, 09:38 AM UTC

   We have rolled out a fix and the issue should be resolved for users. We appreciate your patience while we sorted this out.

3. resolved Dec 09, 2024, 09:46 AM UTC

   This incident has been resolved.

Read the full incident report →