Is Element451 down?

Status: Identified We have identified the root cause of the issue affecting some managed integration. A code deployment at 3:38pm EDT caused some managed integration to fail. Affected components Integrations (Partial outage)

Status: Resolved This incident has been resolved. We've rolled back the code change that caused some managed integrations to fail. Our integrations team is working to resync any failed data. Customers need not take any action at this time. If you continue to experience any problems, please contact our support team. Thank you for your patience while we worked to resolve this issue. Affected components Integrations (Operational)

Status: Monitoring A fix has been implemented for the issue affecting access to the Element451 dashboard. Which has now stabilized. An issue at AWS caused a 4 minute outage for new sessions to load admin/internal user access to the Element451 console. AWS was able to heal the issue quickly. Existing sessions were unaffected as was access to other public facing sites like Events, Pages, Appt Booking and Apps. We will provide a final update once we are confident the issue is fully resolved. Affected components Element451.io (internal user access) (Full outage)

Status: Resolved This incident has been resolved. The issue affecting internal user/admin access to the Element451 console has been fully addressed, and all systems are operating normally. If you continue to experience any problems, please contact our support team. Thank you for your patience while we worked to resolve this issue. Affected components Element451.io (internal user access) (Operational)

Status: Resolved While this incident is resolved and we are fully operational, we wanted to share some additional details. At approximately 2:21pm EDT, we detected a spike in traffic to element451.io: http://element451.io (our internal admin console) of roughly 14x normal levels. This overwhelmed the ECS tasks powering the console, causing them to fail. While AWS typically auto-recovers this type of failure within under 60 seconds, it took approximately 4 minutes this time for ECS to replace the failed tasks and restore access. Existing user sessions were not affected, and access to our other public-facing services (Events, Pages, Appointment Booking, and Apps) remained fully available throughout. We're continuing to investigate the root cause of the traffic spike. In the meantime, we've increased our baseline task count and memory allocation for both the app and proxy layers to add headroom and reduce the chance of a recurrence. Affected components Element451.io (internal user access) (Operational)

Status: Investigating We are currently investigating an issue affecting Bolt Discovery Domain access. Dedicated URLs for Bolt Discovery (discoveredu.ai) are currently not reachable. Our team has identified the issue and is actively working to determine the root cause. We will provide an update as soon as we have more information. Affected components Bolt Discovery (Full outage)

Status: Resolved This incident has been resolved. The issue affecting Bolt Discovery subdomains hosted at discoveryedu.ai has been fully addressed, and all systems are operating normally. Some users may continue to experience temporary loading failures due to locally cached data. These issues should clear as caches refresh and do not indicate an ongoing service disruption. Discovery embeds and Bolt Discovery subdomains hosted at bolt-discover.ai remained available and were not impacted by this incident. Thank you for your patience while we worked to resolve this issue. Affected components Bolt Discovery (Operational)

Status: Investigating We are currently investigating an issue affecting our database. Users may experience slow load times or failed page loads. Our team has identified the issue and is actively working to determine the root cause. We will provide an update as soon as we have more information. Affected components Comet Graphql API (Degraded performance) Element451 API (Degraded performance) Conversations API (Degraded performance)

Status: Identified We have identified the root cause of the issue affecting our servers. The issue is caused by an incorrect process change that writes the database on reads, inflating our worker pools. Our team is implementing a fix and working to restore full functionality as quickly as possible. We will continue to share updates on our progress. Affected components Comet Graphql API (Degraded performance) Element451 API (Degraded performance) Conversations API (Degraded performance)

Status: Resolved This incident has been resolved. The issue affecting web servers has been fully addressed, and all systems are operating normally. If you continue to experience any problems, please contact our support team. Thank you for your patience while we worked to resolve this issue. Affected components Element451 API (Operational) Conversations API (Operational) Comet Graphql API (Operational)

Status: Resolved Due to the performance degradation, managed integrations were also affected. Runs importing data into Element sporadically experienced timeout errors, including Ethos-to-Element and CFNC-to-Element integrations. No action is needed as the Element451 Integrations team will be rerunning failed runs for these managed integrations. Affected components Comet Graphql API (Operational) Element451 API (Operational) Conversations API (Operational)

Status: Identified We've identified some lingering issues from the earlier database performance issue, specifically as it related to our Graphql API. Specially when new items are created (like new knowledge sources or import tasks) they may not immediately show in the UI despite being created. We are currently working to relieve this issue. Affected components Element451 API (Operational) Conversations API (Operational) Comet Graphql API (Degraded performance)

Status: Resolved This incident has been resolved. The issue affecting front end display updates has been fully addressed, and all systems are operating normally. If you continue to experience any problems, please contact our support team. Thank you for your patience while we worked to resolve this issue. Affected components Element451 API (Operational) Conversations API (Operational) Comet Graphql API (Operational)

Status: Investigating We are currently investigating an issue affecting application sites. Users may experience applications loading after login. Our team has identified the issue and is actively working to determine the root cause. We will provide an update as soon as we have more information. Affected components Element451 API (Partial outage)

Status: Monitoring We restored the service by rolling back to an older version. In the mean time a fix to the root cause is under testing. We are currently monitoring the system to ensure stability and confirm that services are fully restored. Users should see normal performance at this time. We will provide a final update once we are confident the issue is fully resolved. Affected components Element451 API (Partial outage)

Status: Monitoring A fix has been implemented for the issue affecting application loading in some sites with conditional logic. We are currently monitoring the system to ensure stability and confirm that services are fully restored. Users should see normal performance at this time. We will provide a final update once we are confident the issue is fully resolved. Affected components Element451 API (Partial outage)

Status: Resolved This incident has been resolved. The issue affecting application loading in some sites with conditional logic has been fully addressed, and all systems are operating normally. If you continue to experience any problems, please contact our support team. Thank you for your patience while we worked to resolve this issue. Affected components Element451 API (Operational)