Edlink Outage History

Started Sep 01, 2026, 03:42 PM UTC· resolved Sep 01, 2026, 03:52 PM UTC

1. investigating Sep 01, 2026, 03:42 PM UTC

   We are currently experiencing degraded API performance. We've traced the incident to our primary hosting provider (GCP) and we are currently exploring ways to relocate resources to our API servers. We are currently processing an approximately-normal level of login requests, but dashboard performance is severely degraded.

2. resolved Sep 01, 2026, 03:52 PM UTC

   Our control plane is back online and we appear to be serving API requests as expected. Syncs and materializations should resume shortly.

Read the full incident report →

Started May 29, 2026, 08:00 AM UTC· resolved May 29, 2026, 08:00 AM UTC

1. resolved May 29, 2026, 03:46 PM UTC

   This morning, we experienced an issue with the execution of custom transformations during the final materialization phase of our data pipeline. Custom transformations are utilized by approximately 22% of our integrations. Integrations without custom transformations were not impacted. Integrations that utilized "Edlink-defined" transformations (such as "Infer Roles") were not impacted. Timeline of Events - Our transformation execution relies on private NPM packages in our NPM organization. - Around 3:12am CT, the first custom transformation failed because it was unable to load packages from NPM. - The first client issues were reported around 8:30am CT, but it was not apparent that this was a widespread phenomenon until approximately 9:50am CT when it was escalated to P1. - We resolved the issue at approximately 10:10am CT, but due to latency, materializations did not resume until approximately 10:23am CT. Root Cause The root causes was related to an expired credit card. After a failed payment, NPM unexpectedly downgraded our account and we were no longer able to access our own packages. We re-upgraded the account and after some brief latency due to their billing system, we are able to access packages again. Changes Going Forward 1. In response, we plan to introduce additional alerting to detect unusual changes in our materializations. 2. We are also seeking alternatives to NPM for holding our private packages for materialization execution. Our use of NPM was a technical requirement when we first implemented our custom transformation requirement, but that may no longer be the case.

Read the full incident report →

Started May 07, 2026, 11:19 PM UTC· resolved May 08, 2026, 08:41 PM UTC

1. monitoring May 07, 2026, 11:19 PM UTC

   Instructure is currently undergoing a data breach (separate from the one earlier this week) and the attackers have gained control of many Canvas instances. Users of these instances will not be able to sign in or access any content. Data syncs (either to or from Instructure) are presumed non-functional at the moment. The situation is developing.

2. resolved May 08, 2026, 08:41 PM UTC

   Instructure has published a status update indicating that they do not believe any data was compromised in the most recent attack. We have published a longer statement on the topic (including our plans) here: https://www.linkedin.com/pulse/re-instructure-security-incident-dakota-gordon-8nyqc/

3. postmortem May 11, 2026, 09:11 PM UTC

   You may know by now that Instructure was breached by bad actors twice in the last week. At this time, we have no reason to believe that our clients \(or Edlink itself\) was compromised. However, out of an abundance of caution and as a best practice, to have all districts using Canvas to rotate their API and LTI keys. Here's why: While it is Instructure’s current position that there’s “no evidence” that any data was accessed,we don’t share their level of conviction. This assumption is based on the fact that attackers hijacked substantially all production Canvas instances to show a ransom message, which would require a fairly high level of access to Instructure’s infrastructure and systems. As a result, if a bad actor did get access to Instucture core systems, they'd likely have access to API and LTI keys. For API & LTI 1.3 integrations, exposed keys mean that attackers can exfiltrate any data that those keys have access to. Attackers can act as your application and use end user tokens to retrieve data. It is unlikely that attackers will be able to impersonate Canvas users to sign into your platform, assuming you have correctly implemented OAuth 2.0 or OIDC \(for LTI\). For LTI 1.1 integrations, exposed keys means that attackers can potentially sign into your product as “legitimate” end users. This can lead to possible data exfiltration from your product and it will be difficult or impossible to tell if traffic is legitimate. As such, we recommend that you immediately rotate all LTI 1.1 keys or upgrade to LTI 1.3, if possible. In either case, the attackers could “sit” on stolen credentials for months or years before they decide to use them. By the time they do make their move, this incident may be a distant memory and it will be unclear to those affected exactly how the unauthorized access was obtained. Later today or early tomorrow, we will release a user interface that we can share with you to share with your Canvas school customers to make the key rotation process as seamless as possible for school IT admins. If you'd like to rotate keys sooner, we would be happy to work directly with you or your schools to knock this out.

Read the full incident report →

Started May 04, 2026, 06:16 PM UTC· resolved May 04, 2026, 07:57 PM UTC

1. monitoring May 04, 2026, 06:16 PM UTC

   Reports of degraded API & SSO performance started coming in around 12:30pm CT. After investigation our logs, it appears that the degraded performance started at exactly 12:07pm Central Time (about 27 minutes prior to first reports). As of 1:04pm Central Time, we have reallocated database resources and the immediate issue has been resolved. SSO & API requests have returned to expected levels. We are continuing to investigate the root cause of the issue before restoring normal pipeline services. Please expect delays in sync and materialization queues while we investigate the issue.

2. monitoring May 04, 2026, 06:30 PM UTC

   We've isolated the issue to our sync service that retrieves data from school data sources. We are re-enabling enrichment & materialization. We are going to revert to the previous stable version of the sync application and re-enable it in a limited capacity for monitoring.

3. resolved May 04, 2026, 07:57 PM UTC

   We've identified the root cause and the pipeline has resumed. We will publish a full post-mortem shortly.

4. postmortem May 04, 2026, 09:08 PM UTC

   **Date of Incident:** May 4, 2026 **Duration:** 12:07 PM – 1:04 PM \(57 Minutes\) **Impact:** Total interruption of user login capabilities. **Status:** Resolved ### **Executive Summary** Between 12:07 PM and 1:04 PM today, Edlink experienced a severe API degradation that prevented substantially all users from logging into client products via SSO. The incident was caused by database connection starvation, which was triggered by a service deployment that utilized an older pinned version of the PostgreSQL \(PG\) library. The issue was mitigated by bringing down the problematic service to free up connections, followed by a full revert of the change. Normal operations have been fully restored. ### **Impact** * **User Experience:** Substantially all users attempting to access Edlink during the 57-minute window were completely unable to log in or authenticate. A small portion were able to authenticate normally. * **System Impact:** Critical services, including the authentication/login service, were starved of database connections and timed out or dropped requests. ### **Root Cause** The incident was traced back to a deployment that occurred today at 12:07pm CT where a part of our service was upgraded. **Background on Dependency Version Pinning:** To provide context on _why_ an older version was in use, we recently implemented a strict policy for managing our software dependencies. This proactive security measure was enacted following an industry-wide supply chain attack incident in March of 2026. Edlink was not affected by this incident directly, but we determined that the attack vector was not one that we were comfortable with, and as such, we moved to proactively update our systems. To safeguard Edlink's platform and ensure our systems remain secure, we instituted a policy to "pin" or lock all our software dependencies to explicitly verified, “safe” versions. This prevents our systems from automatically downloading new, potentially unverified updates. This is the attack vector that was used in the industry-wide attack a few weeks back. While this policy is crucial for preventing malicious, unverified updates from automatically infiltrating our systems, in this specific instance, it had an unintended side effect. In this situation, it forced our deployment to utilize an older version of the PG library than we had intended. This older version of the library managed Postgres connections highly inefficiently and would leak unused connections. Upon deployment, the service rapidly consumed the available database connection pool. Because the connections were not being properly released or managed, other critical infrastructure \(notably the service handling user logins\) was left without available database connections, leading to the system-wide login failures. ### **Timeline** * **12:07 PM:** The deployment containing the outdated PG library goes live. Database connection pools immediately begin to spike. Users start experiencing login failures. * **~12:45 PM:** Emergency mitigation is enacted: the problematic service is brought down, immediately releasing its held connections back to the pool and restoring baseline functionality for the login service. * **1:04 PM:** The deployment is officially reverted to the previous stable state. System stability is confirmed, and the incident is closed. ### **Resolution and Recovery** The immediate bleeding was stopped by intentionally spinning down the problematic service, which allowed the authentication services to recover. The underlying code change was then completely reverted in version control and redeployed to ensure the service could be brought back online safely without causing a secondary outage. ### **Preventative Measures** To ensure we can more quickly identify and resolve similar issues in the future, we have integrated additional monitoring tools focused on detecting failed logins. This will allow us to immediately detect authentication issues and mitigate them before they cause a prolonged system-wide impact. Additionally, we have conducted a review of any other dependency packages that may have been inadvertently downgraded during this “pinning” process.

Read the full incident report →

Started Mar 02, 2026, 08:30 PM UTC· resolved Mar 02, 2026, 08:30 PM UTC

1. resolved Mar 02, 2026, 08:41 PM UTC

   For 172 seconds, our meta API (including SSO endpoints) were unresponsive. This incident took place around 2:30pm CT. The incident was caused by a lock on a table related to single sign on. The lock was the result of a database maintenance query that we were running at the time. The query was expected to complete in less than one second (and therefore, not cause any service disruption), but maintained the lock longer than expected. We terminated the execution of the query approximately two minutes later.

Read the full incident report →

Started Dec 09, 2025, 05:41 PM UTC· resolved Dec 23, 2025, 05:14 AM UTC

1. investigating Dec 09, 2025, 05:41 PM UTC

   We are currently investigating reports of an unusual number of School API errors. This may affect any Schoology integrations, including ones where you are only accessing class roster data via Edlink. We are receiving random (intermittent) 403 errors from AWS Cloudfront (Schoology's CDN provider). These errors do not actually correlate to permissions issues in Schoology as they appear to be random and will succeed after several retries or a significant time delay. Our system, by default, replays external requests up to 5 times, but they are in relatively short succession so this retry logic does not provide a workaround in this situation. Roster data is all still available (the Edlink API itself still returns data as usual), but Edlink may not receive subsequent roster updates from Schoology during this time. We are attempting to work with Schoology to help them determine the root cause.

2. identified Dec 11, 2025, 09:14 PM UTC

   We have a deployed an update to our roster sync code to try to workaround this ongoing issue. This should help roster data begin to flow again. This change does not impact the assignments, grades, and coursework APIs, so you may still experience random, intermittent 403 errors from Schoology.

3. identified Dec 16, 2025, 02:56 PM UTC

   The issue with the Schoology API has been escalated to "major". At this point, it appears that close to 100% of API requests to Schoology are failing. This includes SSO, syncs, coursework, and grade requests. We are temporarily pausing all Schoology syncs in order to free up queue space for other sources.

4. monitoring Dec 17, 2025, 01:10 AM UTC

   We have noticed some API requests begin to succeed again. We are not sure if this is temporary (or if it applies to 100% of API requests) but we are monitoring the situation.

5. monitoring Dec 17, 2025, 04:53 PM UTC

   Everything appears to be back to normal, we're waiting on a few more districts to sync.

6. monitoring Dec 17, 2025, 10:21 PM UTC

   ...and we're back to seeing elevated error rates again after a relatively smooth morning and early afternoon.

7. monitoring Dec 22, 2025, 03:37 PM UTC

   Error rates have dropped again and we are begging to resume syncs. Last word from Powerschool is that they are "looking into it" (as of last Friday).

8. resolved Dec 23, 2025, 05:14 AM UTC

   Although we have not heard any official messages from Schoology, it appears that the API issues have been resolved on their end. Sync have resumed.

Read the full incident report →

Started Nov 18, 2025, 06:06 PM UTC· resolved Nov 19, 2025, 10:10 PM UTC

1. identified Nov 18, 2025, 06:21 PM UTC

   We are currently experiencing a database pressure issue in our US Central region (which holds most of our data). In an abundance of caution, while we sort out the issue, we have voluntarily disabled some nonessential background services, including: - Scheduled syncs - Data enrichments - Materializations - Queued jobs & tasks ***Core platform functionality (API, SSO, Dashboard, Widgets, etc.) remains fully operational.*** What’s Happening: An internal database counter exceeded its safe threshold. When this number gets too high, the database temporarily prevents new write operations. In order to slow the growth of this counter while we fix the issue, we've reduced the number of workloads on the database. What We’ve Done So Far: - Identified the cause - Temporarily paused non-critical syncing and materialization tasks during recovery - We are manually running certain database maintenance processes to help the database recover more quickly Current Impact: - Only background data tasks are delayed (intentionally) - No data has been lost - Primary application operations continue as normal Next Steps: We will monitor the configuration update. If it's enough, we'll bring non-essential services back online. If it isn't, we may need to restart the database which will happen at non-peak time and require < 10 minutes of downtime. Next Update: We will provide another update at the end of the day, or if the situation resolves or escalates, whichever occurs first.

2. identified Nov 18, 2025, 09:04 PM UTC

   Update 3:02pm CT: We are still working on the core issue, but we are beginning to resume background operations. Currently, we are running at partial capacity for enrichment, materialization, queue, and sync jobs. Manually-initiated jobs (i.e. jobs started by clicking a button in the dashboard) will be prioritized in the queue. Please expect longer than usual queue times on sync jobs.

3. resolved Nov 19, 2025, 10:10 PM UTC

   Everything has returned to normal and the job queue is back down close to zero.

Read the full incident report →

Started Jun 12, 2025, 06:59 PM UTC· resolved Jun 12, 2025, 10:30 PM UTC

1. identified Jun 12, 2025, 06:59 PM UTC

   Currently Google Cloud Storage is down globally. This is an insanely rare event. Edlink's API is currently functional, but some requests may fail. The SFTP transfer (Edlink's SFTP servers) are currently down. So new roster data can't be transferred into Edlink via SFTP. Existing roster data is unaffected. Functions of the Edlink Dashboard that involve SFTP (i.e. CSV mapping) may be affected. In some instances, when loading the Administrator Onboarding Flow, Edlink's clients' logos / icons may not load.

2. identified Jun 12, 2025, 07:00 PM UTC

   Here's a link to the Google Cloud page showing their status: https://status.cloud.google.com/regional/americas

3. monitoring Jun 12, 2025, 08:40 PM UTC

   A fix has been implemented by Google Cloud and we are monitoring the results. We are now able to connect to the Edlink SFTP servers and API requests are stable.

4. resolved Jun 12, 2025, 10:30 PM UTC

   Google services are back up and running.

Read the full incident report →

Started Mar 05, 2025, 10:16 PM UTC· resolved Mar 05, 2025, 10:16 PM UTC

1. resolved Mar 05, 2025, 10:41 PM UTC

   Issue: - We updated our Kubernetes HTTP route configuration at approximately 4:16pm CT. - This route configuration was slightly out of date and contained an invalid "catchall" route that used to route traffic to the Edlink Dashboard. - The issue was identified and fixed at approximately 4:26pm CT. Impact: - This issue only affected the Edlink Dashboard, not our API, SSO, Widgets, Upload, Data Pipeline or other related services. - It was likely not noticed by many users as it would only have impacted people who were attempting to load the dashboard in a new tab or window. - If you already had a dashboard page open, it would have continued to work uninterrupted (as our API was unaffected).

Read the full incident report →

Started Nov 12, 2024, 12:37 AM UTC· resolved Nov 12, 2024, 12:47 AM UTC

1. investigating Nov 12, 2024, 12:46 AM UTC

   Right now the Edlink dashboard is not responding. It is returning a message that says "no healthy upstream." We're currently investigating the issue. Edlink's API is not affected. SSO is not affected - all end-users should be able to launch and sign in as normal. Only users attempting to interact with Edlink via the dashboard (i.e NOT teachers or students) are potentially affected, as of now, that we know of.

2. resolved Nov 12, 2024, 12:47 AM UTC

   The incident has been resolved. We will issue a full post-mortem writeup during business hours tomorrow (10am CST - 6pm CST).

Read the full incident report →