HireFire experienced a major incident on January 11, 2024, lasting 1d 1h. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jan 12, 2024, 05:14 PM UTC
On January 11, at 15:00 (UTC), web dynos being autoscaled using the New Relic strategies experienced issues in retrieving metric data from New Relic's API. Upon investigation, it appears that New Relic has increased the processing time for metric data, delaying its availability through their API. Previously, metric data would become available approximately 2 minutes after receipt. Now, it seems to take about 5 minutes. We have applied a patch that accesses data, which, based on our observations, should be consistently available at each checkup. Unfortunately, the downside is that data retrieved from New Relic is now 5 minutes delayed. This might pose an issue depending on the nature of your application and use case. If this delay is problematic and you wish to continue using either the Response Time or Requests Per Minute strategies, we recommend considering a switch to the "Logplex - Response Time" or "Logplex - Requests Per Minute" strategies. Our Logdrain system, integrating with Heroku Logplex, ingests and makes data available within 5 seconds, compared to the current 5-minute delay on New Relic.