Automated Update Risks Under Scrutiny After CrowdStrike Outage

The July 2024 mass IT outage that caused the notorious Blue Screen of Death (BSoD) to 8.5 million Windows machines – wreaking havoc across airlines, hospitals and other user organizations – was initially caused by a defective third-party software configuration update that was issued by cybersecurity company CrowdStrike.

While this unprecedented global event was caused by a glitch and not an attack, there are still important lessons that cybersecurity professionals, IT leaders and software development teams can glean from this event. Among these takeaways is how to reduce the risk that comes with automatic software updates from an upstream third-party partner.

From the user organization’s perspective, it behooves IT and cyber decision-makers to ask themselves: Should all of my devices be receiving automated updates from my software suppliers?

On one hand, automated updates are the most efficient way to install the latest version of a particular software program. And certainly, it is important for businesses to update in a timely manner, especially if a particular software program is harboring severe code vulnerabilities that must be patched before they can be leveraged by malicious actors.

Moreover, in the case of cybersecurity software – CrowdStrike Falcon or otherwise – failure to update your cyber platform might mean that the solution will not be able to optimally identify and flag the latest malware programs should signs of infection reach your endpoints, workloads or other assets.

But the downside to an automated update is that you are ceding control to a third-party  vendor, and accepting the risk associated with letting them execute the deployment of their newest software version. If there is any concern that an unreliable update could disrupt a vital system within your organization, then you may wish to consider requiring manual approval before any such update is executed.

As part of this decision, your company might also want consider the reliability and trustworthiness of the upstream software manufacturer that is providing the update based on historical precedent and third-pasty assessment results.

Companies should also bear in mind the current cyberthreat environment. For instance, does the latest threat intelligence show that known vulnerabilities in your third-party software are being actively exploited in an ongoing campaign that potentially poses a risk to you?

Meanwhile, the provider of the software has its own responsibility to ensure that a faulty automated update does not cause the downfall of its downstream clients. Developers should ask themselves: What more can I do to ensure that the updates I’m distributing are fully vetted for mistakes?

One of CrowdStrike’s mechanisms that is supposed to prevent a troublesome update is its Content Validator tool, which checks for problematic code. Ironically, according to CrowdStrike’s preliminary post-incident review, the validator tool itself had a flaw that caused it to miss “problematic content” in a Rapid Response Content configuration update that “resulted in an out-of-bounds memory read triggering an exception,” which “could not be gracefully handled, resulting in a Windows operating system crash (BSOD).”

In response to these findings, CrowdStrike said that it has now introduced new safeguards to its Rapid Response Content, including more thorough testing, additional validation checks and enhanced error handling. Furthermore, its automated deployment process will now be staggered, so that any problematic update would only affect a subset of recipients before the issue became apparent.

CrowdStrike also said it would improve monitoring of sensor and system performance, provide customers with more discretion over where and when updates are deployed, and introduce new third-party validation processes.

In the wake of July’s major outage, time will tell if even more software suppliers and users will demand the adoption of such practices in the future.