The recent global IT outage caused by a faulty update to CrowdStrike’s Falcon platform has highlighted the vulnerabilities many businesses face when relying on single-vendor solutions. On July 19, 2024, a Rapid Response Content update from CrowdStrike led to widespread crashes of Windows devices, resulting in significant disruptions across various sectors, including healthcare, transportation, and general business operations. This incident affected approximately 8.5 million devices worldwide, necessitating extensive manual intervention to resolve the resulting issues.
The Incident
The problem stemmed from an undetected error in a template instance used by CrowdStrike’s content configuration system. This error led to out-of-bounds memory reads and unhandled exceptions, causing the infamous Blue Screen of Death (BSoD) on affected devices. The repair process was particularly challenging for systems with BitLocker enabled, as recovery keys were often stored on servers that had also crashed.
Despite the extensive impact of this outage, CrowdStrike and Microsoft have since implemented improvements to their testing and deployment processes, including enhanced error handling mechanisms and a staggered deployment strategy for updates.
Approved Systems’ Resilient Approach
At Approved Systems, we are proud to report that neither our company nor our customers were affected by this incident. Our robust IT infrastructure and proactive management practices played a crucial role in ensuring uninterrupted service during this period.
We employ a combination of in-house Remote Monitoring and Management (RMM) tools, alongside world-renowned systems for patch management. Our approach includes a staggered rollout system for Windows patching and updates, which is enabled by default. This method significantly reduces the risk of widespread issues from updates, as it allows us to detect and resolve potential problems before they affect all systems.
Empowering Change Champions
A key component of our strategy is empowering our clients through the nomination of “Change Champions” within their organizations. These individuals are trained to recognize, report, and address issues that may arise during updates. By having knowledgeable staff on hand, our clients can quickly identify and mitigate potential disruptions, ensuring smoother transitions during system updates.
Our Role During the CrowdStrike Outage
During the CrowdStrike incident, Approved Systems extended support to businesses affected by the BSoD boot loop saga. Our team was available to provide expertise and assistance, helping impacted organizations restore their systems and resume normal operations. This commitment to the broader business community underscores our dedication to supporting resilient IT environments.
Conclusion
The 2024 CrowdStrike incident serves as a reminder of the importance of robust IT management and proactive support systems. At Approved Systems, we remain committed to delivering reliable, secure, and efficient IT solutions to our clients. By leveraging advanced tools, empowering Change Champions, and maintaining a flexible update strategy, we ensure that our clients can navigate the complexities of modern IT with confidence.
For more information on our services and how we can support your business, please contact us at helpdesk@approvedsystems.com.au.