• Timeline of Events
• What Happened?
• How was it Addressed?
• Was this a Microsoft Issue?
• What are the Lessons Learned

Timeline of Events¶

CrowdStrike, a leading cybersecurity company known for its advanced endpoint protection platform, notified clients that it was "aware of reports of crashes" of its software on Microsoft Windows operating systems at 5:30 AM EST. This early morning alert set off a chain of events that would unfold throughout the day, affecting organizations across the globe. Shortly after that, there were reports of 911 service outages for several states (including Alaska and Arizona), highlighting the critical nature of the software failure. The disruption to emergency services raised severe concerns about public safety and the vulnerability of essential infrastructure to software glitches. 

In addition to emergency services, the outage had far-reaching effects on transportation and commerce. Airline flights were grounded, causing travel chaos and leaving thousands of passengers stranded at airports worldwide. The grounding of flights inconvenienced travelers and disrupted global supply chains and time-sensitive cargo deliveries. Financial institutions found themselves in a precarious position as their systems went down, potentially compromising transactions, account access, and other critical banking operations. The outage was a stark reminder of the financial sector's reliance on robust cybersecurity measures and the potential economic impact of widespread system failures. 

Healthcare systems were also significantly affected, raising concerns about patient care and data security. With inaccessible medical records and critical systems, healthcare providers faced challenges in delivering timely and effective care, underscoring the importance of reliable technology in modern healthcare settings. News reports showing the dreaded "blue screen" were broadcast throughout the media, visually capturing the frustration and concern felt by users across affected organizations. The iconic blue screen, typically associated with system crashes, symbolized the widespread disruption caused by the CrowdStrike software update gone wrong. 

While not significantly impacted, American Riviera Bank found itself in a fortunate position during this crisis. The bank's ability to withstand challenges is credited to its proactive approach to IT management and strategic partnerships. We were fortunate to have our third-party vendors working hard on our behalf from the early hours of Friday morning. These vendors operate 24/7, 365 days a year, enabling smaller companies like American Riviera Bank to maintain their operations without needing to hire a significant number of IT staff to recover from such events. 

The incident has brought to light the significant value of outsourcing certain IT functions to specialized providers offering round-the-clock support and expertise. This approach, particularly beneficial for smaller organizations, provides access to high-level IT capabilities without the need for a large in-house team, thereby enhancing their cybersecurity measures. 

What Happened?¶

CrowdStrike is an antivirus service with advanced features beyond traditional antivirus software. CrowdStrike's platform uses artificial intelligence, behavioral analytics, and other cutting-edge technologies to detect and prevent sophisticated cyber threats in real time. To effectively prevent malicious activity, CrowdStrike operates at a very low level inside the Microsoft System. This deep integration allows the software to monitor and protect against threats at the kernel level, providing comprehensive security coverage. However, this deep integration also means that any issues with the CrowdStrike software can significantly impact the overall system stability. 

The outages were caused by an update that was pushed out without proper validation. Software updates are critical to maintaining cybersecurity, as they often include patches for newly discovered vulnerabilities. However, in this case, the update itself became the source of the problem. 

Due to the faulty update, anything running Windows that had CrowdStrike installed crashed. The nature of these systems causes a reboot when a critical error occurs. Unfortunately, because the defective software update was still resident on those systems, they crashed again upon reboot and were caught in a 'boot loop'. This continuous cycle of crashing and rebooting rendered affected systems unusable until the issue could be resolved. 

The impact was far-reaching, affecting both Windows desktops and servers. This meant that not only were individual workstations affected but also critical infrastructure components. Emails, websites, and application servers remained down until they could be remediated, causing significant disruptions to business operations across various industries. 

How was it Addressed?¶

The response to this crisis was swift and comprehensive, with IT professionals across the country working tirelessly to fix the issue. The scope and complexity of the problem meant that fixes proved to be exceedingly difficult to complete and time-consuming throughout the day. Initially, one of the most effective solutions was to reboot the system 10 to 15 times, hoping the system stayed up long enough to remove the defective file. While somewhat crude, this approach allowed some organizations to regain control of their systems and begin removing the problematic update. 

As the day progressed, more sophisticated solutions were developed. Scripts were created to help automate the fix, streamlining the process and allowing IT teams to address multiple affected systems more efficiently. These scripts typically involved attempts to boot into safe mode or use command-line tools to remove the defective update files. 

Later in the day, CrowdStrike released instructions leveraging its technology to quarantine the defective file as if it were a virus. This approach proved more effective than the reboot fix, utilizing the existing CrowdStrike infrastructure to isolate and neutralize the problematic component. The incident highlighted the importance of having robust incident response plans and the ability to quickly develop and deploy solutions in the face of unexpected challenges. It also underscored the value of collaboration within the IT community, as professionals shared information and solutions to address the widespread issue. 

Was this a Microsoft Issue?¶

No, this was not a Microsoft issue. The CrowdStrike software update, which affected Microsoft systems, was the cause. It's important to note that Microsoft Windows, one of the most installed operating systems, was the platform on which the issue manifested. However, the root cause lies with the CrowdStrike update, not Microsoft's software. CrowdStrike is widely used in industries such as finance, healthcare, transportation, and government. Heavily regulated industries leverage the software because it's an effective tool for preventing malicious actors from successfully attacking their systems. The widespread adoption of CrowdStrike, particularly in critical sectors, contributed to the far-reaching impact of the outage. 

American Riviera Bank was fortunate that our end-user computing is based on Lynx using "thin clients" that connect back to a virtual desktop. These virtual desktops exist in a cloud-based data center. As a result, software is installed (and uninstalled) in the data center rather than on individual physical machines in the branches. This centralized approach to software management provided an additional layer of protection against the widespread outage, allowing for more controlled and efficient remediation efforts. 

What are the Lessons Learned?¶

The incident provided several valuable lessons for both software providers and users. CrowdStrike immediately identified the deficiency in their process, and it has already been corrected. This rapid response and commitment to improvement are crucial in maintaining trust and reliability in the cybersecurity industry. For organizations and individuals that are not using a solution to protect their business, CrowdStrike does offer a small business suite of products. However, the incident also reminds us of the importance of having diverse security measures and not relying solely on a single solution. 

Groups like the Better Business Bureau also provide free resources for small businesses to consider when evaluating their cybersecurity strategy. These resources can be invaluable for smaller organizations that may not have extensive IT departments or cybersecurity expertise. 

The Federal Communications Commission also has a resource page for small businesses, including a cybersecurity planning tool and links to other helpful information. These tools can help businesses of all sizes develop comprehensive cybersecurity strategies tailored to their specific needs and risk profiles. 

For those seeking to stay updated on the latest cybersecurity trends and best practices, SCORE offers a webinar on cybersecurity for 2021-2022. This resource provides valuable insights into emerging threats and effective countermeasures. FINRA also provides a comprehensive business continuity planning template for those involved in financial services or wealth management. This template can be a valuable starting point for developing robust continuity plans to help organizations weather unexpected disruptions like the CrowdStrike outage. 

In conclusion, while the CrowdStrike outage caused significant disruptions, it also provided valuable lessons about the importance of robust cybersecurity measures, diversified IT strategies, and comprehensive incident response planning. By learning from this event and implementing more robust safeguards, organizations can better protect themselves against future software-related disruptions and maintain continuity in an increasingly digital world. Whatever your industry, it's likely dependent on technology to some degree. Incidents like the CrowdStrike outage serve as a stark reminder that we should constantly be evolving our systems to stay ahead of potential issues. This event presents an excellent opportunity to evaluate your infrastructure and identify the most challenging areas to fix during the crisis. 

With the benefit of hindsight, consider if there are any aspects of your IT setup that you would change. Do you have a robust and effective business continuity plan to recover from such incidents? It's crucial to regularly review and update these plans to ensure they remain relevant and effective in the face of evolving technological landscapes. 

We hope you found this information valuable and that it prompts you to take a proactive approach to your organization's IT infrastructure and security measures. Thank you for reading and staying informed about these critical issues. 

For more detailed information and guidance, please visit this link: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/.