Recent incidents involving Microsoft and CrowdStrike, which disrupted various sectors and geographies, have highlighted the significant risks of forced updates, especially those involving kernel-level access in proprietary software, said a top technology executive.
Forced updates, a common practice among technology vendors, allow them to make changes to systems at any time, often without user verification, leading to frequent disruptions and potential security breaches.
“It amazes me that the industry allows this from both a cybersecurity and a reliability standpoint,” Doug Milburn, President and Founder, 45Drives, said, adding that even minor errors, such as the one that recently required a temporary fix involving a small text file and a safe mode reboot, can cause significant issues.
Citing former US President Ronald Reagan’s often-repeated Russian proverb—’trust, but verify‘—he said that there is an inherent problem with proprietary software like Microsoft Windows, where users cannot verify updates.
Stating that forced updates by tech companies and the inability of enterprise users to verify what is inside the update is a recipe for significant risk, he said businesses must explore open-source technologies for more control over their digital infrastructure.
Edited Excerpts:
The recent CrowdStrike and Microsoft incident highlighted the risks associated with forced updates by proprietary software vendors. How do you view the impact of such practices on cybersecurity and system reliability?
Many companies purchase CrowdStrike Managed Detection and Response (MDR) solutions for enterprise-level antivirus protection. Both Microsoft and CrowdStrike are highly respected companies, known for their solution, services and risk mitigation capabilities. However, they have a practice that we deeply disagree with: forced updates. This practice is also objectionable to many of our customers. What happened was not only inevitable, but it is a regular occurrence, almost weekly. When you give a fully proprietary vendor access to your computer systems, they can make changes anytime they want.
It amazes me that the industry allows this from both a cybersecurity and a reliability standpoint. A single line of code can cause issues. In the case of CrowdStrike and Microsoft, the temporary fix was to remove a small text file and reboot in safe mode. This was a very small error. Such small errors are unavoidable in the tech industry.
In other cases, someone might do something malicious, like injecting backdoors or Trojan horses into software. I do not doubt these companies’ intentions and efforts to be secure, but just as you will never eliminate all bugs, you will never eliminate all possibilities for attacks and backdoors in software. To quote an American President Ronald Reagan: “Trust but verify.” That’s the issue with proprietary software like Microsoft Windows.
Before the CrowdStrike event, our own organisation, with about 400 employees, was adversely affected twice in the same week. We use several hundred Microsoft Windows desktops and some Linux desktops. Our sales team uses Microsoft Outlook, and there was a bug in Outlook that affected our CRM software, causing a crash. Our Salesforce was down for two to three hours while our IT team manually rolled back updates on every machine. Just two days later, another forced update broke the drivers for our Roland industrial printers, shutting down production for four hours. Such occurrences are common with forced updates.
When you have a proprietary software provider, even if they allow you to install updates yourself, you cannot verify what’s inside those updates. This presents potential risks.
Can the potential risks associated with proprietary software providers be minimised by adopting open-source solutions, which seems to offer better transparency and control?
Absolutely. In the open-source world, everything is transparent and voluntary. We rigorously vet updates before deployment to ensure nothing gets broken. An example of open-source transparency: in March, someone discovered unusual behaviour in an updated SSH (Secure Shell) program used to remotely administer Linux. They found that a backdoor had been injected, possibly by a foreign government or hacker group. Because the source code was open, it was found and addressed before any significant damage occurred.
Forced updates by technology providers, no ability to verify, and the potential for major disruptions present significant risks. With open-source solutions, we have control and transparency. Though they require effort to maintain security and reliability, they protect against both malicious activities and inadvertent mistakes.
We have replaced much of our backend and server infrastructure with open-source alternatives. While we still use Microsoft products, we are aware of the challenges their practices pose. Forced updates and unverified changes can lead to costly disruptions and security risks.
While open-source solutions require effort, they offer better security and reliability, protecting against both malicious activities and inadvertent mistakes. Investing in open-source technology is a viable path to achieving these goals.
In this saga, the biggest irony seems to be around resilience. In your view, how can organisations with heterogeneous IT infrastructures and critical operations maintain resilience and security, particularly when dependent on proprietary vendors like Microsoft?
It is simple—redundancy, redundancy, redundancy—and meticulous change management, as issues often arise during changes. The most resilient approach to software is to keep it on-premises and manage changes carefully. When you own the software, it remains stable unless there’s a hardware failure or data corruption, which are manageable and not systemic across your entire network.
We are global experts in using open-source storage clustering software for file systems. We specialise in building systems for various clients, from military organisations to backend infrastructure providers, where downtime is critical. We create rugged storage clusters with high availability, redundancy, elasticity, and scalability.
We implement redundancy at the server level so that a server failure isn’t an issue. Redundant networking, geo-replication, and failover are standard practices. By incorporating these redundancies, you ensure your system is robust. In rare catastrophic events, like what we call a “meteor strike” on your server room, having another server ready for failover is routine for us.
Updates and changes present the next risk. It’s essential to have either in-house expertise or use knowledgeable providers like us who understand mission-critical operations. We mitigate risks by testing updates in limited areas before full deployment. For example, we thoroughly test Enterprise Linux and Enterprise open source software. We are also a founding sponsor of Rocky Linux, the successor to CentOS.
It’s crucial to stay six months to a year behind on most updates to ensure they are well-vetted. We avoid the bleeding edge of technology for mission-critical systems and require extensive redundancy for hard drive, server, network, and power failures, or even against a malicious employee. Software-level redundancy and frequent data snapshots are vital. We also use ransomware protection with our developed behavioural analysis technology.
Regarding the risks with cloud and proprietary systems with forced updates, these risks are unmanageable. You must trust without verification, and any issue in their organisation can propagate to yours. Worst-case scenarios are common occurrences. If external vendors can access your system at will, you are defenceless.
It’s about redundancy, careful change management, and conservative update rollouts. Have knowledgeable staff or work with expert service providers, or a combination of both. Keep your software on-premises and maintain control. No one else should access it until you are absolutely ready. That’s how you stay safe.
Given that CrowdStrike’s Falcon solution has kernel-level access in the operating system, which provides privileged access, do you think there should be a mandatory simulation period on a subset of devices before rolling out any updates globally? Could such a simulation have prevented the current situation we are facing?
Absolutely, a mandatory simulation period before rolling out updates globally is crucial, especially for solutions with kernel-level access like CrowdStrike’s Falcon. This approach mirrors our own practices. We roll out updates very slowly and carefully. First and foremost, we would never advise a customer to allow anyone, at any time, to have access to the kernel.
Our view on security is that while you may trust a proprietary vendor, there is always a risk. For example, someone could trick a developer into injecting malicious code. If that happens with a proprietary company, you are helpless. The malicious update would be deployed across the system, and you would not even know. We believe in owning our software and being responsible for updates. When updates are needed, we ensure they are done correctly.
I started computing a long time ago with microcomputers, DEC PDP-11s, and similar systems. As the PC world evolved with 8086 machines, 286, 386, and beyond, Microsoft led the way, followed by Linux. We embraced updates, always waiting for something better. But when you run a business, you need to evaluate if your current infrastructure is performing well. If it is, what is the benefit of non-security updates? Security updates are necessary, but otherwise, what is the need for constant updates? Most of our systems do not require frequent updates. Just because something is new does not mean it is better for enterprise computing.
In enterprise computing, new is not always better. Many legacy systems are still in use, and moving forward requires caution to avoid putting your organisation at risk. Companies like Microsoft frequently release updates, even for trivial changes like colour schemes. Allowing them to update your system without verification is foolhardy.
It is not about being backward in enterprise computing; it is about being deliberate. There are always risks to change, but what are the benefits? Allowing your provider to change whatever they want is risky. Look at the Windows 8 fiasco. Microsoft made Windows touchscreen-oriented like a smartphone OS, which was a disaster for business users who needed a stable, familiar interface, not a trendy new look.
Another example is VMware. It’s legacy software, and after being bought out by Broadcom, millions of users faced a price regime that did not make sense for their needs. When a company owns you, they run their business for their purposes, which can be detrimental to customers.
In the open-source world, you own your software infrastructure and the hardware it runs on. Even if the world changes, what you have on your premises will still work. You can keep it as it is today, and it will continue to work. You can move forward because open-source is constantly updated. When something like CentOS changes, the community responds. Developers can take the open-source code and create a new branch like Rocky, which focuses on enterprise reliability and security. It is a robust model that offers security at all levels with the advantages of open source, rather than being controlled by a proprietary company.
How can organisations ensure the security and reliability of open-source solutions compared to proprietary systems, especially given the perception that proprietary systems are more secure?
The biggest challenge we see in the adoption of open source during the sales process is competing against legacy and proprietary solutions, which are perceived to be more secure. The reality is that the notion of open source being less secure is simply not true. Such claims lack evidence. A hardened-by-default open-source implementation can be as robust as any proprietary solution. Keeping kernel access locked away makes it even more secure. Although it is difficult to provide specific numbers due to the small number of incidents and the tendency for them to be underreported, major breaches predominantly occur in proprietary systems.
If you review media reports, you will find that most significant breaches involve proprietary systems. A sensible implementation of open source will generally result in a more robust system. However, it all comes down to trust. You must trust your vendors, and with open source, you must trust the projects. Trust without verification in mission-critical systems is a recipe for failure. Companies like Microsoft will never admit to shutting you down or allowing data theft, but these issues happen frequently.
To mitigate these risks, stick with enterprise open source. Be very conservative with distributions and kernel levels. Harden everything by default, close down all unnecessary services, and only open what is absolutely necessary. Be cautious with your architecture and choose your protocols wisely, as some proprietary protocols are more secure than others.
For instance, the cost of an open-source storage cluster like Ceph is a fraction of what major cloud providers charge. The infrastructure is entirely under your control. While remote access does introduce risks, these are universal. The biggest threats are social attacks, which exploit vulnerabilities within your organisation.
In the defence sector, a significant part of our business, we deploy open-source solutions in air-gapped environments, and they perform exceptionally well. While there is always a fundamental risk when not air-gapped, open source can still be highly secure if properly designed. The primary advantage of open source is the level of control it provides. You are not dependent on vendors like CrowdStrike or Microsoft.
Keep your open-source solutions well-protected, be conservative in their use, and thoroughly vet every update. While nothing is foolproof, the security and reliability of open source, when properly managed, are superior.
