How Patch Tuesday Keeps the Beat After 20 Years

On Oct. 9, 2003, Microsoft CEO Steve Ballmer announced that the company would only issue security patches once a month to “reduce the burden on IT administrators by adding a level of increased predictability and manageability.”

Two decades later, Microsoft continues to issue its security updates on the second Tuesday of every month, with occasional exceptions for emergency situations, and many other companies like Oracle and Adobe follow similar rules.

Patch Tuesday turned risk management into a monthly appointment, but like many innovations, it was founded from crisis. At the beginning of 2002, shortly after the world experienced two of its first cyber doomsdays, Code Red and Nimda, Microsoft decided to change its philosophy around security. The two worms, which infected hundreds of thousands of machines in a matter of hours, exploited vulnerabilities for which patches were available.

“The problem was not that we weren’t building patches; the problem was that people weren’t deploying them quickly enough,” says Christopher Budd, who was with Microsoft from 2000 to 2010 and is now a senior manager for threat research at Sophos.

At that time, the trauma of 9/11 was palpable in North America, and there was a growing concern within Microsoft regarding security. On Jan. 15, 2002, Bill Gates sent his famous Trustworthy Computing memo, highlighting the importance of protecting customers and their systems.

One way to improve security was to change how patches were delivered. So instead of announcing them unpredictably, on a ship-when-ready basis, several times during a week, Microsoft began to stack them together to make it easier for customers to keep up with everything.

At first, the idea was implemented as a “silent pilot project,” Budd says. But, as it showed promising results, it was soon made official. The first official Patch Tuesday report was published on Oct. 14, 2003, and it included seven vulnerabilities, five of which were seen as critical.

“Tuesday was the earliest day in the week that we felt we could sustainably offer these,” Budd, who helped build Patch Tuesday, says.

This fixed-schedule approach has become a standard industry practice. But the road to that was not always smooth.

The Early Days of Patch Tuesday

Throughout the years, Microsoft has evolved and refined its approach to security patches, adapting to changing threats. Notable worms that followed Code Red, such as Sasser and Blaster, helped Patch Tuesday to grow and eventually mature.

Budd and his colleagues in the Microsoft Security Response Center, who “suddenly became incredibly important” after the Trustworthy Computing memo, tried to make improvements around patch deployment and detection. One key achievement was to build tools that enabled admins to do a scan and identify the systems that needed security updates — an idea that, while simple, proved to be a game-changer.

The entire process of releasing patches required extensive work, particularly on the Monday before Patch Tuesday, when everything had to be checked. Security experts put in long hours, missed birthday parties, and even showered in Microsoft’s headquarters because they didn’t have the time to go home.

It is why the releases of the bulletins were celebrated with music blasting over loudspeakers.

“When the bulletins were live, it was a big deal for us,” says Dustin Childs, who was with Microsoft between 2008 to 2014, and is now the head of threat awareness at Trend Micro’s Zero Day Initiative.

Typically, the music was picked by the release manager for that bulletin.

“I remember one month it was Klingon music, but it was usually rock and roll,” Childs says.

Occasionally, soon after the music stopped, chaos would ensue because some patches caused unintended consequences. One notable incident happened in February 2010, when thousands of customers reported blue screens almost immediately.

When security experts in Redmond heard about the blue screen issue, they tried to replicate it but could not succeed. For the lack of a better solution, Microsoft bought the computer of a customer who called a support center near Dallas to report the issue.

“And as soon as we got that computer, we found that it had a rootkit installed in it,” Childs says.

The Alureon rootkit made modifications to Windows Kernel binaries, which caused the systems to be unstable. The company reissued the patch and fixed the problem.

“The really funny part, though, was that the people who made the rootkit figured it out faster than we did, and they had updated their rootkit to avoid the patch within 48 hours [of the release],” Childs says. “It took us a week to fix it!”

And it wasn’t the only bizarre episode in Patch Tuesday’s history. Childs remembers, for instance, that an Internet Explorer patch crashed online banking in South Korea, while a Windows Media Player patch broke an entire country — twice.

“For some reason, on systems in Denmark, it blue-screened,” he says. “So we had to recall that patch and fix it, re-release it. And then, the very next month, the same thing happened again. I don’t know what was specific about the systems in Denmark.”

Even today, some patches released by software companies in general, not just Microsoft, can cause issues, which can be frustrating for those installing them. Childs argues that if vendors improve the reliability of these fixes, customers might be more willing to apply them promptly, as opposed to waiting weeks or months to see if others experienced issues.

Taking the waiting approach can be risky, as it leaves their systems vulnerable to known vulnerabilities. It is why Childs recommends users apply patches as soon as possible. He also tells them to call attention to poor-quality patches.

“Let’s hold vendors accountable,” Childs says. “Let’s make sure that they’re producing good patches.”

Aanchal Gupta, deputy CISO at Microsoft, says the company is doing “extensive testing” of patches.

“Before we issue any patch to our customers, it goes through rigorous testing not just within Microsoft. … [W]e also have a set of beta testers, external companies, and they get the patches early on before these are actually made public,” she says. “They can test in their environment and report back to us whether there is something unique in their environment which could make the patch to not work.”

Patch Tuesday Today

Patch Tuesday has come a long way since those early days when Klingon music blasted from the speakers. Over time, the releases have become quieter and even automated, becoming invisible to some users.

In the past decade, Microsoft made several changes to streamline the process. In the mid-2010s, for instance, it announced cumulative updates so that a customer who missed, for instance, five patches only needed to apply the last one because the other ones were included in it. The company also launched machine-readable Security Update Guides, which meant that organizations with huge fleet deployments could rely on automation.

But not every change was welcomed by the community. When Microsoft decided to eliminate security bulletins and replace them with Security Update Guides, customers complained that the information they received was less intuitive. Something similar happened in the fall of 2020, when the tech giant removed executive summaries that included detailed information on vulnerabilities. Once again, many in the industry argued they didn’t receive enough information, which made their decision-making processes difficult.

That was “probably the most disruptive” decision Microsoft made, says security researcher Claire Tills. “I know some defenders also really had trouble with that.”

More recently, in 2022, Microsoft took yet another step towards making Patch Tuesday a regular Tuesday, when it announced Autopatch, which promised to ease the process of addressing vulns for customers with Windows Enterprise E3 and E5 licenses. The move made organizations wonder whether Patch Tuesday as we know it might disappear, a rumor Microsoft denied.

The publicity around Patch Tuesday is getting smaller, and the number of Patch Tuesday vulnerabilities may have peaked. In 2020, a particularly “aggressive year,” Tills counted around 1,200, while in 2022, she saw 663.

“In terms of the types of vulnerabilities, it’s consistently elevation of privilege and remote code execution,” she says. “Every once in a while, we’ll get peaks in information disclosures and security feature bypass — those two also pop up.”

But despite the features meant to streamline the process of applying patches, this task can still be daunting for small businesses who can’t afford a dedicated tech support team. It is why Gupta recommends these clients to move to the cloud.

“Then you can purely focus on your business and you don’t have to worry about patching and managing systems,” she says. “I also encourage people to not disable the default upgrades.”

But, as we transition toward automating the processes, is dedicating one day for updates obsolete?

Is Patch Tuesday Becoming Obsolete?

It is hard to fathom a world of cybersecurity without Patch Tuesday, which has been an integral part of the industry for two decades. As threats become more sophisticated and geopolitics becomes more volatile, however, the traditional model of Patch Tuesday may no longer be sufficient to keep systems secure.

Organizations operating in complex environments, such as in those competitive fields or based in hot areas like Ukraine, cannot afford to make mistakes when it comes to patch management. Threat actors often “targeted technical vulnerabilities rather than specific individuals or organizations,” says Nazar Tymoshyk, founder & CEO of cybersecurity startup UnderDefense, which has protected several organizations during the ongoing war with Russia.

“The exploitation of unpatched vulnerabilities in technology stacks or outdated Web resources still accounts for 50% of [Russia’s] success in cyber intelligence operations,” he says.

Tymoshyk believes that Patch Tuesday is still necessary and should not be retired. However, organizations should build on top of it, adopting additional patching routines depending on the size and complexity of their infrastructure or the types of software and systems they use.

Satnam Narang, senior staff research engineer at Tenable, also favors keeping Patch Tuesday alive because routines can help organizations foster a security-focused culture.

“Each week, the folks come to pick up the trash from our street, and we know that that’s happening every week on a specific day,” he says. “The reason why Patch Tuesday is a valuable resource is that it happens every month on a specific day, so it allows organizations to carve out the time necessary to patch these vulnerabilities.”

A chaotic patching system might not work, Narang says, because security teams are already overwhelmed. Aviv Grafi, CTO and founder at Votiro, agrees. “Time is of the essence, and we need to focus on leveraging technologies and software that give security teams more time back in their day,” Grafi says.

Security researchers also add that continual patch cycles and automatic updates might not always be feasible at enterprise level in some cases. “In the event of a problem, large organizations need to be able to revert systems to a known state, and that requires planning,” says David Farquhar, solutions architect at Nucleus.

Patch Tuesday is a crucial element in the routines of many organizations, but despite that, it can be unpredictable. The past years have shown that its structure can change without prior warnings. “Even though Patch Tuesday feels so foundational and so solid, it can change at the drop of a hat,” says Tills. “The end of Patch Tuesday could feel disastrous for a lot of organizations.”

For the time being, though, it appears that Microsoft will keep the program running. “I wouldn’t worry about Patch Tuesday going away anytime soon,” Gupta says.

Read More HERE