How MuleSoft patched a critical security flaw and avoided a disaster

Image: MuleSoft

John is a software engineer and a pretty good one. He works for a company that processes online payments. On Thursday, August 1, John’s bosses pulled him into an urgent security meeting.

John was scared but also very curious. What could have happened? The last time John was called into a security meeting was in 2017, more than two years ago, during the three ransomware outbreaks that occurred that year — WannaCry, NotPetya, and Bad Rabbit.

A few months before, Microsoft disclosed a major security flaw impacting the Windows OS, named BlueKeep, and his company barely reacted, merely sending an internal security alert, telling software engineers to review RDP access settings on Windows systems.

Sitting in a conference room and waiting for execs and engineers to take their seats, he couldn’t contain his curiosity. If they didn’t react to BlueKeep, what were they reacting to now? What could have caused such a panic and reaction at the company?

But then, the meeting started, and John found that all the ruckus was because of MuleSoft. He snickered. Two seconds later, after he realized what this meant, he stopped thinking it was funny.

MuleSoft, a company now owned by Salesforce, makes middleware. Middleware is what engineers call “software glue.” You can find it in all big companies across the world.

Middleware can be used to link cloud apps distributed across tens, hundreds, or thousands of servers, but it can also be used on smaller networks, to translate data as it moves between apps that work in different formats. Everybody uses middleware. Everybody!

The secret MuleSoft notification emails

As he sat through the meeting, John learned about a security vulnerability in MuleSoft’s Mule runtime and API gateways, two of the company’s most popular products.

And he wasn’t the only one finding out about this security flaw. Countless of other engineers across the world were being pulled in similar meetings or having phone calls with MuleSoft’s security team.

A day before, MuleSoft had sent out an email to a selected list of customers. It urged companies that ran on-premise Mule engines to install the latest patches, released on the same day.

The email, obtained by ZDNet and embedded below, also urged companies to schedule an urgent call with MuleSoft’s staff so they could learn details about a security flaw that had been secretly patched a day before.

The company was planning to release details to the public, but in a month, so they could give companies a head start in patching on-premise systems.

The companies running on-premise systems were processing data so sensitive that it just couldn’t be uploaded into a public cloud, for security, privacy, and compliance reasons. The MuleSoft homepage lists an assortment of banks, payments processors, and cloud providers that would most likely be running on-premise systems, rather than rely on Mule engines and API gatewars provided by Salesforce and MuleSoft’s cloud services (which had been patched before the email was even sent).

From the content of the email, one could see that MuleSoft was taking the security bug very seriously. In a rare step, MuleSoft had asked the recipients of the emails not to share the security alert’s content with anybody, not even verbally. The company described the vulnerability’s existence as a “need to know” issue.

Obviously, the email leaked. It leaked on Twitter, in Slack and Discord channels, and on Telegram groups. It’s easy to see why it got everyone’s attention. What could have been so bad that MuleSoft was describing as a “need to know” issue?

The directory traversal bug

Several people downloaded the patches and analyzed their content. They found fixes in MuleSoft’s code specific with a directory (or path) traversal bug.

This kind of bug can allow a malicious attacker to upload and plant files on a system in unexpected system locations. If the attacker can fine-tune the attack, he can control the places where the malicious files can end up.

There are several locations on a Windows or Linux system where the uploaded files could be executed automatically, leading to a situation where the attacker could run malicious code and take over vulnerable servers entirely.

Since some middleware sits right behind web servers, they effectively sit exposed on the internet, with web server passing external input to the middleware automatically, to be transported to internal APIs, databases, or data processing systems.

It was a very dangerous bug, and MuleSoft knew it.

When ZDNet reached out to MuleSoft for comment, we were almost immediately pulled into a phone conference with MuleSoft Chief Technical Officer Uri Sarid and Salesforce Chief Trust Officer Jim Alkove within the hour, on a late Friday night, when most people had gone home.

They had a well-oiled machine running by that point, and some nosy reporter was about to ruin everything. Sarid and Alkove were afraid that a news article would bring unwanted attention to their company’s security flaw and could lead to attacks on some of their customers.

But instead of denying that anything was wrong, they took the time to explain the complex system they had in place to deal with this vulnerability, and by that point, it would have been irresponsible on ZDNet‘s part to publish.

A novel approach to notifying customers

MuleSoft had put maximum effort behind this issue. The company had put its considerable customer support force on the issue and was intent on notifying each and every one of its customers running on-premise systems, no matter what.

Representatives were ordered to call each company in part. Everyone running an on-site Mule engine or API Gateway was getting a call to check if they received and read the email.

Furthermore, Sarid said that MuleSoft had taken an unprecedented step of seeking out and talking to each company’s security and DevOps departments, and not just secretaries or sales representatives.

They were taking this security flaw very seriously. They wanted their message to reach the proper person in each organization, and they wanted to make sure companies installed the patches.

But they didn’t stop here. MuleSoft also scheduled a second wave of calls after companies installed the patches, verifying that customers followed through, and passing on additional mitigation advice.

And the MuleSoft and Salesforce execs weren’t exaggerating. What they told ZDNet in a first call in early August is what we saw being discussed in various online discussion boards, where numerous programmers were describing similar phone calls and security meetings.

“We literally had thousands of calls with these customers, and the interesting thing is that the customer feedback you might imagine would be worrisome, but it’s actually been overwhelmingly positive,” Sarid told ZDNet in a follow-up phone call this Friday.

“Numerous customers thanked us for taking an unusually proactive approach to actually call and speak to them about the issue. They hadn’t seen vendors do that before,” he said.

Other companies need to take similar approaches

One month after setting off this unique vulnerability disclosure process in motion, MuleSoft has now gone public. Details about this security issue, as promised in its private email notification, have been published on the company’s site. MITRE is also in the process of allocating a CVE (security bug identifier) to this vulnerability, ZDNet understands.

But while MuleSoft customers have patched, Sarid now hopes that other companies learn from MuleSoft’s experience, and take a similarly proactive approach to notifying customers of critical issues, instead of deferring all responsibility to their clients.

“You don’t wanna handle everything in the same way,” Sarid told ZDNet. “The numerous smaller vulnerabilities can be addressed in the standard way, in Patch Tuesdays.

“But the ones that call for urgent action that are being disclosed before being publicly disclosed, for those ones there aren’t any standard procedures that companies can follow.”

Sarid hopes that other companies follow MuleSoft’s approach when dealing with a major security flaw, instead of dumping a security advisory on a support page buried somewhere on their websites, where very few people know to look, without bothering to notify customers via something as basic as an email, let alone a phone.

MuleSoft vs Fortinet & Pulse Secure disclosure fiasco

What Sarid is proposing makes a lot of sense, but something that’s almost never done in the real world.

However, fate gave Sarid and MuleSoft a helping hand, proving what happens when companies take a lackadaisical approach to notifying customers of critical vulnerabilities.

Earlier this year, security researchers found several security flaws in many enterprise VPN products. Products from Fortinet and Pulse Secure were impacted by these flaws. The companies had done their job and patched these issues and then released security advisories on their sites.

The problem is that most customers weren’t aware of these updates, and the fact that they contained fixes for major security flaws. When hackers began exploiting these vulnerabilities in mid-August, most companies using these VPN products were caught unprepared.

Now imagine if these companies had received a phone call from Fortinet or Pulse Secure, similar to how MuleSoft approached dealing with its security issue?

The MuleSoft CTO knows exactly how Fortinet and Pulse Secure customers would have reacted because his company experienced it first hand.

“When you’re talking to the appropriate security person, and you’re telling them more information that only the security person understands, and handles that, they turn around and say ‘thank you!’,” Sarid told ZDNet.

“So it turns this potentially very negative interaction into something very positive, which hopefully is an incentive to other companies, for other vendors like us, to pick this kind of action.”

READ MORE HERE