Trio of TorchServe flaws means PyTorch users need an urgent upgrade

October 4, 2023 TH Author

A trio of now-patched security issues in TorchServe, an open-source tool for scaling PyTorch machine-learning models in production, could lead to server takeover and remote code execution (RCE), according to security researchers.

The three CVEs, collectively dubbed “ShellTorch,” rendered “tens of thousands of exposed instances” vulnerable, wrote software bill of material management firm Oligo Security’s Idan Levcovich, Guy Kaplan, and Gal Elbaz in a report published on Tuesday.

Meta, which along with Amazon manages the open source TorchServe project, downplayed the flaws and said they’ve been addressed.

“The issues in TorchServe — an optional tool for PyTorch — were patched in August rendering the exploit chain described in this blog post moot,” a Meta spokesperson told The Register. “We encourage developers to use the latest version of TorchServe.”

The Meta spokesperson pointed users to an August 28 update to TorchServe version 0.8.2, which fixed a server-side request forgery (SSFR) flaw and an insecure version of the SnakeYAML, plus two other security advisories a month later (here and here), as well as an October 2 update to GitHub project’s SECURITY.md that shows v 0.8.2 is the only supported release.

Amazon issued its own security bulletin on Monday, which also said TorchServe version 0.8.2, released on August 28, addresses the security issues.

“AWS recommends customers using PyTorch inference Deep Learning Containers (DLC) 1.13.1, 2.0.0, or 2.0.1 in EC2, EKS, or ECS released prior to September 11, 2023, update to TorchServe version 0.8.2,” the security bulletin said. “Customers using PyTorch inference Deep Learning Containers (DLC) through Amazon SageMaker are not affected.”

AWS did not respond to The Register‘s additional questions about the flaws and the Oligo research.

While none of the three companies have seen any indication ShellTorch has been exploited, Oligo co-founder and CEO Nadav Czerninski told The Register that the attack chain does not require technical expertise.

“The vulnerabilities can be easily exploited using basic knowledge of TorchServe and its configuration,” Czerninski said.

ShellTorch CVEs

Specifically, the trio of vulnerabilities include what Oligo describes as an “unauthenticated management interface API misconfiguration.”

The application security startup says the first issue is due to the interface being bound to the IP address 0.0.0.0 by default, instead of localhost, and this default configuration makes it accessible to external requests.

“This door for attackers is not just opened: it’s left unattended, letting everyone in, as the management interface lacks authentication, granting unrestricted access to any user,” the researchers wrote.

The second flaw, CVE-2023-43654, is a remote server-side request forgery bug that can lead to code execution. It earned a CVSS score of 7.2 (from Snyk), or 9.8 (National Vulnerability Database), or 10.0 (GitHub).

Whatever the rating the bug deserves, it’s there thanks to errors in TorchServe’s API that, rather than containing logic for an allow list of domains, accepts all domains as valid URLs.

“This means that an attacker can upload a malicious model that will be executed by the server, which results in arbitrary code execution,” the Oligo researchers wrote.

In addition to being used on its own, this CVE can be used to trigger a third bug: CVE-2022-1471. This one is a SnakeYAML deserialization vulnerability — TorchServe versions 0.3.0 to 0.8.1 use an insecure version of the SnakeYAML v1.31 open source library, which allows for deserialization of Java objects.

Google rated this flaw a 8.3, while NDV analysts said it earned a 9.8 CVSS score. An attacker could exploit CVE-2022-1471 by uploading a model with a malicious YAML file to trigger an unsafe deserialization attack leading to RCE.

“This combination of vulnerabilities allows us to remotely run code with high privileges without any authentication,” Levcovich, Kaplan, and Elbaz said.

In addition to updating to TorchServe 0.8.2, the Oligo team suggests changing the management console from the default settings so that miscreants can’t access it remotely.

They also recommend updating the allowed_urls in the config.properties file to ensure your server only fetches models from trusted domains, such as torchserve.pytorch.org — or whatever is applicable to your organization.

And finally, the security team also released a free tool to check if your organization is vulnerable to ShellTorch. ®