Technologist Versus Spy: The xz Backdoor Debate

March 30, 2024 TH Author headline,government,cyberwar,spyware,backdoor

Well — we just witnessed one of the most daring infosec capers of my career.

Here’s what we know so far: some time ago, an unknown party evidently noticed that liblzma (aka xz) — a relatively obscure open-source compression library — was a dependency of OpenSSH, a security-critical remote administration tool used to manage millions of servers around the world. This dependency existed not because of a deliberate design decision by the developers of OpenSSH, but because of a kludge added by some Linux distributions to integrate the tool with the operating system’s newfangled orchestration service, systemd.

Equipped with this knowledge about xz, the aforementioned party probably invented the persona of “Jia Tan” — a developer with no prior online footprint who materialized out of the blue in October 2021 and started making helpful contributions to the library. Up to that point, xz had a single maintainer — Lasse Collin — who was dealing with health issues and was falling behind. Shortly after the arrival of “Jia”, several apparent sock puppet accounts showed up and started pressuring Lasse to pass the baton; it seems that he relented at some point in 2023.

Since then, “Jia” diligently continued the maintenance work — culminating in February 2024 with the seamless introduction of a sophisticated, well-concealed backdoor tucked inside one of the build scripts. Full analysis of the payload is still pending, but it appears to have targeted the pre-authentication crypto functions of OpenSSH; it’s probably safe to assume that it added “master key” functionality to let the attackers access all affected servers at will.

Some time after getting the backdoor in, “Jia” — along with a new cast of sock puppet accounts — started pinging Linux distro maintainers to have the backdoored library packaged and distributed to end users. The scheme worked until Andres Freund — a PostgreSQL developer in the employ of Microsoft — reportedly decided to investigate some unexpected SSH latency caused by a minor bug in the backdoor code.

If this timeline is correct, it’s not the modus operandi of a hobbyist. In today’s world, if you have the technical chops and the patience to pull this off, you can easily land a job that would set you for life without risking any prison time. It’s true that we also have some brilliant folks with sociopathic tendencies and poor impulse control — but almost by definition, such “black hat” groups seek instant gratification and don’t plan heists years in advance. In other words, all signs point to this being a professional, for-pay operation — and it wouldn’t be surprising if it was paid for by a state actor.

With attribution up in the air, it’s still tempting to assign blame. Some pundits are pointing fingers at the supposedly exploitative relationship between Big Tech and the open source community; they claim that the lack of adequate compensation is the source of all malaise. I don’t buy this. The relationship with commercial vendors isn’t always healthy, but many major OSS projects are supported to a significant extent. Countless prominent OSS developers are on Big Tech payroll; quite a few projects receive hefty grants.

The real issue with a lot of small, foundational OSS libraries is just that there isn’t enough to do. They were written decades ago by a single person — and beyond bugfixes, they are not really supposed to change much. You don’t do major facelifts of zlib or giflib every year; even if you wave some cash around, it’s hard to build a sustainable community around watching paint dry. After a while, the maintainer just isn’t all that into it anymore; they are eager to pass the baton to anyone with a pulse and some modicum of skill.

Heck, the same happens on the other side of the equation: even with Big Tech staffing and money, if you have a library that almost never needs any attention, the “ownership” of that code becomes pretty theoretical too. It’s hard to build a rewarding career on being very familiar with some boring, old dependency that’s just taken for granted by everyone else.

More fundamentally, the xz backdoor isn’t a technical problem and it probably can’t be solved with technology alone. It’s ultimately a counterintelligence challenge — squarely within the competencies of governments and a handful of commercial entities with ecosystem-wide surveillance capabilities. This notably includes Google and Microsoft.

In fact, here’s an interesting thought: perhaps they have known for a while. Would we be able to tell the difference between a carefully-timed disclosure — presumably engineered to conceal “methods and sources” — and a serendipitous discovery?