‘We’re finding bugs way faster than we can fix them’: Google sponsors 2 full-time devs to improve Linux security

Interview Worried about the security of Linux and open-source code, Google is sponsoring a pair of full-time developers to work on the kernel’s security.

The internet giant builds code from its own repositories rather than downloading outside binaries, though given the pace at which code is being added to Linux, this task is non-trivial. Google’s open-source security team lead Dan Lorenc spoke to The Register about its approach, and why it will not use pre-built binaries despite their convenience.

But first: the two individuals full-time sponsored by Google are Gustavo Silva, whose work includes eliminating some classes of buffer overflow risks and on kernel self-protection, and Nathan Chancellor, who fixes bugs in the Clang/LLVM compilers and improves compiler warnings.

Both are already working at the Linux Foundation, so what is new? “Gustavo’s been working on the Linux kernel at the Linux Foundation for several years now,” Lorenc tells us. “We’ve actually been sponsoring it within the Foundation for a number of years. The main change is that we’re trying to talk about it more, to encourage other companies to participate. It’s a model that works, we’re trying to expand it, find contributors that want to turn this into a full-time thing, and giving them the funding to do that.”

It is in the nature of open source that Google’s funding benefits other Linux users, and it is also in the company’s interests. How important is Linux to Google? “It’s absolutely critical. Google started on Linux. We use it everywhere,” says Lorenc.

That being the case, why can Google only manage “Gold” membership of the Linux Foundation ($100,000 per annum), whereas others including Microsoft, Intel, Facebook, and Red Hat are “Platinum”, which contributes $500,000 annually? “I’m not sure about that stuff. There are dozens of sub-foundations which we are also members of,” he adds. Google is ahead of AWS, which is a mere “Silver” member ($20,000 a year).

Lorenc explains some of the steps Google takes to ensure the security of open-source code that it uses internally, including Linux. “One of the things that we try to do for any open source that we use, and something we recommend anybody uses, is being able to build it yourself. It is not always easy or trivial to build, but knowing that you can is half the battle, in case you ever need to.

We require that all open source we use is built by us, from our internal repositories

“We require that all open source we use is built by us, from our internal repositories, just to prove that we can, if we ever need to make a patch, and so that we have better provenance, knowing where it is coming from. They are technically forks, but just a copy of the [public] repo that we keep up to date. We rarely carry patches long-term in any of the projects we work on, it is just a maintenance nightmare, but we can if we need to.

“For the most part we build our own Linux kernels, but that’s the Linux model. For Linux it is not strange to be doing it this way, it is strange in a number of other projects where we do it.”

The consequence is that Google loses the convenience most Linux users enjoy, downloading a binary image for a Linux distribution and installing what is needed via a package manager.

What are the challenges in keeping Linux secure? “It’s way better than many projects,” says Lorenc. “They have a well-documented process with tiers of reviewers. The challenge is that the Linux kernel is huge, it’s software and all software has bugs, the more software there is the more bugs there are. The other thing is we’ve gotten very good at finding bugs, static analysis has come a long way, fuzzing has come along way, and we’re finding bugs way faster than we can fix them. The next challenge is finding ways to fix them.

“We don’t want to stop finding them, but if you’re finding bugs that nobody has time to look at, you’re not really solving the whole problem.”

This then is part of the rationale for getting “more hands on keyboard”, but sponsoring two developers who already work on this is not close to a solution.

If you’re finding bugs that nobody has time to look at, you’re not really solving the whole problem

It is still worthwhile, says Lorenc, as there are not many such people. “We did an announcement last week around the Python interpreter. We added one full-time person working on CPython for the next year. Best estimate is that is a 50 per cent increase in the number of people working on the Python interpreter full-time.” The situation with full-time developers working on Linux security may be similar, he says.

With Linux Torvalds regularly commenting on the large amount of new code in kernel releases, how can this be addressed? “We are going to have to get creative,” Lorenc tells The Reg. “Our basic approach to security at scale across Google and in the rest of the industry is to try to engineer away entire classes of problems. We do have to fix the bugs we find, but at the same time think about ways to fix entire classes of bugs.”

Examples would be the buffer overflow problem that has long afflicted software written in system languages like C, used for the Linux kernel. Switch language? “We know we have a whole new set of programming languages now, like Rust and Go and Swift, that operate in completely new ways,” he says. “By using these languages you’ve engineered away entire classes of memory safety bugs.”

Microsoft has had similar thoughts with regard to Windows, but change is not easy. Is Google working on using Rust for kernel code, or rewriting code in Rust? “We’re thinking about it,” Lorenc reveals. “The last update was at the Linux Plumber’s conference, there was a session on what it would take. That was run by a Google engineer from the Android team… it’s not far enough along to know if it’s going to work or not.”

Given the scale of the challenge and the large flow of new kernel code, is the security aspect winning? Lorenc says: “I don’t think we’re losing. We’re making a lot of great progress. Each release is getting bigger but we’re getting better with each release too.”

Not losing, then – but not winning either. ®

READ MORE HERE