Google: Linux kernel and its toolchains are underinvested by at least 100 engineers

Google’s open security team has claimed the Linux kernel code is not good enough, with nearly 100 new fixes every week, and that at least 100 more engineers are needed to work on it.

Kees Cook, a Google software engineer who has devoted much of his time to security features in the Linux kernel, has posted about continuing problems in the kernel which he said have insufficient focus.

“The stable kernel releases (‘bug fixes only’) each contain close to 100 new fixes per week,” he said. This puts pressure on Linux vendors – including those who support the countless products which run Linux – to “ignore all the fixes, pick out only ‘important’ fixes, or face the daunting task of taking everything,” he said.

Cook partly blames the C programming language. “With Linux written in C, it will continue to have a long tail of associated problems,” he said. He added that the Mitre CVE (Common Vulnerabilities and Exposures) list, used by professionals to assess the importance of bugs, is not up to the task since “not all security flaws have CVEs assigned, nor are they assigned in a timely manner.”

The only solution is to continually update to the latest version of the stable release used, but Cook said that “performing continuous kernel updates… faces enormous resistance within an organization due to fear of regressions – will the update break the product?” Another issue is that many vendors use old kernels and backport the fixes, which means redundant work as multiple engineers at different companies fix the same problem.

Cook references Google’s fuzzing tool, Syzkaller, which is currently reporting nearly 1,000 possible issues in the Linux kernel: about 400 a year are fixed, he said, but the number is growing by 100 per year as new ones are found.

What is the solution? Cook has a number of proposals, including moving away from the email-only workflow used for Linux development, introducing more automated testing and fuzzing, continuous integration, and other steps to make the development process “more efficient.” Currently too much kernel testing happens after a version is released, he said.

Cook also proposed improving the Linux toolchain, not least with making sure “Linux can be written in memory-safe languages like Rust.”

According to Cook, “based on our most conservative estimates, the Linux kernel and its toolchains are currently underinvested by at least 100 engineers.” He suggested that companies move in-house engineers working on kernel code and security to work on the upstream kernel instead. “This is the only solution that will ensure a balance of security at reasonable long-term cost.”

Reasonable long-term cost? Linux, which is a free operating system, largely powers many of the world’s most profitable companies, not least Google itself whose parent company Alphabet reported $19.36bn operating profit in its quarter ending 30 June. The company could employ an additional 100 Linux security engineers without blinking, as could Amazon, which likewise runs mostly on Linux and reported revenue for its last quarter of $113.1bn.

In February this year, Google said it was sponsoring two full-time developers to work on upstream kernel security. ®

READ MORE HERE