Linux, the most widely used open source operating system in the world, narrowly escaped a massive cyber attack over Easter weekend, all thanks to one volunteer.<\/p>\n<\/div>\n
The backdoor had been inserted into a recent release of a Linux compression format called XZ Utils, a tool that is little-known outside the Linux world but is used in nearly every Linux distribution to compresses large files, making them easier to transfer. If it had spread more widely, an untold number of systems could have been left compromised for years.<\/p>\n<\/div>\n
And as Ars Technica<\/em> noted in its exhaustive recap<\/a>, the culprit had been working on the project out in the open.<\/p>\n<\/div>\n The vulnerability, inserted into Linux\u2019s remote log-in, only exposed itself to a single key, so that it could hide from scans of public computers. As Ben Thompson writes in Stratechery<\/em><\/a>.\u00a0 \u201cthe majority of the world\u2019s computers would be vulnerable and no one would know.\u201d<\/p>\n<\/div>\n The story of the XZ backdoor\u2019s discovery starts in the early morning of March 29th, as San Francisco-based Microsoft developer Andres Freund posted on Mastodon and sent an email<\/a> to OpenWall\u2019s security mailing list with the heading: \u201cbackdoor in upstream xz\/liblzma leading to ssh server compromise.\u201d <\/p>\n<\/div>\n Freund, who volunteers as a \u201cmaintainer\u201d for PostgreSQL, a Linux-based database, noticed a few strange things over the past few weeks while running tests. Encrypted log-ins to liblzma, part of the XZ compression library, were using up a ton of CPU. None of the performance tools he used revealed anything, Freund wrote on Mastodon. This immediately made him suspicious, and he remembered an \u201codd complaint\u201d from a Postgres user a couple of weeks earlier about Valgrind, Linux\u2019s program that checks for memory errors.\u00a0<\/p>\n<\/div>\n After some sleuthing, Freund eventually discovered what was wrong. \u201cThe upstream xz repository and the xz tarballs have been backdoored,\u201d noted Freund in his email. The malicious code was in versions \u200b\u200b5.6.0 and 5.6.1 of the xz tools and libraries.\u00a0<\/p>\n<\/div>\n Shortly after, enterprise opensource software company Red Hat sent out an emergency security alert<\/a> for users of Fedora Rawhide and Fedora Linux 40. Ultimately, the company concluded that the beta version of Fedora Linux 40 contained two affected versions of the xz libraries. Fedora Rawhide versions likely received versions 5.6.0 or 5.6.1 as well. <\/p>\n<\/div>\n PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA RAWHIDE INSTANCES for work or personal activity. Fedora Rawhide will be reverted to xz-5.4.x shortly, and once that is done, Fedora Rawhide instances can safely be redeployed.<\/p>\n<\/blockquote>\n<\/div>\n Although a beta version of Debian, the free Linux distribution, contained compromised packages, its security team acted swiftly<\/a> to revert them. \u201cRight now no Debian stable versions are known to be affected,\u201d wrote Debian\u2019s Salvatore Bonaccorso in a security alert to users on Friday evening. <\/p>\n<\/div>\n Freund later identified the person who submitted the malicious code as one of two main xz Utils developers, known as JiaT75, or Jia Tan. \u201cGiven the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the \u201cfixes\u201d mentioned above,\u201d wrote Freund in his analysis<\/a>, after linking several workarounds that were made by JiaT75. <\/p>\n<\/div>\n JiaT75 was a familiar name: they\u2019d worked side-by-side with the original developer of .xz file format, Lasse Collin, for a while. As programmer Russ Cox noted in his timeline<\/a>, JiaT75 started by sending apparently legitimate patches to the XZ mailing list in October of 2021. <\/p>\n<\/div>\n Other arms of the scheme unfolded a few months later, as two other identities, Jigar Kumar and Dennis Ens, began emailing complaints<\/a> to Collin about bugs and the project\u2019s slow development. However, as noted in reports by Evan Boehs<\/a> and others, \u201cKumar\u201d and \u201cEns\u201d were never seen outside the XZ community, leading investigators to believe both are fakes that existed only to help Jia Tan get into position to deliver the backdoored code.<\/p>\n<\/div>\n \u201cI am sorry about your mental health issues, but its important to be aware of your own limits. I get that this is a hobby project for all contributors, but the community desires more,\u201d wrote Ens in one message, while Kumar said in another that \u201cProgress will not happen until there is new maintainer.\u201d<\/p>\n<\/div>\n In the midst of this back and forth, Collins wrote that \u201cI haven\u2019t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things,\u201d and suggested Jia Tan would take on a bigger role. \u201cIt\u2019s also good to keep in mind that this is an unpaid hobby project,\u201d he concluded. The emails from \u201cKumar\u201d and \u201cEns\u201d continued until Tan was added as a maintainer later that year, able to make alterations, and attempt to get the backdoored package into Linux distributions with more authority.<\/p>\n<\/div>\n The xz backdoor incident and its aftermath are an example of both the beauty of open source and a striking vulnerability in the internet\u2019s infrastructure.<\/p>\n<\/div>\n\n