Lessons from Exherbo and Gentoo

I contributed to both Exherbo and Gentoo Linux, and upstreamed many musl and clang patches while doing it. Here's why, and what I learned.

The "Ricer" Backstory

I started using Linux in freshman year of highschool. I had my old 2012 Macbook Pro at the time. I switched for several reasons. I was concerned about Apple spying on me. I strongly believed software shouldn't cost money. I wanted to learn more about how operating systems work.

So I whipped Debian Sid on a full-disk-encryption disk setup. I went with the full GNOME 3 setup. I even customized my background. I installed packages via the command line like TLP.

But there were pain points. Apt was slow. I had random daemons like CUPS running that I didn't want. I wanted to close all the ports. I felt like my install was bloated

I then moved on to Arch Linux, tiling window managers, and the obligitory ricing stage. But I went a few steps farther. I hated the ports tree, and I disliked makepkg. I wanted to recompile all packages with optimization flags, and I wanted to use clang. I gave up, and moved onto Gentoo.

Then Gentoo wasn't enough simplicity. OpenRC seemed monsterous. glibc, libstdc++, and gcc were still there. I went on to use a musl libc form of Gentoo, and even went on to create a self-hosted clang version of gentoo-musl which can be found here (thanks to staalmannen, who I've worked with to make this happen).

Near the end of this whole process, I realized that Gentoo itself was too big and unmanageable to support these changes. It's build infrastructure is mostly a hack, and the portage package manager is a disaster. It was time for me to join another software project that I could more easily implement my vision for: Exherbo Linux.

Working with Exherbo

At this point in my Linux career, I had strong opinions, from a software quality and architecture standpoint. Systemd was bad, so were most GNU tools. Exherbo at this time had bitrotted musl libc support. So, I set up a work environment, and started bootstrapping a tarball for Exherbo on top of musl libc. Thanks to this documentation by somasis, it was a relatively fun experience. A few musl patches later, the whole base package set could be built with no issues.

I wrote more and more packages, including tools I used myself like the vis editor. I went on to add elogind support for GNOME to stand a chance running on musl (though there were too many patches and problems to make it work in the end). But I eventually saw how these changes I were making to support musl libc were improving Exherbo as a whole, updating its whole stack from technologies that had never been used.

Then, I started on my own weird technology changes. I created s6-exherbo, a way to bring the s6 init system to exherbo along with service management. This was work that only existed to fill my dreams of what software stack perfection looked like: s6 is a technology that is practically non-existant from the real world. But, motivated by a quest to finally make a reasonable open-source software distribution, I kept going.

At this point, I started to realize the Exherbo project was more than my guinea pig. Exherbo was a real developers OS, and consisted of devs working on relatively common ground. Most of the Exherbo devs assumed the very technologies I was trying to displace: systemd, glibc, and openssl. Most of the amazing work in keeping KDE up to date, as well as Rust and many other packages, relied on systemd, and most likely would not be able to support other init systems well, if at all.

Before giving up entirely on my quest for software perfection, I updated the packages for LLVM to make it easier to install multiple versions on the same system. This was the last step to enable the system-wide default compiler to be clang instead of gcc.

However, instead of moving on to creating a fully self-hosted LLVM/libc++/clang tarball like I did with Gentoo, something within me was stirring, telling me that it just wasn't worth it. The amount of energy required weekly to make sure software wasn't breaking under musl was already significant: how much clang patching on top of that could I possibly handle? Why was I spending most of my time as a developer writing workarounds for other people's crappy code?

The Moral

The Linux kernel is highly successful software. It is ubiquitous, and has daily contribution to it from talent across the world. It is also full of bugs, security issues, and legacy cruft.

The seL4 microkernel is highly unsuccessful software. It is used in specific domain specific areas, mostly closed off to the outside world. Even if it is mathematically devoid of bugs, security issues, and legacy cruft.

We can make this same observation in pair between similar software projects: systemd, s6; glibc, musl; GNU coreutils, toybox.

Why is this pattern the same? Why does the world prefer crap to sophisticated, beautiful works of software?

It's because there's no demand in the world for perfect software. The world just wants something that works. Developers aren't supposed to write perfect software. Perfect software doesn't exist. It exists only in dreams.

A software project is only successful if many developers work together on it. It doesn't matter if you have the coolest program or OS in the world. It doesn't matter if its mathematically verified. Doesn't matter if it is written in pure C. If you're the only full time developer of it, it probably isn't a very successful project.

Software engineers use technology like Linux and distributions that run on GNU stacks because it's what everyone else does. It's what the good software engineer does: understand widespread software to enable collaboration with coworkers and other developers.

We could strive for perfection, say, by writing a web app as a microkernel application running exclusively over 9P/IL connections. But that's not what software engineers do. Thats what professors and proof engineers do.

Software engineers are supposed to hack complex frameworks and technologies and services together into something that works. Useful software is supposed to be hopelessly complex, because we cant possbly make everything from scratch. It's all we can do.

Is this the end?

But what about security?

But what about great performance?

But what about avoiding dependency hell?

It doesn't hurt to be optimistic. We would all like our software to work perfectly. Our security, cpu cycles, and sanity depends on it.

But we should remember there's only so much a single developer can do.

Here's what we should try to remember.

We should strive to write perfect software: each and every one of us. It gets easier every day. But we should never dispair when ourselves, or anyone else, isn't able to write perfect software. In the end, no one can create perfection, and what matters is that we have something that works.

Then we should get up from our chair, sigh a sigh of relief, and take a break to go outside to watch the sunshine while our code is, somehow, still compiling.