A quick look at the history of operating systems
Operating systems (OSes) represent a very large topic and for decades they have been dominated by a single approach: Unix. Indeed, most of modern systems including the majority of GNU/Linux distributions, the *BSDs, and macOS follow the Unix design. (Windows does not, but it does not have many interesting features for the matter at hand.)
In 2000, Rob Pike gave a talk on why Systems Software Research is Irrelevant. Whether out of pessimism or disregard for the community, it seems to completely discard the far cries that many Unix users had compiled in 1994 in The Unix-Haters Handbook. While purposefully sarcastic, this book pointed out some critical issues with Unix systems that still today haven’t really been addressed.
In 2006, Eelco Dolstra published his thesis, The Purely Functional Software Deployment Model, which introduces Nix, a functional package manager. In 2008, he published NixOS: A Purely Functional Linux Distribution. While NixOS reuses a lot of free software that were traditionally meant for Unix-style systems, it departs so much from the Unix design and philosophy that it can hardly be referred to as a “Unix” system anymore.
Nix is a giant step forward in operating system research. Not only does it address most of the criticism of Unix (including those found in the Unix Haters Handbook), it also paves the way for many more features and research explorations that could be critical in this day and age where topics like reliability and trust are at the center of many scientific, but also social and political debates.
Pike was wrong. And this proves another more general point: it’s probably wiser to refrain from claiming that any research has become irrelevant, unless you can prove that no further development is possible. And the Utah talk was hardly a mathematical proof. It only served to further entrench the idea that Unix is “good enough” and that we’ve got to live on with its idiosyncrasies and issues.
This unnecessary pessimism was thankfully short-sighted and would not live long before Nix proved it wrong only a couple of years later.
Guix is a package manager inspired by Nix and Guix System is the operating system equivalent of NixOS. Guix System strives to be the “fully-programmable OS”. Indeed, mostly everything from package management (with Guix) to the init system (GNU shepherd) is written and customizable in Guile Scheme.
It departs significantly from Unix-style operating systems and you might want to read some more before you get a good grasp of the extent of the changes:
The perks of Guix
Guix’ advance is not to underestimate as it is a real game changer to the point that it obsoletes most other operating systems.
My personal favourite features:
- Unbreakable systems: Guix maintains the history of all changes both at the system and user level. Should an update break anything, it’s always possible to roll back. This effectively makes the system unbreakable.
- System integrity: Because the system configuration is declarative, this gives the user or the system administrator a complete guarantee of what’s going on. On other Unices, it’s much harder to tell when some random configuration file has been modified.
- Fully programmable operating system: Program your system configurations and keep them under version control. Many system services can be configured in Guile Scheme, from udev rules to Xorg, PAM, etc. Thanks to Guile, the configuration can be made conditional to the hardware or even the hostname!
- Drop-in replacement of other (not as good) package managers: No need to manage Emacs, Python or TeXlive packages separately, you can use one uniform interface to manage them all! (See below.) It makes user-profile declarations far easier to write and maintain.
- Package definitions using Guile: it’s much more efficient to work out package (re)definitions en masse. It advantageously replaces concepts such as Portage’s USE flags (see below).
- Multiple package outputs: A Guix package can have multiple “outputs” that serves as a mean to separate various component of a program (libraries, extra tools, documentation, etc.). On other operating systems, (most typically Debian), it can be harder to guess which Debian packages belong together.
- Non-propagated inputs: “Inputs,” in Guix parlance, are the package
dependencies. The user profile and environment only contains the packages the
user explicitly installed and not necessarily the dependencies of those
packages. See inxi for instance, a system information reporting tool: if I
only care about inxi system/hardware reports, I don’t necessarily
want 2 or 3 dozens of additional commandline tools added to my
PATH. Guix makes it possible to only expose what the user really wants into the user profile.
- Guix environments: When running
guix environment SOME-PACKAGES, Guix sets up a temporary environment where all the requirements for
SOME-PACKAGESare exposed. This can be used to easily set up a build environment for a project, but it can also be used in multiple ways (see below). One neat property is that it allows to run programs without installing them to the user profile.
- Partial upgrades: They are 100% supported. This is possibly the main cause of breakages on rolling-release systems like Arch Linux and Gentoo: because only a few versions are supported at a time (mostly just one), the whole system has to be updated together. Which means more bandwidth usage on every upgrade. With Guix, it’s perfectly possible to update any package individually.
“Continuous integration” or “Why Guix can work without package maintainers”: Thanks to reproducible builds and partial upgrade support, once a package works in Guix it works “forever”, it won’t break on the next upgrade of some dependency. (More precisely, if a dependency breaks a package, it’s trivial to fix it to use the right library version.) This means that the packaging workload can be transferred to build farms (one running Hydra –from the Nix project–, another running Cuirass). Contrast this to most other GNU/Linux communities which need a couple dozen maintainers to keep thousands of packages up to date. This does not scale: eventually those operating systems stagnate at a couple thousand packages. With Guix, the package count can safely keep growing without fear of bankruptcy. In the meantime, the contributors’ time can be put to better use.
With Guix, it’s just as straightforward to build from source and to install a pre-built package directly. In fact, the distinction is not so important to the end-user: Guix can transparently fall back on building from source if no pre-built package is available.
guix refresh: Generate or update package definitions automatically and recursively. Hundreds of package definitions can be processed all at once. Features like these highlight the benefits of having a real programming language at hand. What is a hard problem on most operating systems becomes relatively easy to implement with Guix.
- Guix channels: One of my favourite features! On Arch Linux or Gentoo, one would have to set up a local repository. Because they don’t support partial upgrades, it means that the user has to do some maintenance once in a while (i.e. make sure dependency upgrades don’t break the user’s packages.) Package inheritance makes it very easy to customize packages with a patch, for instance. Guix channels advantageously replace Arch Linux’ AUR and Gentoo’s overlays, allowing anyone to distribute their package definitions from, for instance, Git repositories. Again, this guarantees full transparency (roll backs, history, etc.).
- Emacs-Guix: Guix is the only distribution I know which comes with a most powerful Emacs user interface!
- Guix packs: They allow for an actual viable alternative to containers like Docker. Most container systems suffer from crucial issues: they are not reproducible and are indeed opaque binaries, which is a big no-no for users who cares about trust, security and privacy. On the contrary, Guix packs are fully specified, reproducible and transparent.
guix system vmand
guix system disk-image: Guix makes it trivial to reproduce itself (the whole current system) as a live USB, inside a virtual machine or on a remote machine.
Guix compared to the competition
Debian, Arch Linux, and most others GNU/Linux distributions
GNU/Linux distributions in general lack the aforementioned perks of Guix. Some of the most critical points include:
- Lack of support for multiple package versions, or “dependency hell.” Say, when the latest mpv requires a new ffmpeg, but upgrading ffmpeg breaks most other programs relying on it, we are stuck we dilemma: either we break some packages or we stick to older versions of other packages. Worse, a program could end up not being packaged or supported by the OS at all. This issue is inherent to most distributions which can not provide the guarantee to fulfill their primary objective: package any program.
- Critical need for maintainers. Non-functional package management means that all package must be tested together at all times. This represents a lot of hard work that relies on a committed team of maintainers. In practice, it means that the quality of package management heavily depends on the work force. A distribution without enough maintainers will inevitably suffer, and possibly die. This work force requirement does not scale well and induces a growing complexity (both for the code base and in terms of management) as the number of packages increases.
Gentoo together with all distributions using the Portage package manager are source-based distributions with an iconic feature: the USE flags, which lets the user enable features across the whole system (for instance disable sound, enable graphical interface support, etc.).
USE flags make it trivial to toggle features that were exposed by the packagers (and they have the benefit of being tested). On the other hand, Portage does not let you configure features that were not thought of in advance by the packager. For instance, if a package has optional audio but the packager didn’t expose the corresponding USE flag, the user cannot do anything about it (beside creating a new package definition).
Conversely, Guix gives you full customization for everything, albeit with a bit more Scheme code. Roughly, in pseudo-code:
(loop-over (TARGET-PACKAGES) (package (inherit TARGET) (changes-here... including patches, build options, etc.))
TARGET-PACKAGES with your changes. None of the changes have
to be carved into the package definition. At any time, the user retains full
control over the changes that can be made to the packages.
I’ve loved Gentoo while I was using it, but after moving to Guix it became apparent that Portage’s approach to package customization is more limited.
- The USE flag system does not let you customize unplanned, arbitrary features.
- USE flags add a whole class of complexity (see the rather complex atom semantic) to describe and manage feature relationships across packages. Guix completely nullifies this complexity layer by using Guile Scheme to program the relationships between packages.
Moreover, Portage suffer from the same aforementioned issue with the lack of proper support for multiple versions. Indeed, USE flags significantly increase the magnitude of the issue (a frequent complaint about Portage): when incompatible USE flags propagate to some dependencies, the user must find a resolution manually. Sometimes, it means that the desired feature won’t be applicable (at least not without significant work on the package definitions).
On the practical level, Guix provides pre-compiled packages which can be a huge time-saver compared to Gentoo which does not (but Portage support binary package distribution).
The *BSD systems (e.g. FreeBSD) suffer from similar issues with the
config customization interface.
Nix was a historical breakthrough in operating system research and Guix owes it almost all its ideas. Today, Nix still remains one of the finest operating systems in activity and Guix would probably not exist if it weren’t for one flaw.
Guix, heavily inspired by Nix, borrows most of its ideas and addresses the main issue that, in my opinion, Nix didn’t get right: instead of coming up with a homebrewed domain-specific language (DSL), Guix uses a full-fledged programming language. And a good one at that, since it’s Guile Scheme, a Lisp-based language.
“Rolling out one’s own programming language” is a very common fallacy in software development. It struck hard many projects where the configuration or programming language was:
- limited in expressiveness and features;
- yet another language to learn (but not something very useful and reusable) which requires some effort on the user end and thus creates an entry barrier;
- harder to read (at least at the beginning);
- and often with poor performance.
Such projects using self-rolled DSLs or too-limiting programming languages are legions:
- XML, HTML (better idea: S-XML)
- Make, Autoconf, Automake, CMake, etc.
- Bash, Zsh, Fish (better ideas: Eshell or scsh)
- JSON, TOML, YAML
- Nix language, Portage’s Ebuild and many other OS package definition syntax rules.
- Firefox when it used XUL (but since then Mozilla has moved on) and most other homebrewed extensibility languages
- Octave, R, PARI/GP, most scientific programs (better ideas: Common Lisp, Racket and other Schemes)
- Regular expressions (better ideas: Emacs’ rx, Racket’s PEG, etc.)
- sed, AWK, etc.
- Most init system configurations, including systemd (better idea: GNU Shepherd)
- cron (better idea: mcron)
- conky (not fully programmable while this is probably the main feature you would expect from such a program)
- TeX, LaTeX (and all the derivatives), Asymptote (better ideas: scribble, skribilo – still in development and as of January 2019 TeX/LaTeX are still used as an intermediary step for PDF output)
- Most programs with configurations that don’t use a general-purpose programming language.
Re-inventing the wheel is generally not a good idea and when it comes to tools as central as programming languages, it has rather dramatic consequences. It creates unnecessary friction and bugs. It scatters communities around. More consolidated communities can be more efficient and make better use of their time improving existing, well designed programming languages.
Not just for the desktop
Guix supports multiple architectures (i686, x8664, ARMv7, and AArch64 as of January 2019) and there are plans to support more kernels beyond Linux. (Say the *BSDs, GNU Hurd, or maybe your own!)
This makes Guix an excellent tool to deploy (reproducible) servers, but also other highly specific systems. I’m thinking embedded systems here in particular, where it could very well compete with OpenWRT. (Although this might require some work before getting there.)
The self-reproducing live USB
guix system disk-image above: It allows for regenerating the
current system on a USB pendrive, for instance.
This makes it rather easy to create a take-away clone of my current system which can be plugged anywhere and replicate my exact computing environment (minus the hardware).
I can include custom data such as my PGP keys and have everything, emails included, readily available straight from boot.
Besides, it’s obviously possible to further clone-install the system on the machine on which I’ve plugged the USB pendrive: instead of being a bare-bones Guix install, it deploys my complete custom operating system.
Replacing other package managers
Emacs, Python, Ruby… And the power of
Guix can replace any package manager, in particular the package managers of programming languages. It has several benefits:
- It brings reproducibility everywhere.
- It allows for rollbacks everywhere.
- It saves you from learning yet another package manager.
At this point I cannot forget to mention
guix environment. This command sets
up a temporary environment with just a specific set of packages, like that of
virtualenv. The killer-feature is that it’s universally available for every
language, or even better, for every language combination, should your project
mix several languages together.
(Disclaimer: As of January 2019, the TeXlive build system of Guix is being revamped.)
I’m honoring TeXlive with a dedicated section since, while also a package manager like the above, it’s exceptionally horrendous :) Which further confirms Guix into its position of a lifesaver!
Most Unix-based operating systems typically package TeXlive as a bunch of bundles, for instance Arch Linux has a dozen of them. Should you need a couple of TeX packages dispatched across different bundles, Arch Linux leaves you little choice but to install the bundles which include thousand of (probably unnecessary) packages. And TeXlive is big, which means that it asks for several hundred megabytes.
An alternative is to install TeXlive manually, but let’s face it,
tlmgr is a
poor package manager and it requires tiresome extra work.
With Guix, it’s possible to install TeXlive packages individually the same way as everything else, which makes it very easy to save your own favourite set of TeXlive packages, or even to create virtual environment specifications to compile some specific documents.
Many operating systems offer only limited support for custom kernels. If the users wish to steer away from the default kernel, the custom kernel has to be maintained manually, which can be a pain.
Gentoo famously “requires” a custom kernel as a recommended (mandatory?) installation step. This is hardly declarative however, and the users have to maintain the kernel’s configuration themselves.
With Guix, the kernel is a fully customizable package like any other. It’s possible to customize everything and pass a custom kernel configuration file to the package definition.
For instance, here follows the definitions of a non-free Linux kernel with the
iwlwifi driver (Warning: I strongly recommend against using non-free drivers
since they pose a serious threat to your privacy and freedom):
(define-module (ambrevar linux-custom) #:use-module (guix gexp) #:use-module (guix packages) #:use-module (guix download) #:use-module (guix git-download) #:use-module (guix build-system trivial) #:use-module ((guix licenses) #:prefix license:) #:use-module (gnu packages linux) #:use-module (srfi srfi-1)) (define-public linux-nonfree (package (inherit linux-libre) (name "linux-nonfree") (version (package-version linux-libre)) (source (origin (method url-fetch) (uri (string-append "https://www.kernel.org/pub/linux/kernel/v4.x/" "linux-" version ".tar.xz")) (sha256 (base32 "1lm2s9yhzyqra1f16jrjwd66m3jl43n5k7av2r9hns8hdr1smmw4")))) (native-inputs `(("kconfig" ,(local-file "./linux-custom.conf")) ,@(alist-delete "kconfig" (package-native-inputs linux-libre)))))) (define (linux-firmware-version) "9d40a17beaf271e6ad47a5e714a296100eef4692") (define (linux-firmware-source version) (origin (method git-fetch) (uri (git-reference (url (string-append "https://git.kernel.org/pub/scm/linux/kernel" "/git/firmware/linux-firmware.git")) (commit version))) (file-name (string-append "linux-firmware-" version "-checkout")) (sha256 (base32 "099kll2n1zvps5qawnbm6c75khgn81j8ns0widiw0lnwm8s9q6ch")))) (define-public linux-firmware-iwlwifi (package (name "linux-firmware-iwlwifi") (version (linux-firmware-version)) (source (linux-firmware-source version)) (build-system trivial-build-system) (arguments `(#:modules ((guix build utils)) #:builder (begin (use-modules (guix build utils)) (let ((source (assoc-ref %build-inputs "source")) (fw-dir (string-append %output "/lib/firmware/"))) (mkdir-p fw-dir) (for-each (lambda (file) (copy-file file (string-append fw-dir (basename file)))) (find-files source "iwlwifi-.*\\.ucode$|LICENSE\\.iwlwifi_firmware$")) #t)))) (home-page "https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi") (synopsis "Non-free firmware for Intel wifi chips") (description "Non-free iwlwifi firmware") (license (license:non-copyleft "https://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/LICENCE.iwlwifi_firmware?id=HEAD"))))
The custom kernel and the firmware can be conditionally included in your
current system configuration (some
(define *lspci* (let* ((port (open-pipe* OPEN_READ "lspci")) (str (get-string-all port))) (close-pipe port) str)) (operating-system (host-name "...") ;;... (kernel (cond ((string-match "Network controller: Intel Corporation Wireless 8888" *lspci*) linux-nonfree) (#t linux-libre))) (firmware (append (list linux-firmware-iwlwifi) %base-firmware))
Then run the following to install your new system configuration:
sudo -E guix system reconfigure config.scm
Without even installing the new kernel, you can also directly generate an image ready to be booted from a USB pendrive.
Since Guix packages are cutting edge (e.g. the latest versions of Mesa are readily available) while it allows for complete kernel customization, it can be an ideal platform for gaming and, in particular, packaging games!
Sadly, the video game industry is still far from going along the free software philosophy and very few games are eligible to be packaged as part of the official Guix project.
While Guix stands for free software and won’t accept anything non-free in its repository, quite ironically many advanced features of Guix make it one of the most ideal package managers for proprietary software.
Some of the perks:
guix environmentallows you to run any application in an isolated container that restricts network access, hides the filesystem (no risk that the closed-source program steals any of your files, say your Bitcoin wallet or your PGP keys) and even system-level information such as your user name. This is essential to run any non-trusted, closed-source program.
- Functional package management: closed-source programs usually don’t stand the test of time and break when a library dependency changes its API. Since Guix can define packages over any version of any dependency (without risk of conflict with the current system), Guix makes it possible to create closed-source packages of games that will work forever.
- Reproducible environment: closed-source programs are usually bad at portability and might behave differently on different systems with slightly different dependencies. The reproducibility trait of Guix implies that if we get the Guix package to work once, it will work forever (minus hardware change/breakage).
For all those reasons, Guix would be an ideal tool to package and distribute closed source games.
This a long topic however and we will keep it for another article. In the meantime, check out the Guix Gaming Channels.
Tips and tricks
One amazing perk of Guix is its Emacs interface: Emacs-Guix lets you install and remove packages, selectively upgrade, search, go to the package definition, manage generations, print the “diff” between them, and much more.
It also has development modes for building and programming, as well as a dedicated Scheme REPL.
It’s a very unique power-user interface to the operating system.
There is also Helm System Packages which somewhat overlaps with Emacs-Guix, but I find the interface nicer for quick package lookups and quick operations.
Since Guix allows you to keep several generations of your system configurations (including all of your package history), it can be more demanding on disk usage than other operating systems.
In my experience, in 2018, a 25GB partition for the store forced me to do some clean up about once a month (considering I have heavy packaging requirements), while a 50GB partition would leave me comfortable for a year.
To clean up the store,
guix gc comes in handy, but it can also remove “too
many packages,” that is to say, packages that you would need right away on next
With Emacs-Guix, there is
M-x guix-store-dead-item which lets you sort
dead packages by size and you can delete them individually.
If you need to analyze dependencies, have a look at
guix gc --references and
guix gc --requisites. It can be combined with the result of
guix build ...
to get access to the various elements of the dependency graph.
For instance, to view the code of one of the build scripts, open the file returned by:
$ guix gc --references $(guix build -d coreutils) | grep builder /gnu/store/v02xky6f5rvjywd7ficzi5pyibbmk6cq-coreutils-8.29-guile-builder
Generate a manifest
It is often useful to generate a manifest of all packages installed in some ad-hoc profile.
This can be done with the following Guile script:
;; Run with: ;; guile -s FILE ~/.guix-profile (use-modules (guix profiles) (ice-9 match) (ice-9 pretty-print)) (define (guix-manifest where) (sort (map (lambda (entry) (let ((out (manifest-entry-output entry))) (if (string= out "out") (manifest-entry-name entry) (format #f "~a:~a" (manifest-entry-name entry) (manifest-entry-output entry))))) (manifest-entries (profile-manifest where))) string<?)) ;; Thanks to Ivan Vilata-i-Balaguer for this: (define (guix-commit) (let ((guix-manifest (profile-manifest (string-append (getenv "HOME") "/.config/guix/current")))) (match (assq 'source (manifest-entry-properties (car (manifest-entries guix-manifest)))) (('source ('repository ('version 0) _ _ ('commit commit) _ ...)) commit) (_ #f)))) (match (command-line) ((_ where) (format #t ";; commit: ~a\n" (guix-commit)) (pretty-print `(specifications->manifest ',(guix-manifest where)))) (_ (error "Please provide the path to a Guix profile.")))
Then call it over
~/.guix-profile for instance:
$ guile -s manifest-to-manifest.scm ~/.guix-profile
I like to keep the result under version control in my dotfiles so that I keep track of the history of my installed packages. Since I also store along the version of Guix, I can revert to the exact state of my system as it was at any point in the past.
All this said, a better approach might be to avoid using ad-hoc profiles altogether and prefer manifests.
Service graph for a given OS configuration
guix system extension-graph /path/to/your/config.scm | dot -Tpdf > t.pdf
to view the “service extension graph” for your OS configuration. This is useful to show how the interaction details between services.
Thanks to Ludovic Courtès for this one.
Build a package from the REPL
$ guix repl GNU Guile 3.0.4 Copyright (C) 1995-2020 Free Software Foundation, Inc. Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. This program is free software, and you are welcome to redistribute it under certain conditions; type `,show c' for details. Enter `,help' for help. scheme@(guix-user)> ,use(guix) scheme@(guix-user)> ,use(guix scripts) scheme@(guix-user)> ,use(gnu packages base) scheme@(guix-user)> (build-package coreutils) $1 = #<procedure 7f2170c05540 at guix/scripts.scm:122:2 (state)> scheme@(guix-user)> ,run-in-store (build-package coreutils) /gnu/store/yvsd53rkbvy9q8ak6681hai62nm6rf31-coreutils-8.32-debug /gnu/store/n8awazyldv9hbzb7pjcw76hiifmvrpvd-coreutils-8.32 $2 = #t
Thanks to Ludovic Courtès for this one.
Links and references
Some web interfaces:
Some non-official packages: