Microchip with the Debian pink

Debian vs. non-free-firmware

Ce billet n’a pas encore été traduit en français. La version anglaise est disponible ci-dessous.

Introduction

The spark

This story starts with the 2022 General Resolution about non-free firmware, with our Social Contract getting an extra sentence in the “Works that do not meet our free software standards” section:

The Debian official media may include firmware that is otherwise not part of the Debian system to enable use of Debian with hardware that requires such firmware.

Debian vs. firmware

Firmware support has always been a major pain point for Debian users (beginners and experts) but it’s grown bigger over time:

  • Laptops were known to be a little problematic, with some wireless network adapters requiring firmware. A usual workaround would be to install using the wired network adapter, adjusting the package manager’s configuration, and installing the required firmware package to get the wireless adapter up and running. With recent laptops, there might be no wired network adapter at all!
  • Nowadays, components that might require firmware include: graphics adapters (even for basic operations, using free Linux kernel modules), wired or wireless network adapters, and even sound cards!
  • Additionally, CPUs are riddled with bugs and one is supposed to apply CPU microcode updates to avoid running into known issues!

Let’s quote the Debian Policy regarding the Debian archive:

The main archive area forms the Debian distribution.

Packages in the other archive areas (contrib, non-free) are not considered to be part of the Debian distribution, although we support their use and provide infrastructure for them.

Unfortunately, firmware packages usually don’t meet the Debian Free Software Guidelines (DFSG), meaning they end up in the non-free area, which means they cannot be included on official media (built using only main).

That explains why installation images combining packages from main, contrib, and non-free have always been marked as unofficial and haven’t been advertised prominently, even if they’ve been increasingly much more useful than the official installation images.

The 2022 General Resolution about non-free firmware changes the status quo, by allowing non-free firmware packages to be included in official installation images directly, making everyone’s life easier, finally!

This blog post discusses Cyril’s journey into making that happen for Bookworm.

The following is organized on a per-topic fashion, but progress hasn’t been linear!

The non-free-firmware archive area

Let’s start with dak, which is used by the FTP team to manage the Debian archive: main, contrib, and non-free have been around forever, and non-free-firmware needed to get added alongside.

At that point, the archive was ready to accept packages into non-free-firmware but a few tweaks were still needed, like synchronizing DEP-11 metadata for this brand new component and a follow-up bugfix (since at some point https://appstream.debian.org/ had data under unstable/non-free/dep11/ but not under testing/non-free/dep11/).

Moving firmware packages

Package uploads

A first draft was sent to the installer team and to the maintainers of firmware packages that seemed most interesting to have on installation images. Merge requests were filed for packages with repositories hosted on https://salsa.debian.org/ and bug reports with patches were filed for the others. Maintainers were quite supportive, some uploaded their packages swiftly with the proposed changes, some asked for sponsorship, and Cyril uploaded the few remaining ones after a while.

The firmware-nonfree source package was a little challenging, as it reuses some parts of the linux packaging, which had been updated. This explains why the Update to new linux-support API, move packages to non-free-firmware/* sections merge request was the biggest set of changes overall: other packages only required an updated Section in the debian/control file and a descriptive changelog entry. A smaller Address autorejects merge request made sure that lintian as run by dak would be happy with that second upload.

Since packages were switching from an archive area to another one, they had to go through NEW for review, which in turn meant that uploads needed to include binaries. Since most packages contain data, they are usually Architecture: all, meaning all uploads were source+all.

Thanks to a great support from the FTP team, all uploads were processed very quickly.

Migration to testing, autobuilding

Usually there’s a bit of a cognitive dissonance between uploads going through NEW requiring binaries alongside the source, and britney (the release team tool managing migrations from unstable to testing) which insists on having all binaries built on build daemons (rather than uploaded by the maintainer).

That being said, the usual source-only, follow-up upload after a package has been accepted from NEW… was not necessary in this specific case since the “Not built on buildd” check is only applied to packages in main. That means britney required not changes at all!

Everything looked good, until the Debian Kernel Team prepared a new upstream release for firmware-nonfree, and only the source package appeared in the archive, while binary packages were nowhere to be found. It turned out that the buildd infrastructure hardcoded components in various places, but thankfully Philipp Kern and Aurélien Jarno took care of it in just a few days, and firmware-nonfree binary packages finally appeared in the archive.

Using moved firmware packages

Debian Installer

hw-detect

Let’s start with the hw-detect component. Its check-missing-firmware script keeps an eye on Linux kernel logs (via dmesg) and looks for a specific pattern to notice modules requesting firmware files. Here’s an example, with the r8152 module requesting rtl_nic/rtl8153a-3.fw:

kernel: [   75.671338] r8152 4-2.4:1.0: firmware: failed to load rtl_nic/rtl8153a-3.fw (-2)

When that happens, check-missing-firmware tries to find the requested files in the firmware packages found in the /firmware directory, if any. As a last resort, users get a prompt with the requested files, and they get asked whether they’d like to search for firmware on external storage (e.g. a USB stick).

Until now, with official installation images, there were no such firmware packages, and the prompt was shown every single time. With unofficial installation images including firmware packages, seeing this prompt was very unlikely (but that could still happen, e.g. for recent hardware).

Starting with Debian Installer Bookworm Alpha 2, official installation images include firmware packages from main and non-free-firmware, and seeing this prompt should be unlikely as well.

On the technical side, check-missing-firmware will automatically use a Contents-firmware index if it’s available in the firmware directory, mapping each firmware file to the firmware package that ships it, and to the archive area it was found in. If there’s no Contents-firmware index, the old implementation is used as a fallback, i.e. checking the contents of each firmware package; and determining the archive area by looking at the Section field of the firmware package.

Now that we have found one or more firmware packages for the requesting files:

  • The firmware packages are unpacked in the installer environment, making their firmware files available to the running Linux kernel.
  • The relevant modules are reloaded, so that they can request their firmware files again… and find them this time.
  • The firmware packages are queued for installation in the system getting installed.
  • The detected archive areas are used to tweak the default APT configuration for the system getting installed.

Of course, that only works for Linux kernel modules that are actually loaded, and actively requesting firmware files. The installer environment uses a generic video driver, doesn’t feature sound support (unless speech synthesis is used but that’s another story)… so many things could get overlooked. Thankfully, we have another way to detect firmware packages that might be needed, based on modalias information: we use the udev database and patterns found in the /firmware directory to detect which firmware packages can be useful. While this feature has been added during the Bullseye release cycle, it’s only useful when firmware packages are indeed included in the installation images!

Feedback loops are usually much shorter when testing inside a virtual machine, and the test setup revealed a fun issue:

  • Standard VM created by libvirt with default settings;
  • Realtek-based Wi-Fi dongle connected over USB on the host, shared as a USB device via libvirt.

This test setup results in the following kernel logs:

rtl8192cu: Loading firmware rtlwifi/rtl8192cufw_TMSC.bin
usb 4-1.5: firmware: failed to load rtlwifi/rtl8192cufw_TMSC.bin (-2)
usb 4-1.5: Direct firmware load for rtlwifi/rtl8192cufw_TMSC.bin failed with error -2
usb 4-1.5: firmware: failed to load rtlwifi/rtl8192cufw.bin (-2)
usb 4-1.5: Direct firmware load for rtlwifi/rtl8192cufw.bin failed with error -2

This means we record the usb module has the one requesting firmware files, which we then try to reload after deploying firmware-realtek. That cannot work!

Therefore, a special case was added: when the module requesting firmware files is usb, search the USB bus using port and device information to find the underlying device and the module managing it. With the above example, this makes it possible to trade usb 4-1.5 for rtl8192cu. Funnily enough, that wasn’t noticed immediately, as this specific module requests firmware files but can work without them!

Since it’s expected that some users don’t want to install firmware packages even if they are present on installation images, Cyril started a discussion on how to make it possible for users to opt out. It was decided to support a firmware=never boot-time option to disable firmware support entirely.

Uploads: hw-detect 1.153, hw-detect 1.154, preseed 1.113.

apt-setup

This component is responsible for configuring APT for the system getting installed. While it defaults to only enabling main, other archive areas can be selected when using an expert installation (asking many more questions).

When installing firmware packages from outside main, it’s important to configure APT accordingly, so that those packages can be upgraded later on.

Small updates were needed for apt-setup to gain non-free-firmware support:

  • A new apt-setup/non-free-firmware template was added, only shown during expert installation (like apt-setup/contrib and apt-setup/non-free), but it can also be enabled by hw-detect when firmware packages from non-free-firmware are deployed (see above).
  • Generators were modified to take all three templates into account, making sure the relevant archive areas are enabled for the base distribution (bookworm) but also for the security suite (bookworm-security), and optional extra suites (e.g. bookworm-updates).

Upload: apt-setup 1:0.174.

netcfg

It’s very tempting to only install a basic system when working on the installer, skipping a full desktop installation. While doing so, even once firmware packages are in place, once hw-detect and apt-setup have done their work flawlessly, rebooting into the installed system can be surprising: the wireless network that was used during the installation is nowhere to be seen, meaning no connectivity at all!

Cyril’s investigation revealed several issues, filed as #1029352 against netcfg.

One of those issues turned out to be a regression in ifupdown 0.8.40: trying to add support for allow-hotplug interfaces in systemctl restart networking ended up breaking support for them at boot-time! Cyril proposed a minimal patch that would make sure the previous behaviour would be kept (ensuring boot-time is fine), while retaining the addition of restart support. Better solutions are certainly possible, but there are still only 24 hours in a day, and swiftly restoring the existing status quo looked very appealing at this point of the release cycle…

The other two issues were fixed in netcfg, by activating support for wireless interfaces when using ifupdown (i.e. when network-manager isn’t used) instead of explicitly disabling it, and by working around what seems to be a file format limitation regarding /etc/network/interfaces and multiple stanzas with wireless options (e.g. DHCP for IPv4, SLAAC for IPv6).

Upload: netcfg 1.182.

Image generation – debian-cd

The debian-installer source package is responsible for stitching many udebs (for µdebs) together to generate various targets. Here’s a full list on amd64:

  • cdrom_isolinux
  • cdrom_gtk
  • cdrom-xen
  • netboot
  • netboot-gtk
  • netboot-xen
  • hd-media
  • hd-media_gtk
  • monolithic

Depending on which udebs are getting patched, it’s usual for developers to use netboot or netboot-gtk which generate dest/netboot/mini.iso and dest/netboot/gtk/mini.iso respectively, which can be passed to kvm via the -cdrom flag.

But since the whole point was improving support for firmware packages on installation images, going one step further was needed: looking into debian-cd! This is the tool that consumes what the debian-installer source package produces, checks what is available in the Debian archive, and generates installation images accordingly: the famous netinst image, CD sets, DVD sets, etc.

This is definitely not the easiest part to set up for a fly-by contribution:

  • the code lives in the debian-cd repository;
  • the configuration lives in the setup repository;
  • debian-cd requires access to a local mirror.

Cyril was already familiar with those requirements, having provided patches in the past, and catching up with latest debian-cd master should have been rather easy. There were a few changes along the way though, related to how local debs and udebs can be included in development builds… That’s a feature that’s never used for official builds (as published on cdimage.debian.org), and several rounds of clean-up were needed to make builds with local packages work again.

Those changes made it possible to build images with easy-build.sh, which is a convenient wrapper for the official build.sh command, using debian-installer build artifacts (via the cdrom_isolinux and cdrom_gtk targets):

  • cdrom/debian-cd_info.tar.gz
  • cdrom/gtk/debian-cd_info.tar.gz
  • cdrom/gtk/vmlinuz
  • cdrom/gtk/initrd.gz
  • cdrom/vmlinuz
  • cdrom/xen/debian.cfg
  • cdrom/initrd.gz

and a set of local packages, including the netcfg and apt-setup udebs that get loaded during the installation process, and a patched ifupdown package to side-step the regression mentioned earlier (#1022843).

Once netinst images could be produced, actual work on debian-cd could start!

  • Extending the Contents-firmware index: initially mapping firmware files to firmware packages, information about archive areas was added, so that hw-detect can configure apt-setup properly when deploying firmware packages based on dmesg parsing (missing firmware files).
  • Improving robustness around DEP-11 metadata: making sure source data files are present (which led to spotting missing files in the archive, see dak merge request mentioned earlier, and some missing architectures in an appstream.debian.org configuration file).
  • Extending the *.patterns files generated from DEP-11 metadata with matching *.component files, so that hw-detect can configure apt-setup properly when deploying firmware packages based on modalias information.
  • Reworking the generate_firmware_patterns file so that its logic could get reused outside image builds (see below).
  • Refreshing the hardcoded list of devices supported by firmware-sof-signed, since that modalias information isn’t available for Intel SOF (audio firmware).

Last but not least, the firmware package lookup was adjusted. Initially, packages with names matching a given pattern would be searched across all archive areas, since the firmware inclusion code was being used only for unofficial builds, with contrib and non-free enabled. That lookup was adjusted to only search the configured archive areas, meaning main, non-free, and non-free-firmware while packages were getting moved from non-free to non-free-firmware. Once all packages were uploaded and migrated to testing, non-free was dropped, giving us the desired main and non-free-firmware combination.

Outside image builds, make-firmware-image is a tool that generates firmware archives (tar, zip, and cpio formats), so that users can fetch a single file that contains all firmware packages in one step. Since those archives are still going to be useful for netboot images (they are produced by a debian-installer build, using packages from main only, meaning absolutely no firmware), it was rewritten to reuse the updated firmware package lookup from generate_firmware_patterns, and to also generate metadata to help hw-detect and apt-setup (i.e. Contents-firmware index plus *.patterns and *.component files). The DebianInstaller/NetbootFirmware wiki page was updated accordingly, pointing to the new, official location for those firmware archives: https://cdimage.debian.org/cdimage/firmware/.

Upload: debian-cd 3.2.0.

Documentation

With firmware packages moved to non-free-firmware and migrated to testing, with smarter Debian Installer components, and with debian-cd including firmware packages with appropriate metadata, all the technical parts seemed in place.

Finally, the focus could move to writing some documentation. Admittedly, we could have started there, but it seemed to make sense to make good progress on the technical side first, and to check whether the initial plan would work out, before thinking about documentation.

The following “must have” were identified, and patches were provided:

The Debian Installer landing page in the Debian Developers’ Corner of the website was updated when Debian Installer Bookworm Alpha 2 was published.

The rest of the website contains many references to the usual main contrib non-free triplet, and a heads-up bug report was filed accordingly, so that Cyril’s colleagues could take over. Meanwhile, https://packages.debian.org/ was updated to also index packages in the new non-free-firmware archive area (with this merge requested getting deployed live so that testing and unstable users could locate packages that seemed to have disappeared).

Meanwhile, an APT patch is cooking, to help users migrating from bullseye to bookworm.

Upload: installation-guide 20230215.

Online documentation:

Stitching everything together: Debian Installer Bookworm Alpha 2

We’ve published the first installer release with hopefully all the bits in place: Debian Installer Bookworm Alpha 2.

You can join the fun, by installing with the brand new images, and by also submitting installation reports. Those reports are always welcome, even if the installation process was uneventful. With those important changes regarding firmware support, it’s very important for us to understand if that’s indeed helping as much as we hoped, or if we have to go back to the drawing board because we missed important things…

The importance of CPU microcode updates was underlined earlier, and support is getting prepared for the next release. Some optimizations are lined up already, and we still need to generate the promised report, since some people might want to keep a close eye on the firmware packages getting installed.

On a personal note, Cyril would like to thank Steve McIntyre for all the support during the last 10 years! There are so many moving pieces involved in publishing Debian Installer releases, it’s always easier when a fellow developer has got your back! 💯