Navigation: blog > 2019 RSS

Explore a single month: 01 04 09

Adding Raspberry Pi CM3 support to Debian Buster

Raspberry-Pi-CM3

Introduction

Since the initial launch in 2012, many Raspberry Pi models have been released. Leaving older versions 1 and 2 aside, one can mention those more recent models:

Below: what a Raspberry Pi 4 Model B looks like (credit: www.raspberrypi.org).

Pi 4 Model B
Raspberry Pi 4 Model B

In addition to those single-board devices, some Computer Module (CM) models are available. Their form factor is slightly different and aims at making it possible to embed them into industrial products.

Below: a few pictures of a Raspberry Pi CM3 (click for full view).

Pi CM3, front Pi CM3, back
Pi CM3, front Pi CM3, back
Pi CM3 (bottom) compared to Pi 3 (top)
1

>> Check the Raspberry Pi website for more details about the Compute Module 3.

For development purposes, it can be mounted on a development kit called the Compute Module IO Board V3. This breakout board exposes many ports, including GPIO pins, HDMI, and USB.

Below: what a Compute Module IO Board V3, without or with a mounted Raspberry Pi CM3, looks like (click for full view).

IO Board V3 (left) and Pi CM3 (right) IO Board V3 with mounted Pi CM3
Compute Module IO Board V3 (left) and Pi CM3 (right) Compute Module IO Board V3 with mounted Pi CM3

>> Check the Raspberry Pi website for more details about the Compute Module IO Board V3.

This article aims at documenting some specifics of this kind of hardware, and Debamax’s contributions to make the Raspberry Pi CM3 usable on Debian systems.

Possible support in Debian Buster

It was already established that Debian Buster ships with all the required pieces to support “regular” Raspberry Pi 3 devices. Thanks to the work of the Debian Raspberry Pi Maintainers, it is rather easy to prepare a Debian image to run on the target device.

Specifically, the Raspberry Pi image specs repository contains various YAML-based recipes that can be used as-is or modified to tweak the configuration a little. This includes being able to add or remove some packages, tweak users and passwords, locales, etc. Here’s a direct link to the raspi3.yaml file. This recipe is to be used with the vmdb2 tool, that will produce an image ready to be copied onto the Raspberry Pi.

Note: It’s possible to build an arm64 image from an amd64 machine as the architecture-specific steps can be performed within QEMU. There’s also a caching mechanism making it possible to avoid re-doing lengthy steps in case someone needs a few tries to get the right settings.

Unfortunately, while an image built from the stock raspi3.yaml configuration file boots fine on a Raspberry Pi 3, that’s not the case for a Raspberry Pi CM3.

Why is that? The hardware is mostly the same, and the Linux kernel contains appropriate code to support it. But it also needs to know what components are available, how they are wired, etc. Such configuration can be made available through a so-called device tree. The bootloader (e.g. U-Boot) can look for a particular, hardware-specific file containing a compiled version (DTB) and hand it over to the Linux kernel so that the latter knows about the actual hardware setup.

The main takeaway is that while a single Linux kernel image can be used on many different devices, one has to make sure there’s a DTB available for this specific device… And as of August 2019, that’s the main difference between the regular Raspberry Pi 3 and the Raspberry Pi CM3 version. The former needs bcm2837-rpi-3-b.dtb or bcm2710-rpi-3-b-plus.dtb, shipped in the linux-image-<ABI> package on arm64, while the latter has a bootloader that expects bcm2710-rpi-cm3.dtb, which is lacking in the initial Buster release (versioned 10.0). Thankfully, Debian stable releases get updated through point releases and adding hardware support might be feasible!

Linux kernel support

The device tree blobs are built from the Linux kernel source, from a variety of *.dts and *.dtsi files. The difference between those two file extensions is a technical detail regarding reusability: one can include some other file to avoid repeating the same contents in many different files. That can happen when devices vary very slightly (see below).

The Linux kernel in Buster is based on the 4.19.y series, and support for the Raspberry Pi CM3 was first released in Linux v4.20-rc1 upstream tag. The needed patches are the following:

  1. ARM: dts: add Raspberry Pi Compute Module 3 and IO board
  2. arm64: dts: broadcom: Add reference to Compute Module IO Board V3
  3. arm64: dts: broadcom: Use the .dtb name in the rule, rather than .dts

Let’s break it down…

Those patches were quite small and mostly additions, so backporting them to the 4.19.y kernel in Buster was trivial. There could have been conflicts in the Makefile but that part didn’t change between both versions anyway.

To generate an updated linux-image-<ABI> for arm64, a quick way is to cross-build it from a fast amd64 host. The following was sufficient for this use case:

DEB_BUILD_OPTIONS="nocheck nodoc" DEB_BUILD_PROFILES="pkg.linux.notools cross nopython" debuild --no-lintian -aarm64 -B -j8 -nc
  

Once the build finishes, a number of binary packages are available in the parent directory:

linux-headers-4.19.0-6-arm64_4.19.67-2_arm64.deb
linux-image-4.19.0-6-arm64-unsigned_4.19.67-2_arm64.deb
linux-image-4.19.0-6-arm64-dbg_4.19.67-2_arm64.deb
linux-headers-4.19.0-6-all-arm64_4.19.67-2_arm64.deb
linux-headers-4.19.0-6-all_4.19.67-2_arm64.deb
linux-config-4.19_4.19.67-2_arm64.deb
linux-libc-dev_4.19.67-2_arm64.deb
linux-image-arm64-signed-template_4.19.67-2_arm64.deb
  

Note: For more details about the unsigned and signed-template packages, see this article: An overview of Secure Boot in Debian.

The resulting linux-image-4.19.0-6-arm64-unsigned binary contains those Raspberry Pi DTBs accordingly:

./usr/lib/linux-image-4.19.0-6-arm64/broadcom/bcm2837-rpi-3-b-plus.dtb
./usr/lib/linux-image-4.19.0-6-arm64/broadcom/bcm2837-rpi-3-b.dtb
./usr/lib/linux-image-4.19.0-6-arm64/broadcom/bcm2837-rpi-cm3-io3.dtb
  

and the third one is indeed what the 3 backported patches were supposed to be achieving. All good?

raspi3-firmware support

Unfortunately, there’s still one part missing: that extra DTB file needs to be made available where the bootloader expects it. This means being available under the /boot/firmware directory. Moreover, it has to be named bcm2710-rpi-cm3.dtb as mentioned in the introduction section, rather than bcm2837-rpi-cm3-io3.dtb as built and shipped in the Linux image package.

Copying the DTB is taken care of by the raspi3-firmware package, and by its /etc/kernel/postinst.d/z50-raspi3-firmware hook specifically. Excerpt:

if [ "$KERNEL" = "auto" ]; then
  pi0w_dtb=${dtb_path}/bcm2835-rpi-zero-w.dtb
  pi1ap_dtb=${dtb_path}/bcm2835-rpi-a-plus.dtb
  pi1bp_dtb=${dtb_path}/bcm2835-rpi-b-plus.dtb
  pi2b_dtb=${dtb_path}/bcm2836-rpi-2-b.dtb
  pi3b_dtb=${dtb_path}/bcm2837-rpi-3-b.dtb
  pi3bp_dtb=${dtb_path}/bcm2837-rpi-3-b-plus.dtb

  [ -e "${pi0w_dtb}"  ] && cp "${pi0w_dtb}"  /boot/firmware/bcm2835-rpi-zero-w.dtb
  [ -e "${pi1ap_dtb}" ] && cp "${pi1ap_dtb}" /boot/firmware/bcm2835-rpi-a-plus.dtb
  [ -e "${pi1bp_dtb}" ] && cp "${pi1bp_dtb}" /boot/firmware/bcm2835-rpi-b-plus.dtb
  [ -e "${pi2b_dtb}"  ] && cp "${pi2b_dtb}"  /boot/firmware/bcm2709-rpi-2-b.dtb
  [ -e "${pi3b_dtb}"  ] && cp "${pi3b_dtb}"  /boot/firmware/bcm2710-rpi-3-b.dtb
  [ -e "${pi3bp_dtb}" ] && cp "${pi3bp_dtb}" /boot/firmware/bcm2710-rpi-3-b-plus.dtb

Two important things can be noted here:

Adding an extra variable and an extra if/cp call makes it possible to get the extra DTB copied under the expected name.

Is that all? Yes! Patching the raspi3.yaml file to install the patched linux-image-<ABI> and raspi3-firmware packages leads to an image which contains the DTB in the right place and under the right name; the bootloader finds it and then the Pi CM3 boots up just fine!

Contributions

Following the usual Upstream first! mantra, Cyril checked the Documentation/process/stable-kernel-rules.rst and it seemed that a DTB addition would fit the “New device IDs and quirks are also accepted.” criterion. So a small patch series was submitted for inclusion into the 4.19.y stable tree so that it would flow downstream to Debian at some point, but that was deemed not to be suitable for stable upstream.

That’s why Cyril resorted to submitting a merge request against the linux source package in Debian to get that fixed downstream: sid: Please add DTB support for Rasperry Pi Compute Module 3; this was promptly merged by Vagrant Cascadian, and released to unstable in version 4.19.37-6.

Since the buster uploads didn’t integrate those patches, a new merge request against the buster branch was submitted as well: buster: Raspberry Pi CM3 support; this was merged by Ben Hutchings, and later uploaded to buster-proposed-updates.

On the raspi3-firmware side, the packaging repository is open to contributions by all Debian Developers, so Cyril pushed the Add support for bcm2837-rpi-cm3-io3.dtb aka. bcm2710-rpi-cm3.dtb commit to the master branch directly.

With a green light from Romain Perier, Cyril also prepared a buster-proposed-updates bug report against release.debian.org: #935386: buster-pu: package raspi3-firmware/1.20190215-1+deb10u1. That is the standard procedure to request approval from the Stable Release Managers to update a package in a stable release. There was a little glitch though, as the patched version didn’t enter unstable immediately, and having bugfix or improvements available there is a usual requirement from the Stable Release Managers. This slight delay in getting the updated package into unstable was due to the source and binary packages getting renamed from raspi3-firmware to raspi-firmware (as mentioned above, Raspberry Pi 4 models are available now), hence the need for a review through the NEW queue.

Without surprises, both the linux and raspi3-firmware packages were approved by the Stable Release Managers, and were made available in the buster-proposed-updates suite. They were then merged into the official buster suite during the 10.1 point release, which happened on 2019-09-08.

The following table sums up which packages are needed to get support for Raspberry Pi CM3.

stable (buster) unstable (sid)
linux 4.19.67-2 4.19.37-6 and above
raspi*-firmware raspi3-firmware 1.20190215-1+deb10u1 raspi-firmware 1.20190718-1

Executive summary: Since the 10.1 point release, the Raspberry Pi CM3 is officially supported in buster and above! Its evolution, the Raspberry Pi Compute Module 3+ (CM3+), is supported as well.


Published: Mon, 09 Sep 2019 14:15:00 +0200

An overview of Secure Boot in Debian

Secure Boot

Foreword

This blog post isn’t meant to be a definitive guide about Secure Boot in Debian. The idea is to give some context about the boot sequence on the PC architecture, about the Secure Boot technology, and about some implementation details in Debian.

Short on time? Jump to current status of Secure Boot in Debian!

How does a system boot?

Let’s start with how the PC architecture gets booted: once upon a time, the BIOS was responsible for locating boot devices and trying them in a configurable order. One would usually configure a bootable disk with a bootloader in its MBR (e.g. LILO ou GRUB), which would then check its own settings, and boot a Linux kernel passing parameters and an optional initramfs.

Things changed “a little” with the UEFI technology, aiming at replacing the BIOS (and at being usable on other architectures like ARM). The initial firmware comes with many more features, and with recommended or required settings, like the ESP partition (usually mounted on /boot/efi) which makes it possible to exchange data between the UEFI-level implementation and the installed system. Regarding booting, there’s a boot manager implemented at the firmware level, which can be configured from the operating system (through the efibootmgr command or through some EFI libraries).

Here’s a shortened efibootmgr -v example showing debian as the default operating system, with the standardized EFI/debian/grubx64.efi path that can be found in the ESP (which is a FAT filesystem, hence the notation with backslashes), and with two PXE-based fallbacks:

$ sudo efibootmgr -v
BootCurrent: 0000
Timeout: 0 seconds
BootOrder: 0000,0001,0002
Boot0000* debian                        HD(1,GPT,0d5445e2-92b4-4a31-9629-c6c796b3e07c,0x800,0xf3800)/File(\EFI\debian\grubx64.efi)
Boot0001* IBA GE Slot 00C8 v1381        BBS(Network,,0x0)AMBO
Boot0002* IBA GE Slot 0200 v1321        BBS(Network,,0x0)AMBO

UEFI-enabled firmwares usually make it possible to use either “UEFI booting” or “Legacy BIOS” (also called CSM).

Booting a Linux kernel with UEFI instead of Legacy BIOS usually leads to some extra information getting exposed through /sys, namely under the /sys/firmware/efi directory. In particular, the efivars.ko module makes it possible to access variables that are stored in NVRAM.

What is Secure Boot?

Secure Boot is a technology that makes it possible to check and possibly trust the boot chain. The initial firmware would check a digital signature on the bootloader; the kernel getting loaded and its modules would get a similar check. The idea is to double check that what is being run as the core of the operating system is the expected system, and that no rogue operations have been taking place. Having support for Secure Boot was a requirement for hardware targetting conformance with the Windows 8 specifications, so Secure Boot enabled devices have been spreading over the past few years…

Digital signatures are all good but who should a firmware trust? Given the market was mainly about machines getting sold with Windows, the firmware would be configured to trust keys from Microsoft by default. Some firmware implementations make it possible to enroll other keys, but that’s not supported by all devices…

Until Linux distributions have designed and implemented a plan to support Secure Boot (getting a bootloader that can check and start a Linux kernel in a suitable fashion, and signed by a trusted key), the usual solution to run a Linux installer would be to fiddle with the UEFI settings, turning Secure Boot off entirely. This can be cumbersome, as some firmware implementations only show this option after an administrator password has been set…

What does the plan look like for Debian?

Secure Boot support has been in the works for quite a long time, and there were many design issues to iron out, including some infrastructure-side changes regarding digital signatures.

Here’s a very quick summary:

The first step is the firmware→shim chainloading. Fortunately, only shim needs to get a signature from Microsoft, so that the machine’s firmware can validate the shim component getting loaded. Being a minimal and auditable component means it shouldn’t need to get updates too often, which should keep the number of roundtrips to Microsoft (to get a new signature) rather low.

The shim→GRUB chainloading is done if the signature on GRUB is validated against the Debian test key or the Debian production key (more on that below). Ditto for the GRUB→Linux kernel chainloading.

From a packaging point of view, these digital signatures are a bit of a nightmare: one wants to be able to build packages on autobuilders (on build daemons, or buildds), possibly in a reliable and reproducible fashion (see the Reproducible Builds initiative). That’s why some modifications to the grub2 and linux source packages have been floating around for some time.

As of early March 2019, the state of the shim and shim-signed packages in Debian unstable was a bit complicated: the shim package was updated with new code while the matching signature from Microsoft wasn’t available for inclusion in an updated shim-signed package yet… Steve McIntyre and Cyril have been working hard exploring various solutions allowing to get back to a set of matching packages (#922179). Some difficulties encountered in doing so highlight the need for reproducible builds on the one hand and for a carefully designed supply chain on the other hand. This is why the next section is focussed on the -signed and -template packages that are in place for the grub2 and linux source packages. As of April 2019, the binaries produced by the shim source package have been reorganized to match the setup used by grub2 and linux.

How are -signed and -template packages handled in Debian?

Quick look into GRUB

Let’s start by looking at the binary packages produced by the grub2 source package. There are 48 binaries as of version 2.02+dfsg1-17 so let’s only list some of them:

The set of packages to be installed would usually be decided by the grub-installer component of the Debian Installer; a machine installed with legacy BIOS could have this set of GRUB packages (taken from Debian Stretch):

… with grub-pc pulling the relevant tools in grub-pc-bin and both of them relying on the grub*-common packages.

On an EFI installation (also Debian Stretch), the set would look like this:

Of course, EFI is available on more than just the amd64 architecture, so the grub-efi package pulls the right grub-efi-$ARCH, which then pulls binaries and the common files.

Signatures for GRUB

All binary packages mentioned above are built on autobuilders, and contain no signatures. But some of them are special, namely:

Those are indeed binary packages produced by the grub2 source package, but they are effectively meant to be the source package for the signed binary packages!

Let’s look at the contents of grub-efi-amd64-signed-template_2.02+dfsg1-17_amd64.deb (letting the usual /usr/share/doc and /usr/share/lintian directories aside):

/usr/share/code-signing/grub-efi-amd64-signed-template/
/usr/share/code-signing/grub-efi-amd64-signed-template/files.json
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/README.source
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/bug-control
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/changelog
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/compat
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/control
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/copyright
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/rules
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/source/
/usr/share/code-signing/grub-efi-amd64-signed-template/source-template/debian/source/format

There seems to be a source package tree there, with metadata gathered in a debian directory.

Let’s check the top-level files.json file that describes which files need to be signed and how:

{
    "version": "2",
    "packages": {
        "grub-efi-amd64-bin": {
            "trusted_certs": [],
            "files": [
                {"sig_type": "efi", "file": "usr/lib/grub/x86_64-efi/monolithic/gcdx64.efi"},
                {"sig_type": "efi", "file": "usr/lib/grub/x86_64-efi/monolithic/grubnetx64.efi"},
                {"sig_type": "efi", "file": "usr/lib/grub/x86_64-efi/monolithic/grubx64.efi"}
            ]
        }
    }
}

Let’s check what the debian/control file looks like:

Source: grub-efi-amd64-signed
Section: admin
Priority: optional
Maintainer: GRUB Maintainers <REDACTED>
Uploaders: Individual Developers <REDACTED>
Standards-Version: 3.9.8
Build-Depends: debhelper (>= 10.1~),
 sbsigntool [amd64 arm64 i386],
 grub-efi-amd64-bin (= 2.02+dfsg1-17)
Rules-Requires-Root: no

Package: grub-efi-amd64-signed
Architecture: amd64
Depends: grub-common (= 2.02+dfsg1-17)
Recommends: shim-signed [amd64]
Built-Using: grub2 (= 2.02+dfsg1-17)
Description: GRand Unified Bootloader, version 2 (amd64 UEFI signed by Debian)
 GRUB is a portable, powerful bootloader.  This version of GRUB is based on a
 cleaner design than its predecessors, and provides the following new features:
 .
  - Scripting in grub.cfg using BASH-like syntax.
  - Support for modern partition maps such as GPT.
  - Modular generation of grub.cfg via update-grub.  Packages providing GRUB
    add-ons can plug in their own script rules and trigger updates by invoking
    update-grub.
 .
 This package contains the binaries signed by the Debian UEFI CA to be used by
 shim-signed.

Let’s highlight a few things:

The last point explains why the grub-installer component of the Debian Installer doesn’t even need to list any *-signed packages, the following chain of dependencies takes care of installing the needed packages for the initial chainloading: grub-efigrub-efi-amd64grub-efi-amd64-bin, the last one recommending grub-efi-amd64-signed, which in turns recommends shim-signed.

Let’s check what the debian/rules file looks like:

#!/usr/bin/make -f

SIG_DIR := debian/signatures/grub-efi-amd64-bin

%:
        dh $@

override_dh_auto_install:
        set -e ; \
        find "$(SIG_DIR)" -name '*.sig' -printf '%P\n' | \
        while read sig; do \
                dst="debian/tmp/$${sig%/monolithic/*}-signed/$${sig##*/}ned" ; \
                install -m 0755 -d "$${dst%/*}" ; \
                install -m 0644 "/$${sig%.sig}" "$$dst" ; \
                sbattach --attach "$(SIG_DIR)/$$sig" "$$dst" ; \
        done

override_dh_install:
        dh_install --sourcedir=debian/tmp .

As mentioned in the SecureBoot/Discussion page of the Debian Wiki, signatures are added to source-template/debian/signatures/<original-binary-package-name>/<complete-path-name>.sig when the -signed-template file is processed by the code signing service (which won’t be detailed in depth in this article): This explains why there is a loop in the dh_auto_install override, to attach the generated signatures to the actual files that were installed because of the build dependencies on grub-efi-amd64-bin.

This leads to the following contents for the resulting grub-efi-amd64-signed binary package:

/usr/lib/grub/x86_64-efi-signed/gcdx64.efi.signed
/usr/lib/grub/x86_64-efi-signed/grubnetx64.efi.signed
/usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed

which are a signed version of the following files in the grub-efi-amd64-bin binary package:

/usr/lib/grub/x86_64-efi/monolithic/gcdx64.efi
/usr/lib/grub/x86_64-efi/monolithic/grubnetx64.efi
/usr/lib/grub/x86_64-efi/monolithic/grubx64.efi

(There’s also an extra file both in the source template and in the binary package to instruct reportbug which package to file bug report against but that’s really anecdotal.)

Wrapping up for GRUB

  1. The grub2 source package generates many binary packages.
  2. Those named *-signed-template are post-processed on a specific code signing service, with their contents being used to generate a source package that builds the final binary packages containing signed files.
  3. Those signed files are the combination of files shipped in other binary packages, with a digital signature appended.

Signatures for the Linux kernel

The same mechanism is used for the linux source package. It is slightly different because of the amount of binary packages that are built from this source package: 1194 as of version 4.19.28-2! There can be various flavours and patchsets involved, for each supported architecture; plus many udebs (components to be used in the Debian Installer), explaining this high number.

One might have noticed that the linux-image-<ABI> packages are no longer built by the linux source package though. Let’s focus on amd64 again:

  1. The linux source package builds a linux-image-amd64-signed-template binary package.
  2. Using this *-signed-template binary package, many signed packages are generated, which include the following, familiar one: linux-image-4.19.0-4-amd64. Those don’t come with a -signed suffix, probably because changing their names in all packages and scripts related to the Linux kernel would have meant huge work for little benefit.

Let’s look at the linux-image-amd64-signed-template binary package as of version 4.19.28-2:

/usr/share/code-signing/linux-image-amd64-signed-template
/usr/share/code-signing/linux-image-amd64-signed-template/files.json
/usr/share/code-signing/linux-image-amd64-signed-template/source-template
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/changelog
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/compat
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/control
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/copyright
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-amd64.postinst
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-amd64.postrm
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-amd64.preinst
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-amd64.prerm
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-cloud-amd64.postinst
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-cloud-amd64.postrm
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-cloud-amd64.preinst
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-cloud-amd64.prerm
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-rt-amd64.postinst
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-rt-amd64.postrm
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-rt-amd64.preinst
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/linux-image-4.19.0-4-rt-amd64.prerm
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/README.source
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/rules
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/rules.gen
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/rules.real
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/source
/usr/share/code-signing/linux-image-amd64-signed-template/source-template/debian/source/format
  

Some differences compared to the previous grub-efi-amd64-signed-template package:

Let’s check the Source stanza of the debian/control file, leaving the 55 Package entries aside for a moment:

Source: linux-signed-amd64
Section: kernel
Priority: optional
Maintainer: Debian Kernel Team <REDACTED>
Uploaders: Ben Hutchings <REDACTED>
Standards-Version: 4.1.1
Build-Depends:
 debhelper (>= 10.1~),
 rsync,
 sbsigntool,
 kernel-wedge (>= 2.99~),
 linux-kbuild-4.19,
 linux-support-4.19.0-4 (= 4.19.28-2),
 linux-image-4.19.0-4-amd64-unsigned (= 4.19.28-2),
 linux-image-4.19.0-4-cloud-amd64-unsigned (= 4.19.28-2),
 linux-image-4.19.0-4-rt-amd64-unsigned (= 4.19.28-2)
Homepage: https://www.kernel.org/
Rules-Requires-Root: no
Vcs-Browser: https://salsa.debian.org/kernel-team/linux
Vcs-Git: https://salsa.debian.org/kernel-team/linux.git

Glancing at the build dependencies, there are 3 linux-image-* binary involved, containing files that will need signatures. Both linux-kbuild-4.19 and linux-support-4.19.0-4 are likely to be use to make the Makefile machinery work, while kernel-wedge is used to dispatch kernel modules into various binary packages.

Instead of looking at all 55 binary packages, let’s concentrate on some of them. Only 3 of them are deb packages, while all others (named *-amd64-di) are udeb packages, for use in the Debian Installer.

Another huge difference compared to GRUB is the files.json file. While the GRUB one was only used to list 3 files for a single package, the Linux kernel one is close to 1 MB in size!

It also comes with different kinds of signatures. On the GRUB side, the signature type was efi; on the Linux side, that happens for some files as well:

which is consistent with the GRUB to Linux chainloading. But all other files that need signatures are the kernel modules, which use the linux-module signature type.

Signatures for other packages

This write-up is overly long already, so the following packages won’t be detailed, let’s just mention both the source and binary packages for each:

What is the current status of Secure Boot in Debian?

Short version: Starting with the Debian Installer Buster RC 1 release, Secure Boot support should work out of the box on amd64!

Longer version: The Debian Installer Buster Alpha 5 release was released with initial support for Secure Boot. Initially, the signing service running on the Debian infrastructure was using a test key, and some manual enrollment was needed. Since the publication of this last alpha, the switch to the production key happened, and the following release (Debian Installer Buster RC 1) shipped with signatures performed with the production key.


Published: Fri, 19 Apr 2019 15:35:00 +0200

Debugging with netconsole

Kernel BUG/OOPS

Why would one need netconsole?

Sometimes the Linux kernel crashes so badly that it leaves no traces in the logs. Even having a shell with a dmesg -w running in the background might prove to be insufficient.

There’s a nice tool in the kernel which makes it possible to send kernel logs over the network. It’s called netconsole. As far as limitations are concerned, one shall note that it’s UDP only, and over Ethernet (in other words: no wireless). The good news is that it can usually make the last crucial lines available, as it requires a rather limited set of features (as opposed to getting files written on a filesystem, which needs to get onto physical storage).

Example: netconsole made it possible to get a stacktrace of a kernel OOPS when writing to some USB mass storage devices, and to file #917206 in the Debian bug tracking system.

Terminology: Let’s call the crashing machine a patient and the logging machine a doctor.

The netconsole module needs to be loaded on the patient only, while the doctor just needs a user-space program to capture traces. If the module’s configuration needs to be updated or fixed, the module can be unloaded at any time through:

sudo modprobe -r netconsole

It is also highly recommended to ask the kernel to log all the things by setting this specific console log level:

sudo dmesg -n 8

The current console log level can be checked by dumping the contents of the /proc/sys/kernel/printk file, and reading the first value. With the default configuration on Debian 9 (Stretch), the console log level is 4, which isn’t sufficient to confirm netconsole is properly set up; it seems one needs at least console log level 7.

Easy case: on a local network

Here’s an example with both machines on a local network:

Local network

Doctor setup

A receiver is needed on the doctor side, which needs to accept UDP packets. There are several nc (short for netcat) implementations, e.g. netcat-traditional and netcat-openbsd, with subtly different flags. Let’s use socat instead:

sudo apt-get install socat
socat UDP-LISTEN:6666,fork - | tee -a ~/netconsole.txt

Let’s dissect those lines:

Of course the doctor needs to accept such packets, and its firewall might need an update accordingly. If it isn’t maintained through shorewall, ferm, or another dedicated firewall software, the following iptables command might serve as a basis to get packets through:

sudo iptables -A INPUT -p udp -m udp --dport 6666 -j ACCEPT

Patient setup

Now, to have the patient send stuff to the doctor, a simple modprobe call is needed:

sudo modprobe netconsole netconsole=@/eth0,6666@192.168.0.2/

What happens here? One requests the netconsole module to be loaded, and one specifies the parameters. Details can be read in the Linux kernel documentation (Documentation/networking/netconsole.txt), but concentrating on the points of interest here:

That should be enough to get this output on the doctor side:

[ 1748.295633] netpoll: netconsole: local port 6665
[ 1748.295637] netpoll: netconsole: local IPv4 address 0.0.0.0
[ 1748.295639] netpoll: netconsole: interface 'eth0'
[ 1748.295640] netpoll: netconsole: remote port 6666
[ 1748.295642] netpoll: netconsole: remote IPv4 address 192.168.0.2
[ 1748.295644] netpoll: netconsole: remote ethernet address AA:BB:CC:DD:EE:FF
[ 1748.295647] netpoll: netconsole: local IP 192.168.0.1
[ 1748.295702] console [netcon0] enabled
[ 1748.295704] netconsole: network logging started

If nothing appears there, one might want to double check the current console log level (see introduction), and possible packet drops/rejects on the firewall side.

Slightly harder case: over internet

Because one might not have a second machine handy, it’s also possible to go through a router and send stuff across the internet. Let’s consider this case:

Over internet

Doctor setup

The instructions are the same as in the local case, even if it would probably make sense to be more selective regarding firewalling: filtering on the source IP would likely be a good idea.

Patient setup

The fundamental change compared to the local network use case is the need for routing. This is supported by netconsole but one needs to specify an extra parameter: the MAC address of the (first) router. To obtain it, one can use net-tools’s arp command or iproute2’s ip neighbour command:

arp 192.168.0.254
ip n show 192.168.0.254

Supposing it returned the 01:02:03:04:05:06 MAC address, loading the module becomes:

sudo modprobe netconsole netconsole=@/,6666@93.184.216.34/01:02:03:04:05:06

Now, if one is running into firewall-related issues, one can change the source port for the UDP packets. The default is 6665, but assuming one wants to send from an unfiltered 1234 port, that becomes:

sudo modprobe netconsole netconsole=1234@/,6666@93.184.216.34/01:02:03:04:05:06

Permanent debugging?

The approach presented here is temporary by nature, as no modifications of the patient’s system configuration are involved. If desired, one can set the various options to be passed to the netconsole module in a modprobe configuration file. Example with a dedicated modprobe.d snippet:

echo options netconsole netconsole=@/eth0,6666@192.168.0.2/ | sudo tee /etc/modprobe.d/netconsole-local-debugging.conf

Even with such an extra configuration file, those settings would only get applied when the netconsole module is loaded. To have it loaded automatically at boot-up, it can be listed in /etc/modules or in a separate modules-load.d snippet:

echo netconsole | sudo tee /etc/modules-load.d/netconsole.conf

Warning: That relies on having network set up early in the boot process (which won’t be documented here because that’s another topic and that would be require a long digression). If the network isn’t configured already at the time netconsole is set up, one can get:

sudo dmesg | grep netconsole
[   11.677066] netpoll: netconsole: local port 6665
[   11.677143] netpoll: netconsole: local IPv4 address 0.0.0.0
[   11.677216] netpoll: netconsole: interface 'eth0'
[   11.677287] netpoll: netconsole: remote port 6666
[   11.677356] netpoll: netconsole: remote IPv4 address 192.168.0.2
[   11.677430] netpoll: netconsole: remote ethernet address ff:ff:ff:ff:ff:ff
[   11.677514] netpoll: netconsole: device eth0 not up yet, forcing it
[   15.432381] netpoll: netconsole: no IP address for eth0, aborting
[   15.432540] netconsole: cleaning up

In any case, it might be a good idea to also automate setting a sufficiently high console log level. Passing loglevel=8 on the kernel command line could be a way, or a tiny start-up script calling dmesg -n 8 or updating the /proc/sys/kernel/printk file.

Enjoy tracking down kernel bugs!


Published: Thu, 03 Jan 2019 10:00:00 +0100