Groundhog Day: reflashing system images made easy

6 février 2023

Ce billet n’a pas encore été traduit en français. La version anglaise est disponible ci-dessous.

Foreword: system images

A recurring topic for Debamax is helping customers design and maintain build systems for Debian-based system images. Such images are usually:

• defined by an image size, a partition layout, and various filesystems;
• prepared by using debootstrap then installing customer-specific packages;
• deployed to target device, where some “first boot” logic runs self-tests, registers the device, requests certificates, etc.

The build system prepares those images and also saves metadata after each build (manifests listing packages, versions, files, etc.), making it possible to run consistency checks, compare successive builds, prepare upgrades, etc.

Building an image is usually easy, but deploying it might be a little more tricky:

• One might want to build or use an existing live system that can be booted off from USB, and embed the system image in a specific partition, to be copied over to internal storage from the live system. This approach works on all x86-based systems.
• One might need to extract internal storage from target device, plug it onto a different system, copy data over, and then insert it back. This approach might be required for some ARM-based systems, and might require a specific host configuration, e.g. using a Compute Module 4 IO Board to connect a Compute Module 4, then using usbboot to make its internal storage accessible via USB mass storage.

Those solutions are rather straightforward and usually acceptable when it comes to assembling components when a product is getting manufactured, but they aren’t really acceptable for developers or testers, because they lead to very long feedback loops.

Side-stepping the need to touch hardware

A different approach would be deploying the system image via the network. For Debamax’s use cases, the main question is the following:

Is the target device still accessible and in a sufficiently good shape?

A positive answer would usually imply:

• The device still boots: it’s possible to break the bootloader or the boot sequence entirely.
• The network connectivity is fine enough for the device to be accessible via the local network: a common setup involves a working DHCP client and root access via SSH keypair, but there’s more than one way to do it.
• The package manager is happy to install some extra packages: sometimes, a botched upgrade can leave some broken packages, which would prevent installing other packages.

If all those boxes are ticked, what about running curl | dd or curl | xz | dd directly from the existing system? That would mean downloading the system image from the build server (or from the developer’s machine), possibly decompressing it on the fly, and then writing it to the target storage, without needing to touch any hardware: no live system on USB, no dismantling, no unscrewing, no storage adapter, etc.

The obvious problem is that programs are running, files are open, and the image getting written to storage will likely get corrupted in the process. Instead of trying to close each and every resource, the Groundhog Day approach is to tweak the initramfs!

Groundhog Day

Technical approach

A bootloader is usually configured to boot a Linux kernel (e.g. vmlinuz-5.10.0-21-amd64) alongside an initramfs (e.g. initrd.img-5.10.0-21-amd64). The latter is an archive that contains scripts, binaries, kernel modules, and makes it possible to prepare the root filesystem before passing control over to init (e.g. systemd).

In Debian, initramfs-tools is responsible for creating and updating those archives, based on configuration files and hooks. Some packages integrate with initramfs-tools to extend it. That’s the case for cryptsetup-initramfs, which makes it possible to get passphrase prompts for LUKS-based storage, and for dropbear-initramfs, which makes it possible to integrate a lightweight SSH server into the initramfs, which in turn makes it possible to specify LUKS passphrases over the network.

The idea behind Groundhog Day is to reuse the existing dropbear-initramfs integration, and to tweak it a little so that the initramfs gets a dropbear instance up and running, so that one can log in and do the curl | dd dance before the system actually starts: this way, the image can be deployed without risking any corruption!

The script found below is what we call the “groundhog payload”, which is only one part of a larger tool that automates reflashing (that one is tailored to customer needs and out of scope for this blog post). This payload is meant to be used once on a given system, and to keep it short, it operates on some files that shouldn’t be touched (below /usr). This was deemed acceptable as the whole idea is to replace an existing system with a different image anyway!

Implementation

The groundhog payload is rather generic and works fine with x86-based machines running amd64 and featuring SSD-based storage, as well as with Raspberry Pi Compute Module 3 and Compute Module 4 running arm64 and featuring eMMC-based storage. The Compute Module 3 requires some tweaks when it comes to the networking stack though.

#!/bin/sh
# © 2022-2023 Cyril Brulebois <cyril@debamax.com>
#
# This helps prepare a system to be reflashed via `curl | dd` from the
# initramfs, over SSH.
set -e

codename=$(lsb_release -sc)
case "$codename" in
  buster|bullseye)
    echo "Supported: $codename"
    ;;
  *)
    echo "Unsupported: $codename"
    exit 1
  ;;
esac

# Sanity check:
if [ ! -s /root/.ssh/authorized_keys ]; then
  echo "E: missing or empty /root/.ssh/authorized_keys"
  exit 1
fi

# Configure APT:
SOURCES=/etc/apt/sources.list
URLS="http://internal-mirror.example.com/debian http://deb.debian.org/debian"
: > "$SOURCES"
for url in $URLS; do
  if curl -k -s "$url" >/dev/null 2>&1; then
    echo "I: Adding $url"
    echo "deb $url $codename main" >> "$SOURCES"
  else
    echo "W: Skipping $url"
  fi
done
if [ ! -s "$SOURCES" ]; then
  echo "E: empty $SOURCES"
  exit 1
fi

# Divert u-i to speed up initial installation:
dpkg-divert --divert /usr/sbin/update-initramfs.disabled --rename /usr/sbin/update-initramfs
ln -s /bin/true /usr/sbin/update-initramfs

# Install packages:
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y dropbear-initramfs cryptsetup-

# Trust root's keys:
cp /root/.ssh/authorized_keys /etc/dropbear-initramfs/authorized_keys

# This doesn't seem required on amd64, but let's make sure everywhere:
sed 's/^DEVICE=.*/DEVICE=eth0/' \
    -i /etc/initramfs-tools/initramfs.conf

# The order is important, otherwise the link doesn't get up automatically!
if grep -qs 3-compute-module /proc/device-tree/compatible; then
  cat >/etc/initramfs-tools/modules <<EOF
mii
phy_generic
smsc95xx
EOF
fi

# Drop condition on /etc/crypttab (seen on buster, not on bullseye):
sed '/^\[ -r \/etc\/crypttab \] || exit 0$/d' \
    -i /usr/share/initramfs-tools/hooks/dropbear

# Inject curl:
sed '/^copy_exec.*dropbear/i copy_exec /usr/bin/curl /bin' \
    -i /usr/share/initramfs-tools/hooks/dropbear

# Wait for eth0 to come up before trying to run dropbear, which would
# fail immediately otherwise. Make sure to match on the call, not the
# function definition:
sed '/^run_dropbear /i max=20; for try in $(seq 1 $max); do if [ -e /sys/class/net/$DEVICE ]; then break; fi; if [ "$try" = "$max" ]; then exit 0; fi; sleep 1; done' \
    -i /usr/share/initramfs-tools/scripts/init-premount/dropbear

# Remember MAC address:
mac=$(ip link show dev eth0 | awk '/link\/ether/ {print $2}')
sed "/^run_dropbear /i ip link set address $mac dev eth0" \
    -i /usr/share/initramfs-tools/scripts/init-premount/dropbear

# Add 15-minute delay before trying to mount everything:
echo 'sleep 15m' \
    >> /usr/share/initramfs-tools/scripts/init-premount/dropbear

# Remove diversion:
rm -f /usr/sbin/update-initramfs
dpkg-divert --rename --remove /usr/sbin/update-initramfs

# Rebuilding for the current ABI is sufficient:
update-initramfs -u -k $(uname -r)

Lines	Comments
8-17	The payload modifies various files shipped by Debian packages. They didn’t change much between Debian 10 and Debian 11, but a few tweaks were needed. Small changes might be required for further versions too, so it makes sense to make it clear what is supported and what is not.
25-40	Depending on development vs. production builds, the existing system might be configured to use a specific repository that only knows about the packages that are actually useful for a given product. This part ensures that a full mirror (either internal to the company or a public Debian mirror) is configured, making it possible to install extra packages.
42-44	This saves some I/O by skipping all update-initramfs calls that happen automatically via dpkg triggers when some packages are installed.
46-48	This saves some I/O by installing dropbear without cryptsetup: they are usually useful together but all we need here is dropbear.
50-51	This configures dropbear with root’s authorized_keys.
53-55	Setting the network device might not be necessary but for the avoidance of doubt, this sets eth0 explicitly — these system images use net.ifnames=0 to force the old naming scheme.
57-64	The Compute Module 3 requires some specific modules to be included in the initramfs for the network interface to be usable.
66-68	As mentioned above, dropbear and cryptsetup are usually useful together, and some version of that dropbear hook would exit early if there’s no cryptsetup configuration. This ensures the hook continues.
70-72	This sed call locates where dropbear gets copied into the initramfs via copy_exec, and inserts a line that copies curl. The copy_exec function copies an executable but also the libraries it depends on. If needed, it’s possible to use the same logic to inject tools to tweak partitioning or filesystems within the initramfs (e.g. sfdisk, partprobe, e2fsck, resize2fs).
74-78	This sed call locates where the run_dropbear function is called, and inserts a waiting loop right before: the idea is to give eth0 a little time to come up, otherwise dropbear would exit immediately.
80-83	This is needed for the Compute Module 3, as it doesn’t set the MAC address automatically, which would result in the system’s getting a different IP address via DHCP. Instead of implementing parsing /proc/cmdline, looking for the smsc95xx.macaddr parameter, and using it to set the right MAC address… save the current MAC address from the running system, and restore it from the initramfs.
85-87	With the usual dropbear and cryptsetup integration, the initramfs would wait for filesystems to be unlocked before continuing. In our case, there are no such filesystems that need to be unlocked, so the boot would continue. Sleeping for 15 minutes makes sure users have time to log in, run their curl | dd command, and reboot into the newly-deployed image. The system continues booting if users didn’t connect during that timeframe.
89-94	This reverts the update-initramfs optimization, and builds a single initramfs, for the kernel version that’s currently running.

Usage

The focus of this blog post is really the groundhog payload detailed in the previous section, and how it gets used really depends on developer or tester needs. The following steps should be rather common anyway:

1. copy the payload to the target system, e.g. via scp;
2. run the script and make sure it exits successfully;
3. collect dropbear fingerprints;
4. reboot the target system;
5. connect to the target system (now waiting in its initramfs), using collected dropbear fingerprints;
6. detect the target storage (e.g. /dev/sda vs /dev/mmcblk0);
7. run curl http://server/product-arch.img.xz | xz -d | dd of=/dev/sda-or-mmcblk0;
8. adjust partitioning and/or filesystems if desired;
9. reboot the target.

At this point the target should be running on the freshly-deployed image!

Some tips:

• Fingerprints: Since dropbear uses a specific key format, new fingerprints will show up when connecting to the target system, when it’s waiting in its initramfs. A known_hosts entry can be crafted in advance, e.g. by using the ecdsa-sha2-nistp256 prefix, followed by the output of this command:
  dropbearkey -f /etc/dropbear-initramfs/dropbear_ecdsa_host_key -y
• Deploying the image: It might be a good idea to fetch both image and fingerprint (e.g. image.img.xz and image.img.shasum), to verify what was just written to storage.
• Adjusting partitioning and/or filesystems would usually require shipping more tools in the initramfs (see comments in the previous section about copy_exec). It is common for a system image to feature some “first boot” logic that extends partitions and filesystems to use the whole storage though.
• Rebooting from the initramfs might be tricky: the poweroff or shutdown commands usually work by talking to init, which is not possible at this stage. Using /proc/sysrq-trigger might work depending on the target hardware, but for some ARM-based devices, it might be much easier to use power sockets that can be switched on and off (e.g. EnerGenie EG-PMS2, with the sispmctl package and #929612 fixed).