RSS

blog

Explore a single year: 2015 2016 2017

Tails: early work on reproducibility

Tails logo

Quick introduction about Tails and build reproducibility

Tails is a live operating system, booting from USB or from DVD, aiming at preserving user’s privacy and anonymity. It is Free Software and based on the Debian GNU/Linux distribution. The Tails website contains a more complete overview of the project.

Over the last few years, an increasing number of software developers and security-focused people have been looking into build reproducibility: when a build system is deterministic, building a given component from a given source should always lead to the exact same binary result (byte-for-byte). The Tor project and the Debian distribution were among the first teams to work towards this goal. More information can be found on the reproducible-builds.org website, whose motto is Provide a verifiable path from source code to binary.

What does it mean in the Tails context? The main “product” of the Tails project is a bootable ISO image which contains the live system containing tools designed and preconfigured to preserve privacy and anonymity. Compromising this image would defeat the whole point of the project and could even endanger lives of journalists or whistleblowers relying on it. Making the image build process reproducible means developers and even users can reproduce it on their own hardware and make sure the ISO image published by the Tails project matches the one which was built locally, or which was verified by others.

How Debamax became involved

Flashback: October 2015.

Cyril had already been working on Debian derivatives for various customers and had been identified by some Tails developers as a potential asset to work on the first steps towards reproducibility. A sprint approach was chosen to tackle the freezable APT repository topic: meet, discuss, design, code; repeat a few times.

What follows is an overview of the results, with a few pointers to code and documentation. They are presented sequentially but all those topics are closely intertwined, and that had to be taken into account during the design phase.

Keeping track of packages in archives

The first objective was to imagine a workflow which would make it possible to build a given ISO image with the exact same set of Debian packages. An interesting data point is that 4 separate archives are used during a Tails build:

Of course, those archives aren’t static: the Debian archive is updated up to 4 times a day, the Debian Security archive is updated whenever a new security update is published, etc. So we needed a way to keep track of all packages used during the build but also of the state of each archive at any point where an image was being built.

It was decided to use reprepro, which is designed to produce custom Debian repositories, while also making it possible to mirror upstream repositories. It also allows to create snapshots, which exactly fits the need to keep packages around! Packages which would normally be deleted or replaced by a new version (when a synchronization happens) are kept as long as there’s at least one snapshot that depends on them.

First results:

Keeping track of packages used during the build

While working for other customers, Cyril already had to keep track of packages used to build Debian images: the idea was to list all packages and versions used for a given build, making it possible to generate changelog-like summaries of changes between two builds.

A similar approach was used here, where triplets are gathered with: package, version, URI. Here’s what the implementation looks like:

That means three files are generated with those triplets: one with binary packages from the bootstrap phase, one with binary packages downloaded through apt-get, and one with source packages downloaded through apt-get as well.

Another script was developed to aggregate those results into what we call a build-manifest; it gathers all origins (the archives mentioned in the previous section), their references (the snapshot used during the build), and all packages along with their versions. Example for the 3.2 release: tails-amd64-3.2.build-manifest.

Keeping track of packages in the long term

At this point we have the following results:

Keeping all packages forever wouldn’t be reasonable, so snapshots are expired after a few days. Since storing packages actually used for releases is the whole point of mastering repositories in the first place, an extra tool on the infrastructure side was developed to generate tagged snapshots from the time-based ones, thanks to references and packages listed in the build-manifest for the release.

This leads to these results:

Putting all the pieces together

Fastforward: November 2017.

Large parts of this initial freezable APT repository sprint were spent designing what the new workflow would look like during development phases, and during freeze periods. Of course, adjustments were made during the following releases, and the current status is documented on the APT repository page. Details can be found there about the custom APT repository (for Tails), about the time-based snasphots, and about the tagged snapshots.

This was only preliminary work, as there are many reasons which can trigger differences in the resulting ISO image. Details can be found in the reproducible builds blueprint. Many issues have been tackled by the Tails developers since then, and that’s how the 3.3 release has been announced as the first reproducible ISO image! (Of course, this is still rather new, and bug #14933 has been filed already, but the current results are amazing already!)

Congratulations to the Tails developers for reaching this milestone, and many thanks for this cooperation opportunity!


Published: Fri, 08 Dec 2017 10:15:00 +0100

Debian Installer: Stretch released

Debian Installer: Stretch

Foreword

Since the previous post, several Debian Installer release candidates were published, and this post sums up everything that happened between the Debian Installer Stretch RC 2 release and the final Stretch release.

Stretch RC 3

Cyril published the Debian Installer Stretch RC 3 release on 2017-04-10, roughly two months after Stretch RC 2.

Improvements

A number of fixes piled up since then, including the following important changes:

Other changes can be found in the release annoucement.

Here are a few screenshots (click for full view) illustrating the Korean rendering issues follow, so that one can visualize the impact of a font issue:

Stretch D-I RC 2 (broken) Stretch D-I RC 3 (fixed)
Broken Korean rendering in language selection screen  
 
Fixed Korean rendering in language selection screen
 
Entirely broken Korean screen Fixed Korean screen

Hardware support

The full list of hardware-related changes is reproduced below:

Stretch RC 4

With the amount of changes in Stretch RC 3, and the Stretch release date approaching (2017-06-17), it was expected to have a smaller number of changes in the Stretch RC 4 release. Thankfully, that's what happened by the end of May (2017-05-27), with most changes being translations updates: the number of full translations saw a bump from 15 to 21.

Improvements

Hardware support

Stretch RC 5

This Debian Installer release happened on 2017-06-13, only a few days before the final Debian release, planned on 2017-06-17. There were still quite a number of changes to merge, and only those with the most visible impact are listed below.

Improvements

Hardware support

Stretch final

Usually, one would use the same debian-installer upload for the last release candidate and for the final release, but Cyril asked the linux maintainers to merge a last change before the release: It started to become clear in early June that the missing i2c-modules udeb on the armhf platform was the likely cause for several issues (#864536, #864457, #856111).

Performing uploads, builds, and unblocks of linux, debian-installer, and debian-installer-netboot-images before the final release wasn't entirely stressless, but it seemed worth trying. Adding new binary packages in point releases is a very rare event, and going through the NEW queue via unstable looked like the right thing to do, even if the timing was very tight!

What's next?

The next Debian Installer report will likely feature a summary of installer-related changes merged into the 8.9 and 9.1 point releases (for Jessie and Stretch respectively).

Also, Cyril will be giving a talk titled “News from the Debian Installer” during DebConf17. This year, the annual Debian Conference takes place in Montreal, Canada (more info is available on the DebConf17 schedule). See you there?


Published: Sat, 05 Aug 2017 12:00:00 -0400

Debian Installer: Stretch RC 2 released

Debian Installer: Stretch RC 2

Foreword

Since the previous blog post, two Debian Installer release candidates were published, so both will be mentioned in this blog post.

Stretch RC 1

As mentioned in the Plans section of the Stretch Alpha 8 summary: with the full freeze coming up, it made sense to switch from the Alpha numbering to the Release Candidate one. That’s why Cyril published the Debian Installer Stretch RC 1 release on 2017-01-15.

Unfortunately, some blockers were found with merged-/usr setups, so the new debootstrap default was reverted. Even if some of these bugs were fixed in the meanwhile, it seemed unreasonable to enable the new code again near the end of the stretch release cycle, so it’s going to be postponed until after the buster release cycle has started.

Here is a list of other changes:

Stretch RC 2

Since the Linux kernel team was finally moving towards the target kernel version for Stretch (4.9, even if earlier discussions mentioned 4.10), it seemed like a good idea to get a new Debian Installer released as soon as possible, which explains why Debian Installer Stretch RC 2 was released on 2017-02-02, only a few weeks after Stretch RC 1.

Another significant change happened besides the Linux kernel update, with the os-prober component receiving major changes. Let’s have a look at its description:

    Package: os-prober
    Description: utility to detect other OSes on a set of drives
     This package detects other OSes available on a system and outputs the
     results in a generic machine-readable format.
  

This component is used to determine which other operating systems might be hanging around on various partitions and discs, and it’s used e.g. by update-grub to include menu entries for other Linux distributions, Windows, etc. Unfortunately, its historical operating system detection code has been triggering issues in some environments involving virtualization, which ended up in data loss in some cases.

The relevant code was heavily overhauled, and one might hit some regressions with new versions of this component (1.72 and later). Details about these significant changes can be found in the changelog entry for the 1.72 upload, and one might notice that the preparations for this new release candidate resulted in a last-minute regression fix in the 1.74 upload. Similar issues have been reported with the dmsetup create command hanging, leading to a frozen progress indicator when grub is being set up (see bug report #853927). This can be worked around by switching to a console and killing the dmsetup process (see this message for more details), until this issue is fully diagnosed and fixed.

Next release candidate

With the full freeze in effect, Debamax is trying to make sure Cyril can spend as much time as possible on two complementary tasks:

More to come in our next Debian Installer report!


Published: Mon, 13 Feb 2017 13:30:00 +0100

Debian Installer: Stretch Alpha 8 released

Debian Installer: Stretch Alpha 8

Release process

It took a few months after Stretch Alpha 7 (published 2016-07-04), but the Stretch Alpha 8 release of the Debian Installer happened a few days ago. Release preparations had to be delayed a bit because a fix was needed in the linux packaging (see bug report #839552) so that mounting FAT partitions worked again, since this is needed for EFI support.

As a release manager, Cyril has to make sure things look good enough for a release. This usually involves freezing udeb-producing packages for a while, so that the main set of packages used to build the Debian Installer doesn’t get any last minute changes that might bring some regressions while stabilization is in progress.

The debian-installer package got uploaded on 2016-10-27 but two major issues popped up:

The first issue was due to the rather recent linux/linux-signed split. The idea behind this move is preparing for Secure Boot support, with linux being used to build linux kernel and modules as usual, and linux-signed holding extra signatures for them, so that they can be verified cryptographically. This exposed an awful and old bug which hadn’t been detected until now. Then an extra commit got added as a work around for the linux/linux-signed specific situation: code comes from the linux source package, so that’s what needs to be listed in Built-Using. Further improvements are planned (see bug report #842719), by checking for a possible Built-Using field in each udeb, so that this workaround can be replaced by some more generic code.

The second issue was due to the reintroduction of InRelease support. There are two ways of validating the contents of a given distribution on a Debian mirror: checking the Release file against its detached signature (Release.gpg), or checking the InRelease file alone, as it contains an inline signature. Since only gpgv is available in a Debian Installer environment, the idea was to split the InRelease file into two files: the Release file and its signature. The tricky part is that the final newline is dropped by GnuPG, so a little tr … | sed … | tr … dance was added to do the same. Unfortunately, while it works fine with usual implementations of those commands, that’s not the case with the busybox implementation used in Debian Installer, leading to a bad signature result during the installation process (see bug report #842591). Thankfully Ansgar Burchardt had a proof of concept ready with a simple state machine in POSIX shell, which Cyril could merge and upload to fix debootstrap-udeb, fixing this showstopper.

Major update: debootstrap and merged-/usr

As mentioned above, debootstrap was updated, but not only for InRelease support. It received a number of fixes and improvements (see the release announce for the details), but the biggest change deserves a longer explanation: debootstrap now defaults to merged-/usr.

Once upon a time, UNIX systems were booted from a floppy disk, and once the boot sequence had finished, one would mount extra resources onto the /usr directory: programs, libraries, home directories, etc. Nowadays, it makes little sense to keep the distinction between boot-time and non-boot-time tools, and it was proposed to get rid of this distinction entirely. One way to achieve this is as simple as setting up symlinks for a number of directories: bin, sbin, lib, and other libXX (one can find lib32, lib64, etc. depending on the architecture), respectively pointing at usr/bin, usr/sbin, usr/lib, etc. This approach means there’s no need to change any single package, it’s just about using a specific directories+symlinks setup at installation time.

The options to enable or disable this feature are --merged-usr and --no-merged-usr respectively. The Debian script (shared across many versions) was updated to default to merged-/usr for stretch and later, which explains why this Debian Installer Stretch Alpha 8 release now defaults to a merged-/usr setup.

Credits: This change was driven by both Marco d’Itri and Ansgar Burchardt, while Julien Cristau worked on most other changes. Thanks!

Some final notes:

Next release: Stretch Alpha 9

A few things are planned for the next release:


Published: Tue, 22 Nov 2016 00:08:00 +0100

Hello, World!

Hello, World!

Debamax SAS has been successfully registered with the Trade and Company Register in Rennes, and has officially started operating in October! The legal notices page has further information regarding this registration and identification numbers.

A Twitter account (@DEBAMAX) is going to be set up to complement this website and its RSS feed.


Published: Wed, 7 Oct 2015 12:00:00 +0200