References

While working on the MIOTY Premium Gateway product, Diehl Metering and Debamax were already planning on reusing as many components as possible for another product: the IZAR IoT GATEWAY Compact.

While the former is Intel-based and the latter is ARM-based, both products share their build system, configuration daemon, upgrade mechanism, and custom components, integrated on top of the same standard Debian distribution. This yields very high levels of synergy and reuse between products.

The main differences between products are the primary RF adapters, and the possible LTE modem. All of those are managed via autodetection, with an extensive self-test run at manufacturing time, making sure detected components match what’s supposed to be present during a given production batch.

Early work on this product also resulted in a number of direct contributions to Debian.

>> See the product page: IZAR IoT GATEWAY Compact

Debamax helped Diehl Metering build a new product, the MIOTY Premium Gateway, by integrating custom components on top of a regular Debian system. This includes setting up a build system, an automatic and secure upgrade mechanism, and a system configuration component to glue everything together.

In addition to building a custom operating system image, Debamax is also responsible for maintaining it over time. This means providing security updates as well as new features, and porting the system image to new major Debian releases, which ensures long term security support.

Debamax also helps with production tools and with monitoring systems. The former means the system image can be easily deployed and fully tested during the manufacturing process. The latter makes it possible to spot possible issues without waiting on customer feedback, and to take corrective measures as soon as possible.

>> See the product page: MIOTY Premium Gateway

After working on freezable APT repositories, Debamax became more involved in the development of the Tails operating system.

One area was the Foundations Team, which is responsible for maintaining the core Tails system. Tasks included preparing for the next major Debian release (as Tails is a Debian derivative) or updating the test suite for the new installation method using USB images.

Another one was Release Management. Tails must be updated on a very regular fashion, making sure known security bugs are fixed as soon as possible. Since Tails ships Tor Browser, itself built on top of Firefox, schedules are usually aligned, meaning a release every 4 weeks (6 weeks initially). Debamax ended up being responsible for two dozen releases during that time.

Since Tails is a live system, its upgrade process involves Incremental Upgrades Kits (IUKs), making it possible to patch an existing system over time, instead of having to redeploy it from scratch. Debamax modified the release process to ensure that release preparations can happen in a timely fashion, despite the ever-growing number of IUKs to build, ensuring parallelism can be used for both local builds and Jenkins builds.

Release Managers are the last safety net before code reaches end users and sometimes it’s best to abort a release entirely. That’s what happened with the 4.2.1 release, after Debamax detected some critical bug in the upgrader.

The initial need was coordinating a Debian upgrade from oldoldstable (Squeeze) to stable (Jessie) across several thousand devices. Initial investigation reported very different configurations, which prompted an iterative approach aimed at getting two sets of devices with similar configurations (amd64 vs. i386) instead of individual snowflakes.

An upgrade path was designed, taking into account upgrade glitches for some packages (in-house developed, or from Debian), which was double checked in virtual machines. After successful tests on bare metal, the upgrade procedure was used on increasingly larger sets of devices in production.

Since then, Debamax has helped plan and execute the migration to every new major Debian release (Stretch, Buster, and Bullseye) in a timely fashion, providing ready-to-use Ansible playbooks.

Debamax has been working on the Tails operating system, both on its building and hosting infrastructures. The goal was to lay the groundwork for reproducible builds for ISO images; this involved setting up snapshots for various Debian repositories, along with adjusting the build system to generate a manifest of packages used during the build.

>> Read more about this mission in this blog post.

The mission was the initial packaging of a web application (Ishtar), based on the Django framework, for Wheezy. After a few bugs were reported and promptly fixed by the upstream authors, a proper separation was introduced between the packaged code and the logic behind the creation of an instance. Related tasks included submitting initial packaging for a few dependencies which weren’t available in the Wheezy distribution.

>> Read more on their websites (in French): Iggdrasil — Ishtar.

Debamax has set up a CI platform for Wifirst, based on the jenkins and jenkins-debian-glue components. It allows for automatic builds of Debian packages for various in-house components, and for making them available through an APT repository. Notifications by mail were configured to let the development team know when build failures are detected, and when they are resolved.

To meet Wifirst’s needs, a set of wrappers was developed to make it possible to ship builds from tags and builds from branches through different suites, allowing for an easy switch between release and development packages.

Debian stable releases don’t change much over time, but some updates happen through point releases every few months, where packages already published through the security archive and those staged into proposed-updates are merged into the stable suite. While such updates are usually small, targeted, and well reviewed, they can sometimes generate extra work or disrupt ongoing maintenance operations.

A solution based on reprepro was set up, making it possible to generate a snapshot of an upstream repository at a given time, allowing for extra control. It was chosen to restrict the mirrored packages to a specific list, limiting the need for disk space.

This solution comes with a wrapper to help track differences between snapshots, making it easier to determine preventively which changes are to be expected when moving from an earlier snapshot to a newer one, and to determine afterwards which updates might have triggered a regression.