<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="3.9.1">Jekyll</generator><link href="https://debamax.com/blog.xml" rel="self" type="application/atom+xml" /><link href="https://debamax.com/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-11-06T05:12:25+01:00</updated><id>https://debamax.com/blog.xml</id><title type="html">DEBAMAX | blog</title><subtitle>DEBAMAX is a Free Software consulting company, providing its expertise on the Debian distribution.</subtitle><entry xml:lang="en"><title type="html">PiRogue Tool Suite: the Debian 13 edition</title><link href="https://debamax.com/blog/2026/11/05/pirogue-tool-suite-the-debian-13-edition/" rel="alternate" type="text/html" title="PiRogue Tool Suite: the Debian 13 edition" /><published>2026-11-05T23:00:00+01:00</published><updated>2026-11-05T23:00:00+01:00</updated><id>https://debamax.com/blog/2026/11/05/pirogue-tool-suite-the-debian-13-edition</id><content type="html" xml:base="https://debamax.com/blog/2026/11/05/pirogue-tool-suite-the-debian-13-edition/"><![CDATA[<p>Previous episodes:</p>

<ul>
  <li><a href="/blog/2022/06/21/helping-with-pirogue-tool-suite/">Helping with PiRogue Tool Suite</a></li>
  <li><a href="/blog/2024/10/31/pirogue-tool-suite-the-debian-12-edition/">PiRogue Tool Suite: the Debian 12 edition</a></li>
</ul>

<h1>Initial work</h1>

<p>Since Debian gets a new stable release roughly every two years, with Debian 13
released in August 2025, the beginning of 2026 looked like a good time to try
catching up. That’s why <span class="small-caps">Debamax</span> and
<a href="https://defensive-lab.agency/">Defensive Lab Agency</a> got together for a
<a href="https://pts-project.org/">PiRogue Tool Suite</a> sprint in April.</p>

<h2>PiRogue images for Raspberry Pi</h2>

<p>Even if the PiRogue project started on Raspberry Pi hardware, the
<a href="/blog/2024/10/31/pirogue-tool-suite-the-debian-12-edition/#virogue-project">“ViRogue project”</a>
lifted that requirement back in 2024. Feedback seems to confirm that users are
now deploying PiRogue on various hardware, even sometimes migrating away from
Raspberry Pi hardware that was being used previously.</p>

<p>That being said, a number of Pi 4 and Pi 5 deployments are still in service, so
it makes sense to keep on providing PiRogue images for Raspberry Pi. The
<code class="language-plaintext highlighter-rouge">pirogue-images</code> approach (also explained in the previous blog post) didn’t need
many changes to build Debian 13 images instead of Debian 12 images: fetching an
<code class="language-plaintext highlighter-rouge">arm64</code> Debian image for Raspberry Pi, and tweaking it a little to match
PiRogue’s needs.</p>

<p>In the Debian 12 case, there was really no hope of using the mainline Linux
kernel (as shipped in Debian) since it entirely lacked Raspberry Pi 5 support,
which is why we resorted to building separate, experimental Pi 5 images that
would embed the heavily modified Raspberry Pi kernel instead.</p>

<p>Since then, even if Debian 13 still doesn’t ship a Linux kernel supporting the
Pi 5, the mainline Linux kernel received a number of commits adding basic
support for the Pi 5. That’s why <span class="small-caps">Debamax</span> experimented with building a
Pi 5 image using a backported Linux kernel (from <code class="language-plaintext highlighter-rouge">trixie-backports</code>), which
meant switching from the <code class="language-plaintext highlighter-rouge">v6.12.y</code> series (found in Debian 13) to the <code class="language-plaintext highlighter-rouge">v6.19.y</code>
series (found in Debian <code class="language-plaintext highlighter-rouge">unstable</code>, <code class="language-plaintext highlighter-rouge">testing</code>, and <code class="language-plaintext highlighter-rouge">trixie-backports</code> suites at
the time).</p>

<p>The experiment ended up being totally successful!</p>

<p>The resulting image deployed on a Pi 5:</p>

<ul>
  <li>was booting (that seems very basic, but that’s never a given when it comes to
specific hardware);</li>
  <li>was accessible over Ethernet;</li>
  <li>made it possible to deploy the <code class="language-plaintext highlighter-rouge">pirogue-base</code> package that pulls everything
needed for a PiRogue environment, including configuring the Wi-Fi access
point.</li>
</ul>

<p>Connecting a phone to that network confirmed that traffic analysis was also
working as expected, with Suricata alerts popping up when suspicious activity
was detected, etc.</p>

<h2>NFStream</h2>

<p>That package could have been a topic when we switched to Debian 12 in 2024: it
needs to be built for both architecture (<code class="language-plaintext highlighter-rouge">amd64</code> and <code class="language-plaintext highlighter-rouge">arm64</code>), and the generated
binary packages end up depending on some native dependencies like
<a href="https://sourceware.org/libffi/"><code class="language-plaintext highlighter-rouge">libffi</code></a>. A rebuild (with or without changes)
might have been needed when we switched from PiRogue 1.x versions (based on
Raspberry OS, itself based on Debian 11 at the time) to PiRogue 2.x versions
(based on Debian 12), but we were lucky as the existing packages were still
installable dependency-wise, and working fine.</p>

<p>In the context of this “let’s upgrade all the things!” sprint, it made sense to
see if we could upgrade NFStream as well:</p>

<ul>
  <li>old version: 6.3.5~ds3, with packages built in June 2023, and with an
upstream version dating back to October 2021!</li>
  <li>new version: 6.6.0~ds1, with packages built in April 2026, and with an
upstream version dating back to February 2026 (still current as the time of
writing).</li>
</ul>

<p>This package was a little challenging initially as it required a number of build
dependencies (as usual), but also a set of specific versions, branches, and/or
patches for 4 different libraries. In the first iterations of the Debian
packaging, those had been “vendorized” to make sure we could build a proper
source package, which could be built cleanly (e.g. in disposable chroot
environments). Fast-forward a few years, installation instructions have been
cleaned up quite a lot, with two external components disappearing entirely, and
the other two being referenced as Git submodules.</p>

<p>That was reflected in the Debian packaging, which still needed to implement
fixes or workarounds for the build system: trivial things like not writing
outside the source directory (e.g. to <code class="language-plaintext highlighter-rouge">/tmp</code>…) or enabling parallel builds; but
also less trivial things like making sure the Debian tooling for Python packages
wouldn’t rename the C extension, as that would break importing the <code class="language-plaintext highlighter-rouge">nfstream</code>
Python module.</p>

<h2>Frida</h2>

<p>Looking back, automation already looked like an important topic in 2024, and
that’s been confirmed with even more numerous releases over the last few years:</p>

<ul>
  <li>2022: 26 releases;</li>
  <li>2023: 22 releases;</li>
  <li>2024: 32 releases;</li>
  <li>2025: 69 releases;</li>
  <li>2026: 14 releases at the time of the sprint (April), 62 releases at the time
of writing (early November).</li>
</ul>

<!-- for y in {2022..2026}; do printf "$y "; git log --decorate github/main --since "1/1/$y" --until "31/12/$y"|grep tag:|grep -v debian/|wc -l; done -->

<p>Regarding the <a href="https://github.com/PiRogueToolSuite/deb-frido/">Frido</a>-based
automation, some documentation was added on top of the initial implementation in
2024, but no changes were required regarding:</p>

<ul>
  <li>either the high-level
<a href="https://github.com/PiRogueToolSuite/deb-frido/blob/master/frido/builds.py">steps</a>
that are used to automatically update the packaging branch for a given
version, dealing with source generation, binary builds, binary checks, and
binary publication (to a staging repository);</li>
  <li>or the low-level
<a href="https://github.com/PiRogueToolSuite/deb-frido/blob/master/frido/actions.py">actions</a>
which implement merging a given upstream version, defining a Debian revision
on top of it, making sure all patches still apply, creating a tentative tag,
and triggering actual builds for both <code class="language-plaintext highlighter-rouge">amd64</code> and <code class="language-plaintext highlighter-rouge">arm64</code> architectures.</li>
</ul>

<p>Unfortunately, since <span class="small-caps">Debamax</span> was busy with other customers, automatic
builds stopped in June 2025, after successfully building binary packages for 15
versions that year. Upstream changes required adjusting the patch series applied
through the Debian packaging, and that was only looked at during the April 2026
sprint. Once the required changes were applied, automated builds resumed and the
catch-up was massive: 49 versions over 3 days! And the automation has been green
ever since!</p>

<h2>Changes required or planned for Debian 13</h2>

<ul>
  <li>Stricter package manager: APT delegates GPG signature verification to a tool
called Sequoia (shipped in the <code class="language-plaintext highlighter-rouge">sqv</code> package), and this was known to be a
critical topic. After a flag day
(<a href="https://salsa.debian.org/apt-team/apt/-/commit/153653d449efe121487641303d43ac523b9b17fb">initially set to 2026-01-01</a>,
then
<a href="https://salsa.debian.org/apt-team/apt/-/commit/0989275c2f7afb7a5f7698a096664a1035118ebf">pushed back to 2026-02-01</a>),
signatures from keys involving SHA-1 would be rejected. One way to address
this topic is to generate new signing keys with higher security standards.
Another way is to follow the Sequoia project’s
<a href="https://book.sequoia-pgp.org/lint.html">migration guide</a>, which makes it
possible to “upgrade” an existing signing key. Cyril followed that guide
months in advance (that problem had been anticipated with other customers),
and the remaining steps were rather easy: make sure the <code class="language-plaintext highlighter-rouge">pirogue.gpg</code> keyring
file that’s made available in the
<a href="https://pts-project.org/debian-12/">Debian 12 PPA</a> and now the
<a href="https://pts-project.org/debian-13/">Debian 13 PPA</a>, as well as shipped by
the <code class="language-plaintext highlighter-rouge">pirogue-archive-keyring</code> package, was refreshed with the updated signing
key.</li>
  <li>The PiRogue ecosystem involves many packages, and some of them considered to
be of lower priority were dropped: <a href="https://www.mitmproxy.org/">mitmproxy</a>,
<a href="https://docs.mvt.re/">Mobile Verification Toolkit (MVT)</a>.</li>
  <li>Changes in Python 3.13 broke VPN support in <code class="language-plaintext highlighter-rouge">pirogue-admin</code>, which was
<a href="https://github.com/PiRogueToolSuite/pirogue-admin/commit/2b66992a9b04f490a75d4528e98a7a1b40a62cf5">easily fixed</a>.</li>
</ul>

<h2>Post-sprint status</h2>

<p>The <span class="small-caps">Debamax</span>/Defensive Lab Agency sprint was considered a success, since
the following could be confirmed:</p>

<ul>
  <li>fixed automated builds for Frida, leading to an empty build queue;</li>
  <li>working PiRogue images for Raspberry Pi, fully tested on Pi 4 and Pi 5;</li>
  <li>working PiRogue confirmed on VPS as well, in the VPN operating mode.</li>
</ul>

<h1>Finalizing the release</h1>

<p>Fast-forward to October 2026, the PiRogue development schedule included the
switch to Debian 13, so it was time to finalize the first release of the 3.x
series.</p>

<p>What we expected was “let’s pick things up where we left off”. Unfortunately,
that wasn’t the case, at all.</p>

<p>Let’s list what needed to be done (possibly going back and forth between
topics):</p>

<ul>
  <li>make sure PiRogue images were working fine;</li>
  <li>decide which Frida version to ship (many new versions became available in
April, and many more piled up since then);</li>
  <li>prepare official releases for the few packages that had been patched in
April;</li>
  <li>check what happened in the
<a href="https://pts-project.org/debian-12/">Debian 12 PPA</a> between April and
October, and make sure any changes were carried over to the
<a href="https://pts-project.org/debian-13/">Debian 13 PPA</a>.</li>
</ul>

<h2>Linux regression</h2>

<p>Unfortunately, we got bad news with the very first topic, as the official
PiRogue image (for Raspberry Pi 3 and Raspberry Pi 4) was still working fine,
but the experimental PiRogue image (for Raspberry Pi 5) was no longer booting.</p>

<p>Since the main difference between both images is about shipping a <code class="language-plaintext highlighter-rouge">linux-image</code>
package from <code class="language-plaintext highlighter-rouge">trixie-backports</code> instead of picking it from <code class="language-plaintext highlighter-rouge">trixie</code> or
<code class="language-plaintext highlighter-rouge">trixie-security</code>, it seemed clear there might be a regression in the Linux
kernel between the version available in April and the one available in October.
The initial investigation confirmed that two upstream series are suitable for
our needs (<code class="language-plaintext highlighter-rouge">v6.19.y</code> and <code class="language-plaintext highlighter-rouge">v7.0.y</code>), while the next two upstream series are not
(<code class="language-plaintext highlighter-rouge">v7.1.y</code> and <code class="language-plaintext highlighter-rouge">v7.2.y</code>).</p>

<p>The exact reason for this regression hasn’t been pinpointed just yet, but
further investigation
(<a href="https://github.com/PiRogueToolSuite/pirogue-images/issues/9">tracked as <code class="language-plaintext highlighter-rouge">pirogue-images#9</code></a>)
will hopefully lead to bug reports where appropriate (in Debian’s bug tracking
system and/or in the Linux kernel’s Bugzilla).</p>

<p>As a temporary measure, it was determined that building the experimental image
for Raspberry Pi 5 using the latest suitable Linux package from
<code class="language-plaintext highlighter-rouge">trixie-backports</code> would be an acceptable solution for the upcoming 3.0.0
release, and that’s been implemented with the help of
<a href="https://snapshot.debian.org/">https://snapshot.debian.org/</a>:</p>

<ul>
  <li><a href="https://github.com/PiRogueToolSuite/pirogue-images/commit/baf4f34855c9d45441459058eda851a98e6bc990">the initial commit</a>
fundamentally brings us back to the sprint’s status, at least kernel-wise
(<code class="language-plaintext highlighter-rouge">v6.19.y</code> series);</li>
  <li><a href="https://github.com/PiRogueToolSuite/pirogue-images/commit/c021c0b590ca2c8040a15a41a7c95d52809ed1e0">this extra commit</a>
ensures that the very first <code class="language-plaintext highlighter-rouge">apt-get dist-upgrade</code> doesn’t deploy a
known-broken Linux kernel;</li>
  <li><a href="https://github.com/PiRogueToolSuite/pirogue-images/commit/c7c22d3601358c4e5b9fb867840f3ec7e06d6886">this last commit</a>
switches series to the latest suitable one (from <code class="language-plaintext highlighter-rouge">v6.19.y</code> to <code class="language-plaintext highlighter-rouge">v7.0.y</code>),
confirmed with a full PiRogue deployment.</li>
</ul>

<h2>Frida</h2>

<p>With the many builds available, we still had to answer one important question:
which version to pick. During the PiRogue 2.x life cycle, it was decided to
stick with Frida 16.x versions when incompatibilities were discovered in 17.x
versions (e.g. the <code class="language-plaintext highlighter-rouge">socketTrace</code> method disappeared).</p>

<p>After investigation, the lead PiRogue developer reported the latest versions
(<code class="language-plaintext highlighter-rouge">17.25.*</code>, as of October/November 2026) wouldn’t work for PiRogue’s needs, and
that we would be using <code class="language-plaintext highlighter-rouge">17.17.0</code> (from August 2026).</p>

<p>Even with that important question answered, we would still have work to do as
some Frida components were found to be completely broken. This was traced back
to missing declarations in the upstream build system, which was easy to patch
locally, twice:</p>

<ul>
  <li><a href="https://github.com/PiRogueToolSuite/deb-frida/commit/498301420b76c2cbc2f7a8ef666a713a2e49560c">once in the <code class="language-plaintext highlighter-rouge">debian/bookworm</code> branch</a> which is used to build the binary
packages we publish in our PPA;</li>
  <li><a href="https://github.com/PiRogueToolSuite/deb-frida/commit/a210927620b859bf89245cec813965d0704ecc0c">once in the <code class="language-plaintext highlighter-rouge">auto/debian/bookworm</code> branch</a>
which is used to track new upstream release, and which is mainly updated
automatically via Frido; this was used to publish a fixed package for the
<code class="language-plaintext highlighter-rouge">17.23.3</code> upstream release (versioned <code class="language-plaintext highlighter-rouge">17.23.3~pirogue2</code>), in addition to
ensuring any later upstream releases wouldn’t be affected by the same issue.</li>
</ul>

<p>Of course, the problem and the solution were also reported upstream, through the
<a href="https://github.com/frida/frida-tools/pull/207"><code class="language-plaintext highlighter-rouge">frida-tools#207</code> pull request</a>
which was just merged! This means that the <code class="language-plaintext highlighter-rouge">17.26.0</code> upstream release cannot be
built automatically anymore, as the patch series no longer applies cleanly. That
boomerang effect is slightly annoying, but that’s totally normal when being a
good <abbr title="Free/Libre and Open Source Software">FLOSS</abbr> citizen:
implement a solution locally, notify upstream, then clean things up when
upstream integrates that solution (or another one).</p>

<h2>PiRogue packages</h2>

<p>Comparing the contents of the <code class="language-plaintext highlighter-rouge">debian-12</code> and <code class="language-plaintext highlighter-rouge">debian-13</code> PPAs made it possible
to establish a list of packages to release and a list of packages that can be
removed.</p>

<p>This took
<a href="https://github.com/PiRogueToolSuite/debian-13/commits/main/?since=2026-10-20&amp;until=2026-11-04">a number of commits</a>,
but that was finalized early November 2026.</p>

<h1>Conclusion</h1>

<p>Some more PiRogue deployment tests were performed:</p>

<ul>
  <li>starting from the official PiRogue image, on a Raspberry Pi 4;</li>
  <li>starting from the experimental PiRogue image, on a Raspberry Pi 5;</li>
  <li>starting from a freshly installed Debian 13 system on a VPS hosted by
Scaleway.</li>
</ul>

<p>Regarding operating modes, the first two tests cover the “access point” one,
while the last test covers the “VPN” one. There was no explicit test of the
“appliance” one, but since that’s basically a simplified version of the “access
point” one (there’s no Wi-Fi access point to manage), we’re crossing fingers and
hoping for the best!</p>

<p>Go grab the release while it’s hot!</p>

<ul>
  <li>Raspberry Pi images:
<a href="https://github.com/PiRogueToolSuite/pirogue-images/releases/tag/arm64_v3.0.0">arm64_v3.0.0 release</a>.</li>
  <li>Other systems: until
<a href="https://pts-project.org/docs/pirogue/installation/">installation instructions</a>
are updated to reflect the move from Debian 12 to Debian 13 as the supported
operating system, feel free to follow the exact same instructions, replacing
every occurrence of <code class="language-plaintext highlighter-rouge">debian-12</code> with <code class="language-plaintext highlighter-rouge">debian-13</code>!</li>
</ul>

<p>Even if <span class="small-caps">Debamax</span> is only working on PiRogue from time to time (as
opposed to PiRogue developers working on this project all year long), it’s
definitely a project that we really care about. It should be quite clear we’re
talking about some long-term commitment, as it’s been four years already since
joining the project!</p>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[Time to catch up with new upstream releases! Linux kernel, Frida, NFStream, and of course the base distribution: Debian. Plus updated PiRogue packages of course.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/pirogue-logo.png" /><media:content medium="image" url="https://debamax.com/pirogue-logo.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">PiRogue Tool Suite: the Debian 12 edition</title><link href="https://debamax.com/blog/2024/10/31/pirogue-tool-suite-the-debian-12-edition/" rel="alternate" type="text/html" title="PiRogue Tool Suite: the Debian 12 edition" /><published>2024-10-31T12:00:00+01:00</published><updated>2026-11-03T15:00:00+01:00</updated><id>https://debamax.com/blog/2024/10/31/pirogue-tool-suite-the-debian-12-edition</id><content type="html" xml:base="https://debamax.com/blog/2024/10/31/pirogue-tool-suite-the-debian-12-edition/"><![CDATA[<p>In case you missed it, the
<a href="/blog/2022/06/21/helping-with-pirogue-tool-suite/">Helping with PiRogue Tool Suite</a>
blog post presented what <a href="https://pts-project.org/">Pirogue Tool Suite</a> is and
why <span class="small-caps">Debamax</span> decided to start supporting this project.</p>

<p>This blog post covers further work that happened in 2024:</p>

<ul>
  <li>keeping up with upstream projects, including the switch to Debian 12
(Bookworm);</li>
  <li>making the PiRogue Tool Suite independent from Raspberry Pi hardware.</li>
</ul>

<h1>Switching to Debian 12 (Bookworm)</h1>

<p>Initially, the project was publishing PiRogue images built using the same tool
as the Raspbian/Raspberry OS project, maintaining a PiRogue-specific branch
dealing with the few tweaks required for PiRogue (like setting up the custom
repository).</p>

<p>In 2023, the approach was changed to work on Raspbian-published images instead,
downloading them and modifying them with a few scripts to produce PiRogue
images. At the time, the input image was the “Raspberry Pi OS Lite” one, for the
<code class="language-plaintext highlighter-rouge">arm64</code> architecture.</p>

<p>In 2024, with the upcoming ViRogue project in mind (detailed in the second part
of this blog post), the feasibility of moving from Raspberry OS to Debian as the
base distribution for PiRogue was discussed. At the time, target hardware was
limited to Raspberry Pi 3 and Raspberry Pi 4. Since those had been fully
supported by the mainline Linux kernel (found in Debian and other distributions)
for a number of releases, switching away from the specialized Raspberry OS
distribution didn’t seem worrisome. The PiRogue-specific HAT could have been a
little bit of a challenge, but thankfully <span class="small-caps">Debamax</span> is quite familiar
with hardware integration on Raspberry Pi devices, including DTS/DTB-related
fun.</p>

<p>All Debian packages were reviewed with two goals in mind:</p>

<ul>
  <li>spotting anything that would be specific to Raspberry OS, and adjusting it to
work with Debian instead;</li>
  <li>making sure those packages would work with Debian 12, as the Raspberry OS in
use at the time was itself based on Debian 11.</li>
</ul>

<p>This meant having to work on several fronts in parallel:</p>

<ul>
  <li>new <code class="language-plaintext highlighter-rouge">debian-12</code> branches in all active Git repositories;</li>
  <li>brand new <a href="https://pts-project.org/debian-12/"><code class="language-plaintext highlighter-rouge">debian-12</code> Debian repository</a>,
separate from the <a href="https://pts-project.org/ppa/">historical PPA</a>, to make
sure existing systems wouldn’t pull work-in-progress packages;</li>
  <li>brand new
<a href="https://github.com/PiRogueToolSuite/pirogue-images/"><code class="language-plaintext highlighter-rouge">pirogue-images</code> repository</a>
which would be used to build Debian-based PiRogue images.</li>
</ul>

<p>Initially, the tooling shipped in the <code class="language-plaintext highlighter-rouge">pirogue-images</code> repository was used to
prepare official PiRogue images supporting Raspberry Pi 3 and Raspberry Pi 4 as
usual, but that was extended to build experimental PiRogue images supporting
Raspberry Pi 5 (see
<a href="https://github.com/PiRogueToolSuite/pirogue-images/blob/main/TODO.md"><code class="language-plaintext highlighter-rouge">TODO.md</code></a>
and
<a href="https://github.com/PiRogueToolSuite/pirogue-images/issues/2"><code class="language-plaintext highlighter-rouge">pirogue-images#2</code></a>
for more information).</p>

<p>Another important topic was <a href="https://frida.re/">Frida</a>, which is used to
instrument and inspect target devices, including mobile phones. Since they are
fast-moving targets, we cannot stay with a given Frida version forever, and it’s
important to keep up. Unfortunately, breaking changes are not uncommon, so we
have to pick and choose carefully, instead of following any kind of “latest and
greatest” approach.</p>

<p><span class="small-caps">Debamax</span> decided to invest some time developing a companion
<a href="https://github.com/PiRogueToolSuite/deb-frido/"><code class="language-plaintext highlighter-rouge">deb-frido</code> repository</a>,
alongside the existing
<a href="https://github.com/PiRogueToolSuite/deb-frida/"><code class="language-plaintext highlighter-rouge">deb-frida</code> repository</a>. The
idea is to automate detecting new upstream releases, merging, patching,
building, and checking generated packages. If anything breaks, a developer must
fix the situation manually. If nothing breaks, the generated packages are
published into a
<a href="https://pirogue.apt.debamax.com/frida-bookworm/">staging repository</a>, from
where PiRogue developers can download packages, experiment with them, before
merging them into the actual <code class="language-plaintext highlighter-rouge">debian-12</code> repository (at which point they become
visible to actual PiRogue users).</p>

<p>Automation matters because there are a lot of new upstream releases each year:</p>

<ul>
  <li>in 2022: 26 releases;</li>
  <li>in 2023: 22 releases;</li>
  <li>in 2024: 29 releases (so far).</li>
</ul>

<p>Most of the design and implementation happened during regular streams on the
<a href="https://www.twitch.tv/cyrilbrulebois">@CyrilBrulebois Twitch channel</a>.</p>

<h1 id="virogue-project">ViRogue project</h1>

<p>Transparency matters: this project was outsourced to <span class="small-caps">Debamax</span> by
<a href="https://defensive-lab.agency/">Defensive Lab Agency</a>. Thanks for trusting us!</p>

<h2>Less hardware, more flexibility</h2>

<p>The idea was to get rid of the “hardware must be a Raspberry Pi” requirement
when it comes to setting up a PiRogue. Said slightly differently, it was about
“virtualizing the Raspberry Pi”, which led to the internal “ViRogue” codename
for this project (as seen in various branches and repositories at the time).</p>

<p>Initially, the network setup would look like this:</p>

<ul>
  <li>the Raspberry Pi is connected to the internet via its Ethernet connection,
and PiRogue interfaces (e.g. the web-based dashboard) are made available
through this connection;</li>
  <li>the Raspberry Pi shares its connectivity through its Wi-Fi connection,
configured as an Access Point (AP), and devices to be investigated use this
connection.</li>
</ul>

<p>As one might imagine, running on a Raspberry Pi is quite orthogonal to that
network setup, and virtually anything could replace the Raspberry Pi. Some
low-cost PC would easily do the trick, and supporting such systems is usually
trivial (no need to wonder about hardware integration at the Linux kernel, DTB,
or bootloader levels). The important part was about making sure to publish the
few architecture-specific PiRogue packages for both <code class="language-plaintext highlighter-rouge">arm64</code> (the historical
architecture, for Raspberry Pi devices) and <code class="language-plaintext highlighter-rouge">amd64</code> (the PC architecture, 64-bit
as well). Most packages are Python-based and available for the <code class="language-plaintext highlighter-rouge">all</code>
pseudo-architecture, but packages for <a href="https://www.nfstream.org/">NFStream</a> and
<a href="https://frida.re/">Frida</a> have to be built for each architecture.</p>

<p>Going one step further, even the “physical machine” requirement can be lifted,
and a virtual machine (VM) would work fine as well. What matters is really about
having one connection for use by the devices getting investigated (the isolated
network), and another one for internet and investigator access (the external
network).</p>

<p>At this point, the isolated network was already no longer tied to a Wi-Fi
interface on a Raspberry Pi configured in AP mode, but could be a physical or
virtual Ethernet interface (physical or virtual machine).</p>

<p>But we wanted to go even further, making it possible for the isolated network to
be provided over a VPN, so that PiRogue could be deployed on some
internet-accessible system (e.g. a virtual private server or VPS). It was
decided to focus on <a href="https://www.wireguard.com/">WireGuard</a>.</p>

<h2>Design and implementation</h2>

<p>Since all devices supported initially (Raspberry Pi 3 and Pi 4) looked similar,
a few things had been hardcoded in various components, like the <code class="language-plaintext highlighter-rouge">eth0</code> and
<code class="language-plaintext highlighter-rouge">wlan0</code> interfaces. With the updated design, it became critical to have the
proper interface names (for the isolated one as well as for the external one)
configured in the right components, but we also needed to introduce the
concept of “operating mode”:</p>

<ul>
  <li>Originally, we only had the “access point” mode (1 wireless, 1 wired).</li>
  <li>Moving to supporting two arbitrary interfaces meant introducing the
“appliance” mode, where we wouldn’t fiddle with a wireless interface and its
configuration as an access point (through <code class="language-plaintext highlighter-rouge">hostapd</code>). In this mode, it’s very
likely that both interfaces are considered to be wired ones.</li>
  <li>Adding a “VPN” mode alongside the “appliance” one was the logical next step.
Both modes are very similar, except we have a few more things to take care of
in the VPN case, like managing peers (WireGuard doesn’t distinguish between
client and server roles as other VPN technologies like OpenVPN would).</li>
</ul>

<div align="center">
  <p><img src="pirogue-operating-modes.png" alt="PiRogue operating modes" /></p>
</div>

<p>Further information can be found in the
<a href="https://pts-project.org/docs/pirogue/installation/#installing-pirogue-on-an-arbitrary-machine">PiRogue installation instructions</a>.</p>

<p>Given the requirements, <span class="small-caps">Debamax</span> suggested an approach that proved to be
quite robust for other customers: having a single component be responsible for
dealing with everything configuration-related, deploying the relevant
configuration files across all other components, enabling/disabling/restarting
services as required. That’s why the <code class="language-plaintext highlighter-rouge">pirogue-admin</code> component was introduced,
and also integrated in the <code class="language-plaintext highlighter-rouge">pirogue-base</code> post-installation script. This made it
possible for <code class="language-plaintext highlighter-rouge">pirogue-admin</code> to automatically choose and deploy a “best guess”
configuration depending on the detected configuration, based on whether the
system is running on a Raspberry Pi, and on the number and type of available
interfaces.</p>

<p>Documentation was written accordingly, and further information can be found on
the website:</p>

<ul>
  <li><a href="https://pts-project.org/docs/pirogue/version_2.x/configuration/">Configuration</a>:
This page is mainly intended for PiRogue administrators.</li>
  <li><a href="https://pts-project.org/docs/pirogue/version_2.x/system-integration/">System integration</a>:
This page is mainly intended for PiRogue developers.</li>
</ul>

<h1>Conclusion: Tying everything together</h1>

<p>Let’s quote the documented requirements for
<a href="https://pts-project.org/docs/pirogue/installation/#installing-pirogue-on-an-arbitrary-machine">installing PiRogue of on an arbitrary machine</a>:</p>

<blockquote>
  <p>The PiRogue ecosystem has been extended to cover more use cases, and it’s now
possible to deploy it on a machine featuring two interfaces. The requirements
are the following:</p>

  <ul>
    <li>a Debian 12 installation on a physical or virtual machine;</li>
    <li>using the amd64 or arm64 architecture;</li>
    <li>with at least 4GB of RAM and 40GB of disk;</li>
    <li>an initial network configuration allowing internet access.</li>
  </ul>
</blockquote>

<p>This could look like a natural conclusion to this 2-part blog post, combining
the switch to Debian with the brand new flexibility regarding hardware and
network configuration. However, the devil is in the details, as there’s no
single definition for “a Debian 12 installation”!</p>

<p>Since we wanted to support as many configurations as possible, many tests were
run, including:</p>

<ul>
  <li>PiRogue images deployed on Raspberry Pi 3, Pi 4, and Pi 5 (obviously);</li>
  <li>PiRogue deployed on top of Debian-provided cloud-oriented images, running in
virtual machines;</li>
  <li>PiRogue deployed on top of Debian systems deployed with Debian Installer
netinst images.</li>
</ul>

<p>Since users might want to install a desktop environment, we experimented with
all the desktop environment options offered by the Debian Installer: each one
declares a dependency on a given package that manages the network configuration
(e.g. <code class="language-plaintext highlighter-rouge">ifupdown</code> or <code class="language-plaintext highlighter-rouge">network-manager</code>), and <code class="language-plaintext highlighter-rouge">pirogue-admin</code> needed to know about
the various options. Tests even included chaining two laptops together, mixing up
connectivity between wired and wireless interfaces to test as many combinations
as possible.</p>

<p>Of course, those
<a href="https://pts-project.org/docs/pirogue/version_2.x/system-integration/#testing-the-system-integration">system integration tests</a> were documented extensively as well.</p>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[Looking beyond Raspberry Pi and Raspberry OS…]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/pirogue-logo.png" /><media:content medium="image" url="https://debamax.com/pirogue-logo.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Packaging CrowdSec for Debian: Bookworm (Debian 12)</title><link href="https://debamax.com/blog/2023/06/13/packaging-crowdsec-for-debian-bookworm/" rel="alternate" type="text/html" title="Packaging CrowdSec for Debian: Bookworm (Debian 12)" /><published>2023-06-13T10:00:00+02:00</published><updated>2023-06-13T10:00:00+02:00</updated><id>https://debamax.com/blog/2023/06/13/packaging-crowdsec-for-debian-bookworm</id><content type="html" xml:base="https://debamax.com/blog/2023/06/13/packaging-crowdsec-for-debian-bookworm/"><![CDATA[<h1>Introduction</h1>

<p>What CrowdSec is, how <span class="small-caps">Debamax</span> got involved, and what happened during
the Bullseye release cycle can be found in the previous blog post:
<a href="/blog/2023/06/12/packaging-crowdsec-for-debian-bullseye/">Packaging CrowdSec for Debian: Bullseye (Debian 11)</a>.</p>

<p>There were two main goals during the Bookworm release cycle: upgrading the
existing <code class="language-plaintext highlighter-rouge">crowdsec</code> package (Security Engine) to catch up with the new upstream
releases (from 1.0.x to 1.4.x), and packaging some bouncers (Remediation
Components). Bouncers are responsible for taking action when needed.</p>

<h1>Build dependencies</h1>

<h2>Bouncers</h2>

<p>It was mentioned in the previous blog post that <code class="language-plaintext highlighter-rouge">crowdsec</code> was an end-user
program, packaged accordingly. That means the generated executables (<code class="language-plaintext highlighter-rouge">crowdsec</code>
and <code class="language-plaintext highlighter-rouge">cscli</code>) are shipped in the <code class="language-plaintext highlighter-rouge">crowdsec</code> binary package, alongside
configuration files and a copy of hub data (under <code class="language-plaintext highlighter-rouge">/usr/share/crowdsec/hub</code>).</p>

<p>Since one of the goals is to add support for bouncers, that means building a
library package as well, since parts of the <code class="language-plaintext highlighter-rouge">crowdsec</code> source code are required.
Now, the <code class="language-plaintext highlighter-rouge">crowdsec</code> source package builds the <code class="language-plaintext highlighter-rouge">crowdsec</code> binary package (as it
did before) and also the <code class="language-plaintext highlighter-rouge">golang-github-crowdsecurity-crowdsec-dev</code> one. This
makes it possible to have packages that include
<code class="language-plaintext highlighter-rouge">github.com/crowdsecurity/crowdsec</code> in the <code class="language-plaintext highlighter-rouge">require</code> part of their <code class="language-plaintext highlighter-rouge">go.mod</code>
file. This addition was first staged in an upload to <code class="language-plaintext highlighter-rouge">experimental</code> so that it
could be reviewed from <code class="language-plaintext highlighter-rouge">NEW</code> by the FTP team whenever they had time, without
disrupting ongoing work in <code class="language-plaintext highlighter-rouge">unstable</code>/<code class="language-plaintext highlighter-rouge">testing</code>.</p>

<p>With that new package, it was possible to introduce
<code class="language-plaintext highlighter-rouge">golang-github-crowdsecurity-go-cs-bouncer</code> which is the main building block for
bouncers, and is required by both the <code class="language-plaintext highlighter-rouge">firewall</code> and <code class="language-plaintext highlighter-rouge">custom</code> bouncers. The
<code class="language-plaintext highlighter-rouge">firewall</code> one also needed an extra package, to deal with firewall rules, hence
<code class="language-plaintext highlighter-rouge">golang-github-google-nftables</code> on the bouncer-related graph below (important
packages in magenta, new dependencies in green).</p>

<p><a href="bookworm-bouncers.svg"><img src="bookworm-bouncers.svg" alt="Bouncer-related packages in Bookworm" /></a></p>

<h2>crowdsec</h2>

<p>A bigger amount of work was required for the <code class="language-plaintext highlighter-rouge">crowdsec</code> package itself, since its
<code class="language-plaintext highlighter-rouge">go.mod</code> grew many new dependencies between the 1.0.x versions and the 1.4.x
ones. The plan was published as a request for comments:
<a href="https://lists.debian.org/debian-go/2022/10/msg00018.html">Updating crowdsec, adding and updating other packages</a>.</p>

<p>As usual, there were some rather easy additions, but also some packages to
rename:</p>

<ul>
  <li>The <em>ent</em> ecosystem was shuffled around a little and since the upstream
locations are directly linked to module names, that meant introducing new
packages that weren’t entirely new (e.g. <code class="language-plaintext highlighter-rouge">golang-entgo-ent-dev</code> replacing
<code class="language-plaintext highlighter-rouge">golang-github-facebook-ent-dev</code>).</li>
  <li>Some of the packages introduced in the previous release cycle for the initial
<code class="language-plaintext highlighter-rouge">crowdsec</code> upload were also forked/adopted officially by the CrowdSec team
(e.g. <code class="language-plaintext highlighter-rouge">golang-github-crowdsecurity-grokky-dev</code> replacing
<code class="language-plaintext highlighter-rouge">golang-github-logrusorgru-grokky-dev</code>).</li>
</ul>

<p>Regarding existing packages, Cyril could count on Shengjing Zhu (Debian Go Packaging Team) to take care of updating the
<code class="language-plaintext highlighter-rouge">golang-github-apparentlymart-go-textseg</code> package which was a little tricky.
There were some inconsistencies between documentation and actual usage when it
comes to including a trailing <code class="language-plaintext highlighter-rouge">/vN</code> suffix in the <code class="language-plaintext highlighter-rouge">go.mod</code> files versus the
actual contents of the package, where there’s no <code class="language-plaintext highlighter-rouge">vN</code> directory.
Long story short: there was no need to introduce a new
<code class="language-plaintext highlighter-rouge">golang-github-apparentlymart-go-textseg-v13</code> package.</p>

<p>The <code class="language-plaintext highlighter-rouge">golang-github-hashicorp-hcl-v2</code> package was an entirely different story:
the existing <code class="language-plaintext highlighter-rouge">golang-github-hashicorp-hcl</code> has 98 reverse dependencies
and it seemed much safer to just introduce a new package, even if only used by
<code class="language-plaintext highlighter-rouge">crowdsec</code> initially. Adding that package makes it possible for the
aforementioned reverse dependencies to migrate progressively, but it should also help other
packages that ship a copy of <code class="language-plaintext highlighter-rouge">hashicorp/hcl/v2</code> in a <code class="language-plaintext highlighter-rouge">vendor</code> directory (like
<code class="language-plaintext highlighter-rouge">golang-github-hashicorp-nomad-dev</code>).</p>

<p>There were two extra existing packages to update: <code class="language-plaintext highlighter-rouge">golang-github-gin-gonic-gin</code>
and <code class="language-plaintext highlighter-rouge">golang-github-zclconf-go-cty</code>. The <code class="language-plaintext highlighter-rouge">ratt</code>-powered process described in the
previous blog post confirmed the updates were reasonable, as the only packages
that wouldn’t build from source with the updated packages were already riddled
with release critical bugs, so they were no factor.</p>

<p>Here is a complete graph of all relevant packages: 16 new packages (green) and 3
updated packages (red), in addition to <code class="language-plaintext highlighter-rouge">crowdsec</code> itself (magenta).</p>

<p><a href="bookworm-complete.svg"><img src="bookworm-complete.svg" alt="New and updated packages in Bookworm" /></a></p>

<h2>Complete dependency graph</h2>

<p>Adding the bouncer-related packages to the previous graph gives the full picture
for the Bookworm release cycle: 18 new packages (green) and 3 updated packages
(red), in addition to <code class="language-plaintext highlighter-rouge">crowdsec</code> and its two bouncers (magenta).</p>

<p><a href="bookworm-complete+bouncers.svg"><img src="bookworm-complete+bouncers.svg" alt="All new and updated packages in Bookworm" /></a></p>

<h1>Updating crowdsec</h1>

<h2>Upstream patches</h2>

<p>This is the same story as for Bullseye: software must be tailored to the
distribution’s needs or best practices, which is usually implemented via a
series of patches. Here is a quick summary of the patches found in the Bookworm
version:</p>

<ul>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0003-adjust-systemd-unit.patch">0003</a>:
Adjusts the <code class="language-plaintext highlighter-rouge">crowdsec.service</code> systemd unit to match Debian’s needs and best
practices.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0004-disable-geoip-enrich.patch">0004</a>:
Drops support for <code class="language-plaintext highlighter-rouge">geoip-enrich</code> which relies on non-free GeoIP data.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0005-adjust-config.patch">0005</a>:
Tweaks paths in the config file to point at <code class="language-plaintext highlighter-rouge">/var/lib/crowdsec</code> instead of
<code class="language-plaintext highlighter-rouge">/etc/crowdsec</code> for the hub.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0007-automatically-enable-online-hub.patch">0007</a>:
Enables the online hub transparently when <code class="language-plaintext highlighter-rouge">cscli hub update</code> is called.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0008-r3labs-diff-versions.patch">0008</a>:
Adjusts the <code class="language-plaintext highlighter-rouge">require</code> and <code class="language-plaintext highlighter-rouge">import</code> directives, replacing
<code class="language-plaintext highlighter-rouge">github.com/r3labs/diff/v2</code> with <code class="language-plaintext highlighter-rouge">github.com/r3labs/diff/v3</code>.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0009-disable-kafka-acquisition-module.patch">0009</a>:
Disables an acquisition module which would require adding many more packages,
which isn’t warranted since it’s not indispensable.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0010-disable-some-tests.patch">0010</a>:
Disables some tests that would require deploying a significant test
infrastructure. Those tests are run on upstream’s CI infrastructure anyway.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0011-refresh-protobuf-code.patch">0011</a>:
Refreshes code generated from Protobuf specifications, to make sure we don’t
run into version skews between build-time and run-time. This patch was
provided by upstream very quickly when we understood where the test failures
were coming from. This is definitely the kind of strong commitment Debian
Developers like to see from their upstream!</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0013-skip-flakky-tests.patch">0013</a>:
Disables some tests that are supposed to check performance. Unfortunately,
those error out at random, be it on the build daemon network while building
the package (when the test suite is being run, which leads to a build failure),
and/or on <a href="https://ci.debian.net/">Debian’s CI infrastructure</a> (where
<code class="language-plaintext highlighter-rouge">autopkgtest</code>-based testing happens). Since those tests are being picky about what the
results should look like, and since test failures don’t really indicate actual
problems, disabling them is best.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0014-silence-yaml-patching.patch">0014</a>:
Silences some log related to <em>YAML patching</em>. That’s a feature that was
introduced in the 1.4.x versions, that can be used by <code class="language-plaintext highlighter-rouge">crowdsec</code> itself and by
bouncers as well. Details can be found in later sections.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0015-silence-not-latest-version.patch">0015</a>:
Silences some messages about the version’s being outdated. We’re shipping the
package in a stable distribution, it’s totally fine not to try and update all
the time! Even more so since upstream is committed to providing long term
support for versions that end up in Debian stable releases.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/patches/0016-try-to-make-reproducible-build-work-2119.patch">0016</a>:
Adjusts some tests around syslog parsing. No longer hardcoding the reference
values for year-less syslog messages means two things. First, rebuilding the
package next year is not going to fail (due to an off-by-one error), which
could affect security updates or point releases. Second, that means happier
<a href="https://tests.reproducible-builds.org/debian/reproducible.html">reproducible builds</a>:
one of the common variations there is to build a given package in the past or
in the future, which would trigger discrepancies and test failures without that
patch.</li>
</ul>

<h1>Packaging separate resources</h1>

<p>There are still two <em>additional original tarballs</em> for the new upstream release:</p>

<ul>
  <li>the <code class="language-plaintext highlighter-rouge">data</code> tarball got refreshed from the various sources it aggregates;</li>
  <li>the <code class="language-plaintext highlighter-rouge">hub</code> tarball received many changes as the hub kept growing, and
bundling the contents of the upstream <code class="language-plaintext highlighter-rouge">v1.4.6</code> branch meant a significant
bump: from 279 files in <code class="language-plaintext highlighter-rouge">crowdsec_1.0.9.orig-hub1.tar.gz</code> to 1304 files in
<code class="language-plaintext highlighter-rouge">crowdsec_1.4.6.orig-hub1.tar.gz</code>!</li>
</ul>

<p>While the logic behind those <em>additional original tarballs</em> stayed exactly the
same as it was before, the fact there are many more items being shipped in the
offline hub meant we had to adjust the strategy when it comes to deploying
collections: not all of them are useful for a default Linux setup, and some of
them can be very resource-intensive!</p>

<h1>Plugging everything together: maintainer scripts</h1>

<p>Foreword: the <code class="language-plaintext highlighter-rouge">crowdsec.postinst</code> script is called after installation, but it can
be called in many ways
<a href="https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html">as documented in the Debian Policy</a>.</p>

<p>The main focus in this section is the <code class="language-plaintext highlighter-rouge">configure</code> action, which is used when
configuring the package for the first time (fresh installation) but also when
upgrading it. In the latter case, an extra parameter is provided by <code class="language-plaintext highlighter-rouge">dpkg</code>, so
that the script knows which version of the package it is upgrading from, which
helps make informed decisions.</p>

<h2>Case 1: Initial installation</h2>

<p>For a fresh installation, the new strategy regarding collections is to only
deploy a few of them, following upstream’s recommendations:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">crowdsecurity/linux</code></li>
  <li><code class="language-plaintext highlighter-rouge">crowdsecurity/apache2</code></li>
  <li><code class="language-plaintext highlighter-rouge">crowdsecurity/nginx</code></li>
</ul>

<p>Since there’s no tooling around enabling collections from the local filesystem
(offline hub), the <code class="language-plaintext highlighter-rouge">crowdsec.postinst</code> script has to create symlinks on its own,
and to do that correctly, it also needs to consider all items each collection
requires (i.e. parsers, scenarios, postoverflows, and other collections…)
and enable them as well.</p>

<p>When everything is fine, <code class="language-plaintext highlighter-rouge">cscli collections list</code> would return something like
this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>COLLECTIONS
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                                📦 Status   Version   Local Path
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
 crowdsecurity/apache2               ✔️ enabled   0.1       /etc/crowdsec/collections/apache2.yaml
 crowdsecurity/base-http-scenarios   ✔️ enabled   0.6       /etc/crowdsec/collections/base-http-scenarios.yaml
 crowdsecurity/http-cve              ✔️ enabled   1.9       /etc/crowdsec/collections/http-cve.yaml
 crowdsecurity/linux                 ✔️ enabled   0.2       /etc/crowdsec/collections/linux.yaml
 crowdsecurity/nginx                 ✔️ enabled   0.2       /etc/crowdsec/collections/nginx.yaml
 crowdsecurity/sshd                  ✔️ enabled   0.2       /etc/crowdsec/collections/sshd.yaml
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
</code></pre></div></div>

<p>A missing item (e.g. after failing to create a symlink to a required parser) would
lead to the following warning:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>crowdsecurity/linux                 ⚠️ enabled,tainted   0.2       /etc/crowdsec/collections/linux.yaml
</code></pre></div></div>

<p>Enabling upstream-recommended collections and their dependencies is implemented
via two lists:
<a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/crowdsec.postinst/#L39-L44"><code class="language-plaintext highlighter-rouge">UPSTREAM_COLLECTIONS</code> and <code class="language-plaintext highlighter-rouge">UPSTREAM_ITEMS</code></a>.</p>

<h2>Case 2: Upgrading from an earlier version</h2>

<p>Since there’s also no tooling to maintain existing collections on the local
filesystem (offline hub), it seemed reasonable to concentrate on the three
collections mentioned in the previous section: they were deployed initially, and
they would only need to get a few more items (which could be called “new dependencies”)
enabled to make sure they appear as <code class="language-plaintext highlighter-rouge">enabled</code> and not <code class="language-plaintext highlighter-rouge">enabled,tainted</code>.</p>

<p>Since it’s important to respect the admin’s decisions, that only happens
if all three collections are still enabled when performing the upgrade. If one
of them is missing, it’s assumed the admin took control, and will deal
with collections on their own.</p>

<p>This is also implemented by using the same two lists as mentioned above:
<a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/crowdsec.postinst/#L39-L44"><code class="language-plaintext highlighter-rouge">UPSTREAM_COLLECTIONS</code> and <code class="language-plaintext highlighter-rouge">UPSTREAM_ITEMS</code></a>.</p>

<h2>New topic: SQLite-related warnings</h2>

<p>Upstream is really careful about many things, and the default settings are
meant to be safe. The default configuration relies on a small SQLite database, and to
avoid problems on some filesystems, the
<abbr title="Write-Ahead Logging">WAL</abbr> feature is disabled by default.
This is fine except it generates warnings in various places, and it seemed best
to address it.</p>

<p>That’s why <code class="language-plaintext highlighter-rouge">crowdsec.postinst</code> detects whether a <em>local configuration</em> is
present. If there’s none, it will try and detect whether the database backend is
<code class="language-plaintext highlighter-rouge">sqlite</code>, whether WAL is enabled, and the path to the database. If the answers
to the first two questions is yes and no respectively, it will check which
filesystem is used for the database, and it will automatically enable the WAL
feature unless <code class="language-plaintext highlighter-rouge">nfs*</code> is detected.</p>

<p>To avoid modifying the package-provided main configuration file,
<code class="language-plaintext highlighter-rouge">/etc/crowdsec/config.yaml</code> (which would trigger prompts during further
upgrades), the <code class="language-plaintext highlighter-rouge">db_config.use_wal</code> boolean is stored in what upstream calls a
<em>YAML patch</em>, in a new <code class="language-plaintext highlighter-rouge">/etc/crowdsec/config.yaml.local</code> file, that gets layered
on top of the main configuration file. Again, to respect any changes that
might have been deployed by the admin, the auto-detection is skipped if
that file is already present.</p>

<p>Starting to use this
<a href="https://docs.crowdsec.net/docs/configuration/crowdsec_configuration/#overriding-values"><em>YAML patch</em></a>
feature is why the <code class="language-plaintext highlighter-rouge">0014</code> patch mentioned above was introduced: without it, some warning
would be printed when the daemon starts, and whenever a command reads the config
(i.e. each and every <code class="language-plaintext highlighter-rouge">cscli</code> call).</p>

<h1>Checking the upgrade path from Bullseye to Bookworm</h1>

<p>At that point, it looked like everything should be in place: upstream patches,
updated data and hub files, updated <code class="language-plaintext highlighter-rouge">crowdsec.postinst</code> code to manage initial
collection deployment and also upgrading from a previous version.</p>

<p>Since fresh installations were tested very easily and regularly (purging the
package ensures starting from a clean slate every time), it was time to focus on
verifying the upgrade path from the version found in Bullseye. This time, not just
making sure collections are handled properly, but also ensuring the runtime would
work fine as well.</p>

<p><strong>Spoiler alert:</strong> there were a few more changes to the <code class="language-plaintext highlighter-rouge">crowdsec.postinst</code>
script!</p>

<h2>Problem #1: delays</h2>

<p>The first bad surprise was huge delays during the upgrade, for no obvious
reasons at first, filed as <a href="https://bugs.debian.org/1031326">#1031326</a>. Upon
investigation, it became clear the old daemon had troubles stopping, and one
needed to wait until the default <code class="language-plaintext highlighter-rouge">TimeoutStopSec</code> before <code class="language-plaintext highlighter-rouge">systemd</code> switched from
<code class="language-plaintext highlighter-rouge">SIGTERM</code> to <code class="language-plaintext highlighter-rouge">SIGKILL</code>. Upstream confirmed there were various issues with the
event loops in the 1.0.x versions, and that it was unlikely a solution could be
found and deployed via a point release. This meant we couldn’t fix the version
getting upgraded from, so that it wouldn’t exhibit this problem.</p>

<p>To avoid waiting 90 seconds, it was agreed with upstream to
<a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/crowdsec.postinst/#L237-L249">implement a workaround</a>,
lowering the timeout to 20 seconds, which should leave coroutines plenty of time to
commit any pending changes to the database, even if a clean exit couldn’t be
reached afterwards. This is done only when upgrading
from 1.0.x, by setting a temporary override for the systemd unit
(<code class="language-plaintext highlighter-rouge">crowdsec.service</code>), that’s only valid until the next reboot (temporary storage under
<code class="language-plaintext highlighter-rouge">/run/systemd/system</code>).</p>

<h2>Problem #2: SQLite incompatibilities</h2>

<p>Another upgrade issue is related to one of the few native libraries used by
the <code class="language-plaintext highlighter-rouge">crowdsec</code> package. While most of the code is written in Go and statically
linked, there’s an external dependency on <code class="language-plaintext highlighter-rouge">libsqlite3-0</code>. Unfortunately, two
incompatibilities were discovered:</p>

<ul>
  <li>The old <code class="language-plaintext highlighter-rouge">crowdec</code> can’t work with the new <code class="language-plaintext highlighter-rouge">libsqlite3-0</code> since assumptions in the
<em>ent</em> framework would no longer be valid. This was filed as
<a href="https://bugs.debian.org/1033029">#1033029</a> against new <code class="language-plaintext highlighter-rouge">libsqlite3-0</code>, so
that its maintainer could consider adding a <code class="language-plaintext highlighter-rouge">Breaks</code> relationship against the
old <code class="language-plaintext highlighter-rouge">crowdsec</code> package to avoid this particular combination. László
Böszörményi quickly uploaded the package with the proposed patch, and it ended
up in <code class="language-plaintext highlighter-rouge">testing</code> a few days later.</li>
  <li>Conversely, the new <code class="language-plaintext highlighter-rouge">crowdsec</code> can’t work with the old <code class="language-plaintext highlighter-rouge">libsqlite3-0</code> since it
relies on syntax that was only added in a version between Bullseye’s version
and Bookworm’s, tracked in <a href="https://bugs.debian.org/1033132">#1033132</a>. What
happens here is that the dependency on the native library is computed using
what’s called the <code class="language-plaintext highlighter-rouge">shlibs</code>/<code class="language-plaintext highlighter-rouge">symbols</code> mechanism, which determines the minimal
version that’s needed to ensure the <code class="language-plaintext highlighter-rouge">libsqlite3-0</code> package contains all the
ELF-level symbols used by <code class="language-plaintext highlighter-rouge">crowdsec</code>. Unfortunately, that doesn’t account for SQLite
syntax support, and
<a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/control/#L85">some higher dependency was hardcoded</a>
to reflect the actual runtime needs regarding SQLite language support, and not
just the required symbols.</li>
</ul>

<p>Combining both fixes ensures a correct upgrade path from Bullseye to Bookworm,
making sure partial upgrades are fine: even when the package manager is
performing a full upgrade from <code class="language-plaintext highlighter-rouge">oldstable</code> to <code class="language-plaintext highlighter-rouge">stable</code>, it splits work into sets
of packages, and it was critical to avoid having both packages in different sets!</p>

<h2>Problem #3: missing decisions</h2>

<p>Yet another upgrade issue was missing decisions after the upgrade, for several
hours, filed as <a href="https://bugs.debian.org/1033138">#1033138</a>. Most users are
likely to expect to get decisions from the Central API (CAPI),
via the <em>Community blocklist</em> alert. Unfortunately, the format
changed between 1.0.x and 1.4.x, and even if the relevant entries in the
database would still exist, they wouldn’t be taken into account by the new
version. Even worse, since the last CAPI pull is considered recent enough, this issue would
persist for several hours, leaving users without any kind of protection.</p>

<p>An easy fix would have been to purge all alerts before starting the new
version, but that would also lose local alerts (e.g. manual additions by the
admin). Another easy fix would have been to purge the <em>Community
blocklist</em> before restarting, but the old API didn’t know how to perform specific
deletions.</p>

<p>That’s why the following was implemented:</p>

<ul>
  <li>when upgrading from 1.0.x;</li>
  <li>once restarted into 1.4.x;</li>
  <li>if there are alerts matching <em>Community blocklist</em>, delete them;</li>
  <li>if deletions happened, restart once more.</li>
</ul>

<p>The
<a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/crowdsec.postinst/#L253-L266">deletion and restart logic</a>
is <a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/crowdsec.postinst/#L248">conditioned by a flag</a> set
when upgrading from 1.0.x. It makes sure there’s an immediate CAPI pull, instead
of waiting for the next scheduled one, which would default to waiting 2 hours
without any kind of protection via the <em>Community blocklist</em>.</p>

<h2>Problem #4: missing decisions, again</h2>

<p>Seeing how decisions could go missing during upgrades, it seemed important to
inspect what happens on fresh installations as well. That’s how it was spotted
that there are no guarantees regarding the first time <code class="language-plaintext highlighter-rouge">crowdsec</code> starts, with
many coroutines running in parallel, and there might not be any successful CAPI
pull. This also means leaving users without any kind of protection right after
the initial installation, for several hours.</p>

<p>That led to
<a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/crowdsec.postinst/#L268-L294">implementing another workaround</a>:</p>

<ul>
  <li>when installing;</li>
  <li>poll the logs once every second;</li>
  <li>as soon as a message about added/deleted entries appears, the CAPI pull is
confirmed, and we can move on;</li>
  <li>if there’s no such message after 20 seconds, force a CAPI pull by restarting
the daemon.</li>
</ul>

<h1>Bouncers</h1>

<h2>Introduction</h2>

<p>The official term in the
<a href="https://docs.crowdsec.net/docs/bouncers/intro">upstream documentation</a> is
Remediation Components, but they’re called bouncers most of the time. They are in
charge of acting upon decisions provided by the Security Engine. Again, after
discussion with upstream, it was decided to focus on two bouncers.</p>

<p>The <a href="https://docs.crowdsec.net/docs/bouncers/firewall/">firewall bouncer</a> will
periodically fetch new, expired, and deleted decisions, and adjust the firewall
configuration to reflect those changes. This is how one deploys a crowd-powered
<code class="language-plaintext highlighter-rouge">fail2ban</code>-like setup.</p>

<p>The <a href="https://docs.crowdsec.net/docs/bouncers/custom/">custom bouncer</a> is a
rather simple yet very flexible bouncer, which must be configured with a command
via the <code class="language-plaintext highlighter-rouge">bin_path</code> configuration key. It will also periodically fetch new,
expired, and deleted decisions, but it will pass them as arguments to the
configured command. There are no requirements on that command, it can be a shell
script or anything else.</p>

<p>There are various ways to deploy <code class="language-plaintext highlighter-rouge">crowdsec</code> (the Security Engine) and bouncers,
and this flexibility is reflected in the packaging as well. Both bouncers have
<code class="language-plaintext highlighter-rouge">crowdsec</code> only in <code class="language-plaintext highlighter-rouge">Recommends</code>, which is basically an optional dependency (as
opposed to <code class="language-plaintext highlighter-rouge">Depends</code>), allowing for different setups:</p>

<ul>
  <li>One can deploy <code class="language-plaintext highlighter-rouge">crowdsec</code> on a particular machine of the infrastructure,
dealing both with Central API exchanges and exposing the Local API for use by
other machines on the network. Bouncer(s) can then be deployed on other
machines, configured to use the Local API exposed by the first machine,
without needing a <code class="language-plaintext highlighter-rouge">crowdsec</code> package installed locally. In this case, each
bouncer would require at least the <code class="language-plaintext highlighter-rouge">api_url</code> and <code class="language-plaintext highlighter-rouge">api_key</code> keys to be set in
their configuration file.</li>
  <li>One can opt for a simpler, all-in-one solution: installing bouncer(s) on a
machine will automatically install <code class="language-plaintext highlighter-rouge">crowdsec</code> (unless the admin requested
otherwise), which by default configures its Local API to listen on the
loopback. In that case, bouncers will detect that <code class="language-plaintext highlighter-rouge">crowdsec</code> is present on the
same machine, and will automatically register themselves, meaning no extra
steps for the admin (no need for <code class="language-plaintext highlighter-rouge">api_url</code> or <code class="language-plaintext highlighter-rouge">api_key</code> at least).</li>
</ul>

<p>This auto-registration seemed to be working fine initially but the order in which
packages are configured when <code class="language-plaintext highlighter-rouge">Recommends</code> are involved isn’t guaranteed like it
would be when using <code class="language-plaintext highlighter-rouge">Depends</code>. Thankfully there was still some time left in the
Bookworm release cycle to fix those issues with minimal changes, that were
swiftly reviewed and approved by the release team. The updated logic is the
following:</p>

<ul>
  <li>Bouncers can still auto-register as they did before, if <code class="language-plaintext highlighter-rouge">crowdsec</code> is
installed <strong>and</strong> if it is configured already.</li>
  <li>If <code class="language-plaintext highlighter-rouge">crowdsec</code> is only installed but hasn’t been configured yet, they can
queue their registration. In this case, <code class="language-plaintext highlighter-rouge">crowdsec.postinst</code> is responsible for
an extra task:
<a href="https://sources.debian.org/src/crowdsec/1.4.6-4/debian/crowdsec.postinst/#L296-L314">processing the registration queue</a>.</li>
</ul>

<p>The configuration for the bouncers is similar to what happens on the <code class="language-plaintext highlighter-rouge">crowdsec</code>
side: an upstream-provided configuration file is shipped under
<code class="language-plaintext highlighter-rouge">/etc/crowdsec/crowdsec-&lt;NAME&gt;-bouncer.yaml</code>. The <em>YAML patching</em> is also
supported, that’s why the <code class="language-plaintext highlighter-rouge">postinst</code> script of each bouncer creates a <code class="language-plaintext highlighter-rouge">.local</code>
file instead of touching the main configuration file. This includes the <code class="language-plaintext highlighter-rouge">api_key</code> when auto-registering, but also other
parameters that are detected on the local system, and/or that are expected to be
changed by the admin. Additionally, a <code class="language-plaintext highlighter-rouge">.id</code> is left behind with the
bouncer’s name, so that the <code class="language-plaintext highlighter-rouge">postrm</code> script (run by <code class="language-plaintext highlighter-rouge">dpkg</code> when removing a
package) can also proceed to auto-unregistration.</p>

<h2>Firewall bouncer</h2>

<p>It can be trivially installed on Debian 12 with just the following command
(which will pull and configure <code class="language-plaintext highlighter-rouge">crowdsec</code> automatically as described above):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install crowdsec-firewall-bouncer
</code></pre></div></div>

<p>The <a href="https://sources.debian.org/src/crowdsec-firewall-bouncer/0.0.25-3/debian/crowdsec-firewall-bouncer.postinst/"><code class="language-plaintext highlighter-rouge">crowdsec-firewall-bouncer</code> script</a>
detects which firewall system is currently deployed, and
can deal with <code class="language-plaintext highlighter-rouge">nftables</code> (the default since Debian 11), or <code class="language-plaintext highlighter-rouge">iptables</code>. In the
latter case, <code class="language-plaintext highlighter-rouge">ipset</code> must also be installed. Some warning message is shown in
the tricky case: <code class="language-plaintext highlighter-rouge">nftables</code> and <code class="language-plaintext highlighter-rouge">iptables</code> are both installed, with the
<code class="language-plaintext highlighter-rouge">iptables</code> alternative pointing to <code class="language-plaintext highlighter-rouge">iptables-legacy</code> (instead of the now-default
<code class="language-plaintext highlighter-rouge">iptables-nft</code>)… while <code class="language-plaintext highlighter-rouge">ipset</code> isn’t installed. Unfortunately, the <code class="language-plaintext highlighter-rouge">Depends</code>
syntax can’t really express <code class="language-plaintext highlighter-rouge">nftables</code> or <code class="language-plaintext highlighter-rouge">iptables</code>+<code class="language-plaintext highlighter-rouge">ipset</code>, especially with
<code class="language-plaintext highlighter-rouge">iptables</code> listing <code class="language-plaintext highlighter-rouge">nftables</code> in <code class="language-plaintext highlighter-rouge">Recommends</code> to provide an upgrade path for
older systems.</p>

<p>The installation triggers those messages on successful installation:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Setting up crowdsec-firewall-bouncer (0.0.25-1+b2) ...
I: Configuring nftables [see README.Debian]
To adjust the config: editor /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.local &amp;&amp; systemctl restart crowdsec-firewall-bouncer
Created symlink /etc/systemd/system/multi-user.target.wants/crowdsec-firewall-bouncer.service -&gt; /lib/systemd/system/crowdsec-firewall-bouncer.service.
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">README.Debian</code> file can be found on disk under <code class="language-plaintext highlighter-rouge">/usr/share/doc/crowdsec-firewall-bouncer/</code>
as usual, and can also be
<a href="https://sources.debian.org/src/crowdsec-firewall-bouncer/0.0.25-3/debian/README.Debian/">viewed online</a>.</p>

<p>After installation, with the default, upstream-recommended collections mentioned
earlier for <code class="language-plaintext highlighter-rouge">crowdsec</code>, a few thousand lines are expected when asking for the
ruleset list:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nft list ruleset | wc -l
</code></pre></div></div>

<h2>Custom bouncer</h2>

<p>The installation is almost as trivial as the firewall bouncer one:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install crowdsec-custom-bouncer
</code></pre></div></div>

<p>But this time around, it <strong>must</strong> be configured to start doing something useful,
as suggested when the configuration happens:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Setting up crowdsec-custom-bouncer (0.0.15-1+b2) ...
W: Using /usr/bin/true as default custom script
To adjust the config: editor /etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml.local &amp;&amp; systemctl restart crowdsec-custom-bouncer
Created symlink /etc/systemd/system/multi-user.target.wants/crowdsec-custom-bouncer.service -&gt; /lib/systemd/system/crowdsec-custom-bouncer.service.
</code></pre></div></div>

<p>The <a href="https://sources.debian.org/src/crowdsec-custom-bouncer/0.0.15-3/debian/crowdsec-custom-bouncer.postinst/"><code class="language-plaintext highlighter-rouge">crowdsec-custom-bouncer.postinst</code> script</a>
initially provisions the local configuration with <code class="language-plaintext highlighter-rouge">bin_path: /usr/bin/true</code>,
which gives a nice no-op placeholder that lets the service start successfully,
even before the admin starts adjusting the configuration.</p>

<p>While this bouncer is very flexible and can run anything, one should probably
keep in mind that using it means spawning many processes. For example, after
simply setting <code class="language-plaintext highlighter-rouge">bin_path: /usr/bin/logger</code> and restarting the bouncer, one can
see the following:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># journalctl | grep 'root\[' | head -1
Jun 13 08:00:00 crowdsec root[14812]: add 1.2.3.4 666 crowdsecurity/ssh-bf {/* JSON1 */}

# journalctl | grep 'root\[' | tail -1
Jun 13 08:00:05 crowdsec root[21905]: add 5.6.7.8 666 crowdsecurity/http-probing {/* JSON2 */}
</code></pre></div></div>

<p>(IP addresses are redacted, and JSON metadata is skipped for readability.)</p>

<p>This means that the custom command was spawned more than 7000 times!</p>

<p>One should also note that the <code class="language-plaintext highlighter-rouge">bin_path</code> parameter expects an absolute path to
an executable (program or script), and that cannot include options (e.g.
<code class="language-plaintext highlighter-rouge">bin_path: "/usr/bin/logger -t custom-test"</code> wouldn’t work) or rely on shell
expansion. If the custom command is actually a well-known program that should be
called with specific options, a standard solution would be to write a simple
wrapper, store it under <code class="language-plaintext highlighter-rouge">/usr/local/bin</code> for example, and point <code class="language-plaintext highlighter-rouge">bin_path</code> to
that wrapper.</p>

<h1>Related tasks for the Debian Go team</h1>

<p>Here are two additional pointers to related tasks:</p>

<ul>
  <li><a href="https://lists.debian.org/debian-go/2023/03/msg00004.html">RFC: Fixing golang-github-tidwall-gjson’s RC bugginess for bookworm</a>:
there were known security issues in one of the libraries used by <code class="language-plaintext highlighter-rouge">crowdsec</code>, Thorsten Alteholz (another member of the Debian Go Packaging Team)
<a href="https://lists.debian.org/debian-go/2023/03/msg00005.html">agreed with the plan</a>,
then the release team <a href="https://bugs.debian.org/1032862#10">agreed as well</a>, and
everything happened a few months before the Bookworm release.</li>
  <li><a href="https://lists.debian.org/debian-go/2023/03/msg00046.html">RFC: Removing some unneeded packages from unstable (and testing)</a>:
since some packages introduced earlier are no longer needed, and haven’t
become dependencies of other packages, some clean-up was suggested. Since
there were many things to keep everyone busy, that didn’t happen before the
Bookworm release. Removal requests are likely to be filed early in the Trixie (new
<code class="language-plaintext highlighter-rouge">testing</code> and future Debian 13) release cycle.</li>
</ul>

<h1>Conclusion</h1>

<p>While there were
<a href="/blog/2023/06/12/packaging-crowdsec-for-debian-bullseye/">significant efforts during the Debian 11 release cycle</a>,
there wasn’t enough time to package both the Security Engine (<code class="language-plaintext highlighter-rouge">crowdsec</code>) and
Remediation Components (bouncers). Thankfully, the initial packaging work proved to be
appropriate groundwork for the <code class="language-plaintext highlighter-rouge">crowdsec</code> upgrade from 1.0.x to 1.4.x, even if a
number of fixes or workarounds were needed to ensure a smooth upgrade experience.
Adding <code class="language-plaintext highlighter-rouge">crowdsec-firewall-bouncer</code> and <code class="language-plaintext highlighter-rouge">crowdsec-custom-bouncer</code> along
<code class="language-plaintext highlighter-rouge">crowdsec</code> itself completed the work started right before the Bullseye freeze,
finally making it possible to take action on Bookworm!</p>

<div style="margin-top: 2em; margin-bottom: 2em">

  <table class="table local-no-fullwidth local-is-centered is-bordered">
    <thead>
      <tr>
        <th>Package</th>
        <th>Version</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>crowdsec</td>
        <td>1.4.6-4</td>
      </tr>
      <tr>
        <td>crowdsec-custom-bouncer</td>
        <td>0.0.15-3</td>
      </tr>
      <tr>
        <td>crowdsec-firewall-bouncer</td>
        <td>0.0.25-3</td>
      </tr>
    </tbody>
  </table>

  <p class="has-text-centered"><strong>Version table for Debian 12 “Bookworm”</strong></p>

</div>

<div class="message is-info">
  <div class="message-header">
    <p>About upstream</p>
  </div>
  <div class="message-body">
    <p><span class="small-caps">Debamax</span> as a company and Cyril as a
Debian Developer are very pleased with all interactions with CrowdSec’s
upstream. Developers are clearly interested in what is best for Debian users,
and they’re willing to compromise and/or work a little more to make sure
creating, then updating the Debian packages goes as smoothly as possible. This
includes proposing solutions to avoid hard-to-solve, Protobuf-related version
skews. Very nice cooperation!</p>
  </div>
</div>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[Moving to the next level during the Bookworm release cycle: catching up with new releases of the Security Engine and packaging Remediation Components.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/crowdsec.png" /><media:content medium="image" url="https://debamax.com/crowdsec.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Packaging CrowdSec for Debian: Bullseye (Debian 11)</title><link href="https://debamax.com/blog/2023/06/12/packaging-crowdsec-for-debian-bullseye/" rel="alternate" type="text/html" title="Packaging CrowdSec for Debian: Bullseye (Debian 11)" /><published>2023-06-12T10:00:00+02:00</published><updated>2023-06-12T10:00:00+02:00</updated><id>https://debamax.com/blog/2023/06/12/packaging-crowdsec-for-debian-bullseye</id><content type="html" xml:base="https://debamax.com/blog/2023/06/12/packaging-crowdsec-for-debian-bullseye/"><![CDATA[<h1>Introduction</h1>

<p>Let’s start by quoting the <a href="https://www.crowdsec.net/">CrowdSec website</a>:</p>

<blockquote>
  <p>CrowdSec is an open-source and collaborative security stack leveraging the
crowd power. Analyze behaviors, respond to attacks &amp; share signals across the
community. Join the community and let’s make the Internet safer, together.</p>
</blockquote>

<p>Put in another way, we’re basically talking about a global <code class="language-plaintext highlighter-rouge">fail2ban</code> solution
rather than a local one.</p>

<h1>Ready. Steady. Go!</h1>

<p><span class="small-caps">Debamax</span> started working on the packaging very late in the Bullseye
(Debian 11) release cycle, i.e. in December 2020, while the soft freeze was
scheduled for early February 2021. This meant a very short window to get many
new packages into <code class="language-plaintext highlighter-rouge">unstable</code> and then into <code class="language-plaintext highlighter-rouge">testing</code>, since new packages can no
longer be introduced into <code class="language-plaintext highlighter-rouge">testing</code> once the soft freeze starts. Therefore it
was decided to concentrate immediate efforts on the Security Engine itself, to
be packaged as <code class="language-plaintext highlighter-rouge">crowdsec</code>, and to delay work on Remediation Components
(bouncers) until the beginning of the Bookworm (Debian 12) release cycle.</p>

<p>Thankfully, the Debian Go Packaging Team was very welcoming, offered some
guidance when needed, and also promptly validated the various plans that were
proposed when it came to updating existing packages. The
<a href="https://go-team.pages.debian.net/">team’s documentation</a> is rather extensive and
made it easy to get started with Go packages.</p>

<h1>Build dependencies</h1>

<p>The <a href="https://people.debian.org/~stapelberg/2015/07/27/dh-make-golang.html"><code class="language-plaintext highlighter-rouge">dh-make-golang</code> command</a>
really helps bootstraping new source packages. The remaining hard parts are
mainly about checking the copyright and license information
(<code class="language-plaintext highlighter-rouge">debian/copyright</code>), and keeping track of dependencies that are spotted by
inspecting the upstream <code class="language-plaintext highlighter-rouge">go.mod</code> files, and that might not be in the archive
yet, to build the whole dependency tree.</p>

<p>This is how the first part of the plan,
<a href="https://lists.debian.org/debian-go/2020/12/msg00019.html">Packaging crowdsec and some other packages</a>,
ended up being mostly about introducing many new <code class="language-plaintext highlighter-rouge">golang-*</code> packages into the
archive, to be used as <code class="language-plaintext highlighter-rouge">Build-Depends</code> for the upcoming <code class="language-plaintext highlighter-rouge">crowdsec</code> package.</p>

<p>Introducing new packages seemed relatively easy, with the main question being
when they would be processed from <code class="language-plaintext highlighter-rouge">NEW</code>, i.e. reviewed by the FTP team. Since most Go
packages are very similar, and since the MIT/X license is used all over the
place, it seems they are reviewed very quickly, and packages were accepted in
record time.</p>

<p>The second part of the plan was a little more complicated, as it was about
updating an existing package, which counts many reverse dependencies, and that
update also involved adding new packages. Let’s take a step back and sum up the
relevant characteristics of the Go ecosystem.</p>

<p>Library packages in Go are very different from the historical shared libraries
and their development packages: Go library packages ship the source code, period.
End-user programs like <code class="language-plaintext highlighter-rouge">crowdsec</code> list library packages in their
<code class="language-plaintext highlighter-rouge">Build-Depends</code> and the build system aggregates their source code inside the
build directory. This means the usual API/ABI stability concepts don’t apply,
and this is compensated by having some tooling to determine whether an update is
considered safe.</p>

<p>This is where <code class="language-plaintext highlighter-rouge">ratt</code> (which stands for “rebuild all the things!”) comes into
play: once an updated package has been prepared, using metadata found in <code class="language-plaintext highlighter-rouge">Sources</code>
files, <code class="language-plaintext highlighter-rouge">ratt</code> determines the reverse dependencies and builds them all in turn,
using the updated package. It leverages <code class="language-plaintext highlighter-rouge">sbuild</code>, enriching the build
environment with the updated package automatically. Since most Go packages come
with a test suite, just building them gives a pretty good indication as to
whether the update should be regarded as safe.</p>

<p>If failures (<abbr title="Failure To Build From Source">FTBFS</abbr>) come up
during such a systematic rebuild of all reverse dependencies, it makes sense to
see if those failures are already happening with the package already in the archive:</p>

<ul>
  <li>If that’s the case, check whether the Debian <abbr title="Bug Tracking System">BTS</abbr> already knows about that failure, and file a new bug report
if needed.</li>
  <li>If that’s a new regression, further planning must happen. That might mean
trying to package a new version, backporting a compatibility fix, etc.</li>
</ul>

<p>Two external factors come in:</p>

<ul>
  <li>There are regular rebuilds of all packages in the archive, so an existing
failure to build would usually be documented in the BTS already.</li>
  <li>The severity of such bug reports is <code class="language-plaintext highlighter-rouge">serious</code>, which is considered
<abbr title="release-critical">RC</abbr> and is ground for removal from
<code class="language-plaintext highlighter-rouge">testing</code>.</li>
</ul>

<p>This means that packages only in <code class="language-plaintext highlighter-rouge">unstable</code> are usually not considered a blocker
when preparing an updated version of an existing package.</p>

<p>Going back to the issue at hand, the
<a href="https://lists.debian.org/debian-go/2020/12/msg00059.html">Updating golang-github-gin-gonic-gin to 1.6.3</a> plan
listed 3 failing packages out of the 35 detected reverse dependencies. Thankfully, one of them
was a false positive (due to <code class="language-plaintext highlighter-rouge">ratt</code>’s lack of support for multiple source
versions), and the other two were known issues. A green light was given to
update this package.</p>

<p>Finally, the third part was about updating two existing packages that provide GeoIP support:
<a href="https://lists.debian.org/debian-go/2020/12/msg00060.html">Updating golang-github-oschwald-maxminddb-golang and golang-github-oschwald-geoip2-golang</a>. The
maintainer gave a green light for both uploads, which were relatively low risk
anyway since it only involved three packages: the two getting updated, and a
single reverse dependency.</p>

<p>Plugging eveything together, here is a complete graph of all relevant packages:
21 new packages (green) and 3 updated packages (red), in addition to <code class="language-plaintext highlighter-rouge">crowdsec</code>
itself (magenta).</p>

<p><a href="bullseye-complete.svg"><img src="bullseye-complete.svg" alt="New and updated packages in Bullseye" /></a></p>

<h1>Patching upstream</h1>

<p>Having a bunch of build dependencies is only a part of the packaging work.
Another one is tailoring the software to the distribution’s needs or best
practices, which is usually implemented via a series of patches. Here is a quick
summary of the patches found in the Bullseye version:</p>

<ul>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0001-use-a-local-machineid-implementation.patch/">0001</a>:
Implements the <code class="language-plaintext highlighter-rouge">machineid</code> feature, which spares a dependency on yet another
new package.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0002-add-compatibility-for-older-sqlite-driver.patch/">0002</a>:
Ensures compatibility with the version of the SQLite driver found in Bullseye.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0003-adjust-systemd-unit.patch/">0003</a>:
Adjusts the <code class="language-plaintext highlighter-rouge">crowdsec.service</code> systemd unit to match Debian’s needs and best
practices.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0004-disable-geoip-enrich.patch/">0004</a>:
Drops support for <code class="language-plaintext highlighter-rouge">geoip-enrich</code> which relies on non-free GeoIP data.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0005-adjust-config.patch/">0005</a>:
Tweaks paths in the config file to point at <code class="language-plaintext highlighter-rouge">/var/lib/crowdsec</code> instead of
<code class="language-plaintext highlighter-rouge">/etc/crowdsec</code> for the hub.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0006-prefer-systemctl-restart.patch/">0006</a>:
Tweaks information messages to mention <code class="language-plaintext highlighter-rouge">systemctl restart crowdsec</code> instead of
just <code class="language-plaintext highlighter-rouge">reload</code>, which was buggy in the 1.0.x versions.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0007-automatically-enable-online-hub.patch/">0007</a>:
Enables the online hub transparently when <code class="language-plaintext highlighter-rouge">cscli hub update</code> is called.</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0008-hub-disable-broken-scenario.patch/">0008</a>:
Removes a broken scenario (backported from upstream).</li>
  <li><a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/patches/0009-Improve-http-bad-user-agent-use-regexp-197.patch/">0009</a>:
Avoids false positives by using regular expressions with word boundaries
(backported from upstream).</li>
</ul>

<h1>Packaging separate resources</h1>

<p>Two different things live outside the
<a href="https://github.com/crowdsecurity/crowdsec/">main CrowdSec repository</a> and
need special consideration.</p>

<h2>Data</h2>

<p>The data part is mostly static, and scattered across other repositories. Let’s
look at the files shipped alongside <code class="language-plaintext highlighter-rouge">crowdsec</code> as of version <code class="language-plaintext highlighter-rouge">1.0.9</code>, which
should give an idea what they are all about (the filenames are quite
self-explanatory):</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">backdoors.txt</code></li>
  <li><code class="language-plaintext highlighter-rouge">bad_user_agents.txt</code></li>
  <li><code class="language-plaintext highlighter-rouge">cloudflare_ips.txt</code></li>
  <li><code class="language-plaintext highlighter-rouge">http_path_traversal.txt</code></li>
  <li><code class="language-plaintext highlighter-rouge">ip_seo_bots.txt</code></li>
  <li><code class="language-plaintext highlighter-rouge">rdns_seo_bots.regex</code></li>
  <li><code class="language-plaintext highlighter-rouge">rdns_seo_bots.txt</code></li>
  <li><code class="language-plaintext highlighter-rouge">sensitive_data.txt</code></li>
  <li><code class="language-plaintext highlighter-rouge">sqli_probe_patterns.txt</code></li>
  <li><code class="language-plaintext highlighter-rouge">xss_probe_patterns.txt</code></li>
</ul>

<h2>Hub</h2>

<p>The hub part is more challenging: <code class="language-plaintext highlighter-rouge">crowdsec</code> itself is a Security Engine, that
needs a number of collections to define what it should do. Those collections can
point to parsers, scenarios, postoverflows, and to other collections as well.</p>

<p>The hub is <a href="https://hub.crowdsec.net/">accessible online</a> but it felt a little
weird to force all users to start by having to download things right after
installing the package. It was agreed with upstream to ship a copy of the hub
data alongside the main <code class="language-plaintext highlighter-rouge">crowdsec</code> source tarball, and to install those files directly
in the <code class="language-plaintext highlighter-rouge">crowdsec</code> binary package. Those files would be used by default upon
installation, but users wishing to switch to the online hub can do so by using
the <code class="language-plaintext highlighter-rouge">cscli hub update</code> command.</p>

<h2>Implementation</h2>

<p>To deal with both the extra data and hub parts, the <code class="language-plaintext highlighter-rouge">crowdsec</code> source package
— which uses the  <code class="language-plaintext highlighter-rouge">3.0 (quilt)</code> source format —
leverages what the <code class="language-plaintext highlighter-rouge">dpkg-source</code> manpage calls <em>additional original tarballs</em>.
Let’s see what the file list of the <code class="language-plaintext highlighter-rouge">crowdsec</code> source package looks like:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">crowdsec_1.0.9-2.dsc</code>: entry point, containing metadata and listing other files.</li>
  <li><code class="language-plaintext highlighter-rouge">crowdsec_1.0.9-2.debian.tar.xz</code>: Debian packaging (<code class="language-plaintext highlighter-rouge">debian/</code> directory).</li>
  <li><code class="language-plaintext highlighter-rouge">crowdsec_1.0.9.orig.tar.gz</code>: original tarball, “main” one (from <a href="https://github.com/crowdsecurity/crowdsec">https://github.com/crowdsecurity/crowdsec</a>).</li>
  <li><code class="language-plaintext highlighter-rouge">crowdsec_1.0.9.orig-data1.tar.gz</code>: additional original tarball, for data files.</li>
  <li><code class="language-plaintext highlighter-rouge">crowdsec_1.0.9.orig-hub1.tar.gz</code>: additional original tarball, for the hub.</li>
</ul>

<p>When deploying the source package, the first additional original tarball is
automatically extracted under the <code class="language-plaintext highlighter-rouge">data1</code> directory, below the top-level
<code class="language-plaintext highlighter-rouge">crowdsec-1.0.9</code> directory, while the second one is extracted under the <code class="language-plaintext highlighter-rouge">hub1</code>
directory.</p>

<p>The <code class="language-plaintext highlighter-rouge">1</code> suffix is there just in case we needed or wanted to update either
additional original tarball while keeping the same <code class="language-plaintext highlighter-rouge">crowdsec</code> upstream version (i.e. <code class="language-plaintext highlighter-rouge">1.0.9</code> in this case).</p>

<p>Another well-known example for such additional original tarballs are
<em>language packs</em>. In Bullseye, <code class="language-plaintext highlighter-rouge">firefox-esr</code> comes with 97 of them!</p>

<h1>Plugging everything together: maintainer scripts</h1>

<p>Let’s start by mentioning two core components of the CrowdSec ecosystem:</p>

<ul>
  <li>the <a href="https://docs.crowdsec.net/docs/central_api/intro/">Central API</a>;</li>
  <li>the <a href="https://docs.crowdsec.net/docs/local_api/intro/">Local API</a>.</li>
</ul>

<p>Since CrowdSec is really about leveraging efforts from the community, it
seemed best to have the machine automatically register itself against the
Central API upon installation, by default. Since there are many ways to set up
CrowdSec, advanced users can disable automatic registration by creating a file
in advance, as documented in the
<a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/README.Debian/">README.Debian</a>.
This registration happens via <code class="language-plaintext highlighter-rouge">cscli api register</code>.</p>

<p>Regarding the Local API, it semeed best to stick to the loopback by default,
letting admins adjust the network configuration if desired. The machine
registers itself on the Local API via <code class="language-plaintext highlighter-rouge">cscli machines add</code>.</p>

<p>The final topic is not the easiest: collections!</p>

<p>Since replacing the offline hub with the online one should be possible, files
are deployed under <code class="language-plaintext highlighter-rouge">/var/lib/crowdsec</code> (making sure they aren’t treated as
<code class="language-plaintext highlighter-rouge">conffiles</code>, which would otherwise lead to many prompts during upgrades).
Unfortunately, at the moment, <code class="language-plaintext highlighter-rouge">crowdsec</code> doesn’t come with a way to manage local
collections. For the 1.0.x versions packaged in Bullseye, the hub was small
enough that enabling everything seemed reasonable and easy, and that’s what
got implemented: for each item found in the offline hub, create a symlink
under <code class="language-plaintext highlighter-rouge">/etc/crowdsec/</code>, pointing at it.</p>

<p>Everything mentioned in this section is implemented in
<a href="https://sources.debian.org/src/crowdsec/1.0.9-2/debian/postinst/">the <code class="language-plaintext highlighter-rouge">crowdsec.postinst</code> maintainer script</a>
which is called by <code class="language-plaintext highlighter-rouge">dpkg</code> after installation, in particular with the <code class="language-plaintext highlighter-rouge">configure</code>
action.</p>

<h1>Conclusion</h1>

<p>Debian 11 shipped with the Security Engine packaged as <code class="language-plaintext highlighter-rouge">crowdsec</code>, which made it possible
to join the crowd and report suspicious behaviours. Unfortunately, there wasn’t
enough time to also package Remediation Components (bouncers), which are responsible for
actually taking action.</p>

<div style="margin-top: 2em; margin-bottom: 2em">

  <table class="table local-no-fullwidth local-is-centered is-bordered">
    <thead>
      <tr>
        <th>Package</th>
        <th>Version</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>crowdsec</td>
        <td>1.0.9-2+b4</td>
      </tr>
    </tbody>
  </table>

  <p class="has-text-centered"><strong>Version table for Debian 11 “Bullseye”</strong></p>

</div>

<p>See what happened after that, in the next blog post:
<a href="/blog/2023/06/13/packaging-crowdsec-for-debian-bookworm/">Packaging CrowdSec for Debian: Bookworm (Debian 12)</a>.</p>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[Taking a look back at the collaboration between CrowdSec and DEBAMAX during the Bullseye release cycle.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/crowdsec.png" /><media:content medium="image" url="https://debamax.com/crowdsec.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Debian vs. non-free-firmware</title><link href="https://debamax.com/blog/2023/02/27/debian-versus-non-free-firmware/" rel="alternate" type="text/html" title="Debian vs. non-free-firmware" /><published>2023-02-27T02:30:00+01:00</published><updated>2023-02-27T02:30:00+01:00</updated><id>https://debamax.com/blog/2023/02/27/debian-versus-non-free-firmware</id><content type="html" xml:base="https://debamax.com/blog/2023/02/27/debian-versus-non-free-firmware/"><![CDATA[<h1>Introduction</h1>

<h2>The spark</h2>

<p>This story starts with the <a href="https://www.debian.org/vote/2022/vote_003">2022 General Resolution about non-free
firmware</a>, with our <a href="https://www.debian.org/social_contract">Social
Contract</a> getting an extra sentence in
the “Works that do not meet our free software standards” section:</p>

<blockquote>
  <p>The Debian official media may include firmware that is otherwise not part of
the Debian system to enable use of Debian with hardware that requires such
firmware.</p>
</blockquote>

<h2>Debian vs. firmware</h2>

<p>Firmware support has always been a major pain point for Debian users (beginners
and experts) but it’s grown bigger over time:</p>

<ul>
  <li>Laptops were known to be a little problematic, with some wireless network
adapters requiring firmware. A usual workaround would be to install using the
wired network adapter, adjusting the package manager’s configuration, and
installing the required firmware package to get the wireless adapter up and
running. With recent laptops, there might be no wired network adapter at all!</li>
  <li>Nowadays, components that might require firmware include: graphics adapters
(even for basic operations, using free Linux kernel modules), wired or
wireless network adapters, and even sound cards!</li>
  <li>Additionally, CPUs are riddled with bugs and one is supposed to apply CPU
microcode updates to avoid running into known issues!</li>
</ul>

<p>Let’s quote the <a href="https://www.debian.org/doc/debian-policy/ch-archive.html">Debian
Policy</a> regarding the
Debian archive:</p>

<blockquote>
  <p>The main archive area forms the Debian distribution.</p>

  <p>Packages in the other archive areas (contrib, non-free) are not considered to
be part of the Debian distribution, although we support their use and provide
infrastructure for them.</p>
</blockquote>

<p>Unfortunately, firmware packages usually don’t meet the <a href="https://www.debian.org/social_contract#guidelines">Debian Free Software
Guidelines (DFSG)</a>, meaning
they end up in the <code class="language-plaintext highlighter-rouge">non-free</code> area, which means they cannot be included on
official media (built using only <code class="language-plaintext highlighter-rouge">main</code>).</p>

<p>That explains why installation images combining packages from <code class="language-plaintext highlighter-rouge">main</code>, <code class="language-plaintext highlighter-rouge">contrib</code>,
and <code class="language-plaintext highlighter-rouge">non-free</code> have always been marked as unofficial and haven’t been advertised
prominently, even if they’ve been increasingly much more useful than the
official installation images.</p>

<p>The <a href="https://www.debian.org/vote/2022/vote_003">2022 General Resolution about non-free
firmware</a> changes the status quo, by
allowing non-free firmware packages to be included in official installation
images directly, making everyone’s life easier, finally!</p>

<p>This blog post discusses Cyril’s journey into making that happen for Bookworm.</p>

<p>The following is organized on a per-topic fashion, but progress hasn’t been
linear!</p>

<h1>The <code class="language-plaintext highlighter-rouge">non-free-firmware</code> archive area</h1>

<p>Let’s start with <abbr title="Debian Archive Kit">dak</abbr>, which is used by
the FTP team to manage the Debian archive: <code class="language-plaintext highlighter-rouge">main</code>, <code class="language-plaintext highlighter-rouge">contrib</code>, and <code class="language-plaintext highlighter-rouge">non-free</code>
have been around forever, and <code class="language-plaintext highlighter-rouge">non-free-firmware</code> needed to get <a href="https://salsa.debian.org/ftp-team/dak/-/commit/895e796bcb6373a13b46eb09b5422af5bec05b0b">added
alongside</a>.</p>

<p>At that point, the archive was ready to accept packages into <code class="language-plaintext highlighter-rouge">non-free-firmware</code>
but a few tweaks were still needed, like <a href="https://salsa.debian.org/ftp-team/dak/-/merge_requests/267/commits">synchronizing DEP-11 metadata for this
brand new
component</a>
and <a href="https://salsa.debian.org/ftp-team/dak/-/commit/09c13f37c04f8dcd994f3ee2e3e2137b03db7cbb">a follow-up
bugfix</a>
(since at some point <a href="https://appstream.debian.org/">https://appstream.debian.org/</a> had data under
<code class="language-plaintext highlighter-rouge">unstable/non-free/dep11/</code> but not under <code class="language-plaintext highlighter-rouge">testing/non-free/dep11/</code>).</p>

<h1>Moving firmware packages</h1>

<h2>Package uploads</h2>

<p><a href="https://lists.debian.org/debian-boot/2023/01/msg00150.html">A first draft</a> was
sent to the installer team and to the maintainers of firmware packages that
seemed most interesting to have on installation images. Merge requests were
filed for packages with repositories hosted on <a href="https://salsa.debian.org/">https://salsa.debian.org/</a> and
bug reports with patches were filed for the others. Maintainers were quite
supportive, some uploaded their packages swiftly with the proposed changes, some
asked for sponsorship, and Cyril uploaded the few remaining ones after a while.</p>

<p>The <code class="language-plaintext highlighter-rouge">firmware-nonfree</code> source package was a little challenging, as it reuses
some parts of the <code class="language-plaintext highlighter-rouge">linux</code> packaging, which had been updated. This explains why
the <a href="https://salsa.debian.org/kernel-team/firmware-nonfree/-/merge_requests/48">Update to new linux-support API, move packages to non-free-firmware/*
sections</a>
merge request was the biggest set of changes overall: other packages only
required an updated <code class="language-plaintext highlighter-rouge">Section</code> in the <code class="language-plaintext highlighter-rouge">debian/control</code> file and a descriptive
changelog entry. A smaller <a href="https://salsa.debian.org/kernel-team/firmware-nonfree/-/merge_requests/49">Address
autorejects</a>
merge request made sure that <code class="language-plaintext highlighter-rouge">lintian</code> as run by dak would be happy with that
second upload.</p>

<p>Since packages were switching from an archive area to another one, they had to
go <a href="https://ftp-master.debian.org/#new">through NEW for review</a>, which in turn
meant that uploads needed to include binaries. Since most packages contain data,
they are usually <code class="language-plaintext highlighter-rouge">Architecture: all</code>, meaning all uploads were <code class="language-plaintext highlighter-rouge">source+all</code>.</p>

<p>Thanks to a great support from the FTP team, all uploads were processed very
quickly.</p>

<h2>Migration to testing, autobuilding</h2>

<p>Usually there’s a bit of a cognitive dissonance between uploads going through
NEW requiring binaries alongside the source, and britney (the release team tool
managing migrations from <code class="language-plaintext highlighter-rouge">unstable</code> to <code class="language-plaintext highlighter-rouge">testing</code>) which insists on having all
binaries built on build daemons (rather than uploaded by the maintainer).</p>

<p>That being said, the usual source-only, follow-up upload after a package has
been accepted from NEW… was not necessary in this specific case since the “Not
built on buildd” check is only applied to packages in <code class="language-plaintext highlighter-rouge">main</code>. That means britney
required not changes at all!</p>

<p>Everything looked good, until the Debian Kernel Team prepared a new upstream release for
<code class="language-plaintext highlighter-rouge">firmware-nonfree</code>, and only the source package appeared in the archive, while
binary packages were nowhere to be found. It turned out that the <a href="https://buildd.debian.org">buildd
infrastructure</a> <a href="https://bugs.debian.org/1030301">hardcoded components in various
places</a>, but thankfully Philipp Kern and
Aurélien Jarno took care of it in just a few days, and <code class="language-plaintext highlighter-rouge">firmware-nonfree</code> binary
packages finally appeared in the archive.</p>

<h1>Using moved firmware packages</h1>

<h2>Debian Installer</h2>

<h3>hw-detect</h3>

<p>Let’s start with the <code class="language-plaintext highlighter-rouge">hw-detect</code> component. Its <code class="language-plaintext highlighter-rouge">check-missing-firmware</code> script
keeps an eye on Linux kernel logs (via <code class="language-plaintext highlighter-rouge">dmesg</code>) and looks for a specific pattern
to notice modules requesting firmware files. Here’s an example, with the <code class="language-plaintext highlighter-rouge">r8152</code>
module requesting <code class="language-plaintext highlighter-rouge">rtl_nic/rtl8153a-3.fw</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kernel: [   75.671338] r8152 4-2.4:1.0: firmware: failed to load rtl_nic/rtl8153a-3.fw (-2)
</code></pre></div></div>

<p>When that happens, <code class="language-plaintext highlighter-rouge">check-missing-firmware</code> tries to find the requested files in
the firmware packages found in the <code class="language-plaintext highlighter-rouge">/firmware</code> directory, if any. As a last
resort, users get a prompt with the requested files, and they get asked whether
they’d like to search for firmware on external storage (e.g. a USB stick).</p>

<p>Until now, with official installation images, there were no such firmware
packages, and the prompt was shown every single time. With unofficial
installation images including firmware packages, seeing this prompt was very
unlikely (but that could still happen, e.g. for recent hardware).</p>

<p>Starting with Debian Installer Bookworm Alpha 2, official installation
images include firmware packages from <code class="language-plaintext highlighter-rouge">main</code> and <code class="language-plaintext highlighter-rouge">non-free-firmware</code>,
and seeing this prompt should be unlikely as well.</p>

<p>On the technical side, <code class="language-plaintext highlighter-rouge">check-missing-firmware</code> will automatically use a
<code class="language-plaintext highlighter-rouge">Contents-firmware</code> index if it’s available in the firmware directory, mapping
each firmware file to the firmware package that ships it, and to the archive
area it was found in. If there’s no <code class="language-plaintext highlighter-rouge">Contents-firmware</code> index, the old
implementation is used as a fallback, i.e. checking the contents of each
firmware package; and determining the archive area by looking at the <code class="language-plaintext highlighter-rouge">Section</code>
field of the firmware package.</p>

<p>Now that we have found one or more firmware packages for the requesting files:</p>

<ul>
  <li>The firmware packages are unpacked in the installer environment, making their
firmware files available to the running Linux kernel.</li>
  <li>The relevant modules are reloaded, so that they can request their firmware
files again… and find them this time.</li>
  <li>The firmware packages are queued for installation in the system getting
installed.</li>
  <li>The detected archive areas are used to tweak the default APT configuration for
the system getting installed.</li>
</ul>

<p>Of course, that only works for Linux kernel modules that are actually loaded,
and actively requesting firmware files. The installer environment uses a generic
video driver, doesn’t feature sound support (unless speech synthesis is used but that’s another story)… so many things could get
overlooked. Thankfully, we have another way to detect firmware packages that
might be needed, based on modalias information: we use the <code class="language-plaintext highlighter-rouge">udev</code> database and
patterns found in the <code class="language-plaintext highlighter-rouge">/firmware</code> directory to detect which firmware packages
can be useful. While this feature has been added during the Bullseye release
cycle, it’s only useful when firmware packages are indeed included in the
installation images!</p>

<p>Feedback loops are usually much shorter when testing inside a virtual machine,
and the test setup revealed a fun issue:</p>

<ul>
  <li>Standard VM created by <code class="language-plaintext highlighter-rouge">libvirt</code> with default settings;</li>
  <li>Realtek-based Wi-Fi dongle connected over USB on the host, shared as a USB
device via <code class="language-plaintext highlighter-rouge">libvirt</code>.</li>
</ul>

<p>This test setup results in the following kernel logs:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rtl8192cu: Loading firmware rtlwifi/rtl8192cufw_TMSC.bin
usb 4-1.5: firmware: failed to load rtlwifi/rtl8192cufw_TMSC.bin (-2)
usb 4-1.5: Direct firmware load for rtlwifi/rtl8192cufw_TMSC.bin failed with error -2
usb 4-1.5: firmware: failed to load rtlwifi/rtl8192cufw.bin (-2)
usb 4-1.5: Direct firmware load for rtlwifi/rtl8192cufw.bin failed with error -2
</code></pre></div></div>

<p>This means we record the <code class="language-plaintext highlighter-rouge">usb</code> module has the one requesting firmware files,
which we then try to reload after deploying <code class="language-plaintext highlighter-rouge">firmware-realtek</code>. That cannot
work!</p>

<p>Therefore, a special case was added: when the module requesting firmware files
is <code class="language-plaintext highlighter-rouge">usb</code>, search the USB bus using port and device information to find the
underlying device and the module managing it. With the above example, this makes
it possible to trade <code class="language-plaintext highlighter-rouge">usb 4-1.5</code> for <code class="language-plaintext highlighter-rouge">rtl8192cu</code>. Funnily enough, that wasn’t
noticed immediately, as this specific module requests firmware files but <em>can</em>
work without them!</p>

<p>Since it’s expected that some users don’t want to install firmware packages even
if they are present on installation images, Cyril <a href="https://bugs.debian.org/1029848">started a
discussion</a> on how to make it possible for
users to opt out. It was decided to support a <code class="language-plaintext highlighter-rouge">firmware=never</code> boot-time option
to disable firmware support entirely.</p>

<p>Uploads:
<a href="https://tracker.debian.org/news/1412823/accepted-hw-detect-1153-source-into-unstable/">hw-detect 1.153</a>,
<a href="https://tracker.debian.org/news/1415097/accepted-hw-detect-1154-source-into-unstable/">hw-detect 1.154</a>,
<a href="https://tracker.debian.org/news/1415094/accepted-preseed-1113-source-into-unstable/">preseed 1.113</a>.</p>

<h3>apt-setup</h3>

<p>This component is responsible for configuring APT for the system getting
installed. While it defaults to only enabling <code class="language-plaintext highlighter-rouge">main</code>, other archive areas can be
selected when using an expert installation (asking many more questions).</p>

<p>When installing firmware packages from outside <code class="language-plaintext highlighter-rouge">main</code>, it’s important to
configure APT accordingly, so that those packages can be upgraded later on.</p>

<p>Small updates were needed for <code class="language-plaintext highlighter-rouge">apt-setup</code> to gain <code class="language-plaintext highlighter-rouge">non-free-firmware</code> support:</p>

<ul>
  <li>A new <code class="language-plaintext highlighter-rouge">apt-setup/non-free-firmware</code> template was added, only shown during
expert installation (like <code class="language-plaintext highlighter-rouge">apt-setup/contrib</code> and <code class="language-plaintext highlighter-rouge">apt-setup/non-free</code>),
but it can also be enabled by <code class="language-plaintext highlighter-rouge">hw-detect</code> when firmware packages from
<code class="language-plaintext highlighter-rouge">non-free-firmware</code> are deployed (see above).</li>
  <li>Generators were modified to take all three templates into account, making sure
the relevant archive areas are enabled for the base distribution (<code class="language-plaintext highlighter-rouge">bookworm</code>)
but also for the security suite (<code class="language-plaintext highlighter-rouge">bookworm-security</code>), and optional extra
suites (e.g. <code class="language-plaintext highlighter-rouge">bookworm-updates</code>).</li>
</ul>

<p>Upload:
<a href="https://tracker.debian.org/news/1412348/accepted-apt-setup-10174-source-into-unstable/">apt-setup 1:0.174</a>.</p>

<h2>netcfg</h2>

<p>It’s very tempting to only install a basic system when working on the installer,
skipping a full desktop installation. While doing so, even once firmware
packages are in place, once <code class="language-plaintext highlighter-rouge">hw-detect</code> and <code class="language-plaintext highlighter-rouge">apt-setup</code> have done their work
flawlessly, rebooting into the installed system can be surprising: the wireless
network that was used during the installation is nowhere to be seen, meaning no
connectivity at all!</p>

<p>Cyril’s investigation revealed several issues, filed as
<a href="https://bugs.debian.org/1029352">#1029352</a> against <code class="language-plaintext highlighter-rouge">netcfg</code>.</p>

<p>One of those issues turned out to be a regression in <code class="language-plaintext highlighter-rouge">ifupdown</code> 0.8.40: trying
to add support for <code class="language-plaintext highlighter-rouge">allow-hotplug</code> interfaces in <code class="language-plaintext highlighter-rouge">systemctl restart networking</code>
ended up breaking support for them at boot-time! Cyril proposed a <a href="https://salsa.debian.org/debian/ifupdown/-/commit/219d60b64111681066478fc3672e386918bec68b">minimal
patch</a>
that would make sure the previous behaviour would be kept (ensuring boot-time is
fine), while retaining the addition of <code class="language-plaintext highlighter-rouge">restart</code> support. Better solutions are
certainly possible, but there are still only 24 hours in a day, and swiftly
restoring the existing status quo looked very appealing at this point of the
release cycle…</p>

<p>The other two issues were fixed in <code class="language-plaintext highlighter-rouge">netcfg</code>, by activating support for wireless
interfaces when using <code class="language-plaintext highlighter-rouge">ifupdown</code> (i.e. when <code class="language-plaintext highlighter-rouge">network-manager</code> isn’t used)
instead of explicitly disabling it, and by working around what seems to be a
file format limitation regarding <code class="language-plaintext highlighter-rouge">/etc/network/interfaces</code> and multiple stanzas
with wireless options (e.g. DHCP for IPv4, SLAAC for IPv6).</p>

<p>Upload:
<a href="https://tracker.debian.org/news/1412965/accepted-netcfg-1182-source-into-unstable/">netcfg 1.182</a>.</p>

<h2>Image generation – <code class="language-plaintext highlighter-rouge">debian-cd</code></h2>

<p>The <code class="language-plaintext highlighter-rouge">debian-installer</code> source package is responsible for stitching many udebs
(for µdebs) together to generate various targets. Here’s a full list on <code class="language-plaintext highlighter-rouge">amd64</code>:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">cdrom_isolinux</code></li>
  <li><code class="language-plaintext highlighter-rouge">cdrom_gtk</code></li>
  <li><code class="language-plaintext highlighter-rouge">cdrom-xen</code></li>
  <li><code class="language-plaintext highlighter-rouge">netboot</code></li>
  <li><code class="language-plaintext highlighter-rouge">netboot-gtk</code></li>
  <li><code class="language-plaintext highlighter-rouge">netboot-xen</code></li>
  <li><code class="language-plaintext highlighter-rouge">hd-media</code></li>
  <li><code class="language-plaintext highlighter-rouge">hd-media_gtk</code></li>
  <li><code class="language-plaintext highlighter-rouge">monolithic</code></li>
</ul>

<p>Depending on which udebs are getting patched, it’s usual for developers to use
<code class="language-plaintext highlighter-rouge">netboot</code> or <code class="language-plaintext highlighter-rouge">netboot-gtk</code> which generate <code class="language-plaintext highlighter-rouge">dest/netboot/mini.iso</code> and
<code class="language-plaintext highlighter-rouge">dest/netboot/gtk/mini.iso</code> respectively, which can be passed to <code class="language-plaintext highlighter-rouge">kvm</code> via the
<code class="language-plaintext highlighter-rouge">-cdrom</code> flag.</p>

<p>But since the whole point was improving support for firmware packages on
installation images, going one step further was needed: looking into
<code class="language-plaintext highlighter-rouge">debian-cd</code>! This is the tool that consumes what the <code class="language-plaintext highlighter-rouge">debian-installer</code> source
package produces, checks what is available in the Debian archive, and generates
installation images accordingly: the famous <code class="language-plaintext highlighter-rouge">netinst</code> image, CD sets, DVD sets,
etc.</p>

<p>This is definitely not the easiest part to set up for a fly-by contribution:</p>

<ul>
  <li>the code lives in the <a href="https://salsa.debian.org/images-team/debian-cd">debian-cd</a> repository;</li>
  <li>the configuration lives in the <a href="https://salsa.debian.org/images-team/setup">setup</a> repository;</li>
  <li><code class="language-plaintext highlighter-rouge">debian-cd</code> requires access to a local mirror.</li>
</ul>

<p>Cyril was already familiar with those requirements, having provided patches in
the past, and catching up with latest <code class="language-plaintext highlighter-rouge">debian-cd</code> master should have been rather
easy. There were a few changes along the way though, related to how local debs
and udebs can be included in development builds… That’s a feature that’s never
used for official builds (as published on <a href="https://cdimage.debian.org/cdimage/"><code class="language-plaintext highlighter-rouge">cdimage.debian.org</code></a>), and
several rounds of clean-up were needed to make builds with local packages work
again.</p>

<p>Those changes made it possible to build images with <code class="language-plaintext highlighter-rouge">easy-build.sh</code>, which is a
convenient wrapper for the official <code class="language-plaintext highlighter-rouge">build.sh</code> command, using <code class="language-plaintext highlighter-rouge">debian-installer</code>
build artifacts (via the <code class="language-plaintext highlighter-rouge">cdrom_isolinux</code> and <code class="language-plaintext highlighter-rouge">cdrom_gtk</code> targets):</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">cdrom/debian-cd_info.tar.gz</code></li>
  <li><code class="language-plaintext highlighter-rouge">cdrom/gtk/debian-cd_info.tar.gz</code></li>
  <li><code class="language-plaintext highlighter-rouge">cdrom/gtk/vmlinuz</code></li>
  <li><code class="language-plaintext highlighter-rouge">cdrom/gtk/initrd.gz</code></li>
  <li><code class="language-plaintext highlighter-rouge">cdrom/vmlinuz</code></li>
  <li><code class="language-plaintext highlighter-rouge">cdrom/xen/debian.cfg</code></li>
  <li><code class="language-plaintext highlighter-rouge">cdrom/initrd.gz</code></li>
</ul>

<p>and a set of local packages, including the <code class="language-plaintext highlighter-rouge">netcfg</code> and <code class="language-plaintext highlighter-rouge">apt-setup</code> udebs that
get loaded during the installation process, and a patched <code class="language-plaintext highlighter-rouge">ifupdown</code> package to
side-step the regression mentioned earlier
(<a href="https://bugs.debian.org/1022843">#1022843</a>).</p>

<p>Once <code class="language-plaintext highlighter-rouge">netinst</code> images could be produced, actual work on <code class="language-plaintext highlighter-rouge">debian-cd</code> could
start!</p>

<ul>
  <li>Extending the <code class="language-plaintext highlighter-rouge">Contents-firmware</code> index: initially mapping firmware files to
firmware packages, information about archive areas was added, so that
<code class="language-plaintext highlighter-rouge">hw-detect</code> can configure <code class="language-plaintext highlighter-rouge">apt-setup</code> properly when deploying firmware
packages based on <code class="language-plaintext highlighter-rouge">dmesg</code> parsing (missing firmware files).</li>
  <li>Improving robustness around DEP-11 metadata: making sure source data files are
present (which led to spotting missing files in the archive, see <code class="language-plaintext highlighter-rouge">dak</code> merge
request mentioned earlier, and some missing architectures in an
<code class="language-plaintext highlighter-rouge">appstream.debian.org</code> configuration file).</li>
  <li>Extending the <code class="language-plaintext highlighter-rouge">*.patterns</code> files generated from DEP-11 metadata with matching
<code class="language-plaintext highlighter-rouge">*.component</code> files, so that <code class="language-plaintext highlighter-rouge">hw-detect</code> can configure <code class="language-plaintext highlighter-rouge">apt-setup</code> properly
when deploying firmware packages based on modalias information.</li>
  <li>Reworking the <code class="language-plaintext highlighter-rouge">generate_firmware_patterns</code> file so that its logic could get
reused outside image builds (see below).</li>
  <li>Refreshing the hardcoded list of devices supported by <code class="language-plaintext highlighter-rouge">firmware-sof-signed</code>,
since that modalias information isn’t available for Intel SOF (audio firmware).</li>
</ul>

<p>Last but not least, the firmware package lookup was adjusted. Initially,
packages with names matching a given pattern would be searched across all archive
areas, since the firmware inclusion code was being used only for unofficial
builds, with <code class="language-plaintext highlighter-rouge">contrib</code> and <code class="language-plaintext highlighter-rouge">non-free</code> enabled. That lookup was adjusted to only
search the configured archive areas, meaning <code class="language-plaintext highlighter-rouge">main</code>, <code class="language-plaintext highlighter-rouge">non-free</code>, and
<code class="language-plaintext highlighter-rouge">non-free-firmware</code> while packages were getting moved from <code class="language-plaintext highlighter-rouge">non-free</code> to
<code class="language-plaintext highlighter-rouge">non-free-firmware</code>. Once all packages were uploaded and migrated to testing,
<code class="language-plaintext highlighter-rouge">non-free</code> was dropped, giving us the desired <code class="language-plaintext highlighter-rouge">main</code> and <code class="language-plaintext highlighter-rouge">non-free-firmware</code>
combination.</p>

<p>Outside image builds, <code class="language-plaintext highlighter-rouge">make-firmware-image</code> is a tool that generates firmware
archives (<code class="language-plaintext highlighter-rouge">tar</code>, <code class="language-plaintext highlighter-rouge">zip</code>, and <code class="language-plaintext highlighter-rouge">cpio</code> formats), so that users can fetch a single
file that contains all firmware packages in one step. Since those archives are
still going to be useful for <code class="language-plaintext highlighter-rouge">netboot</code> images (they are produced by a
<code class="language-plaintext highlighter-rouge">debian-installer</code> build, using packages from <code class="language-plaintext highlighter-rouge">main</code> only, meaning absolutely no
firmware), it was rewritten to reuse the updated firmware package lookup from
<code class="language-plaintext highlighter-rouge">generate_firmware_patterns</code>, and to also generate metadata to help <code class="language-plaintext highlighter-rouge">hw-detect</code>
and <code class="language-plaintext highlighter-rouge">apt-setup</code> (i.e. <code class="language-plaintext highlighter-rouge">Contents-firmware</code> index plus <code class="language-plaintext highlighter-rouge">*.patterns</code> and
<code class="language-plaintext highlighter-rouge">*.component</code> files). The
<a href="https://wiki.debian.org/DebianInstaller/NetbootFirmware">DebianInstaller/NetbootFirmware</a>
wiki page was updated accordingly, pointing to the new, official location for
those firmware archives: <a href="https://cdimage.debian.org/cdimage/firmware/">https://cdimage.debian.org/cdimage/firmware/</a>.</p>

<p>Upload:
<a href="https://tracker.debian.org/news/1423362/accepted-debian-cd-320-source-into-unstable/">debian-cd 3.2.0</a>.</p>

<h1>Documentation</h1>

<p>With firmware packages moved to <code class="language-plaintext highlighter-rouge">non-free-firmware</code> and migrated to <code class="language-plaintext highlighter-rouge">testing</code>,
with smarter Debian Installer components, and with <code class="language-plaintext highlighter-rouge">debian-cd</code> including firmware
packages with appropriate metadata, all the technical parts seemed in place.</p>

<p>Finally, the focus could move to writing some documentation. Admittedly, we
could have started there, but it seemed to make sense to make good progress on
the technical side first, and to check whether the initial plan would work out,
before thinking about documentation.</p>

<p>The following “must have” were identified, and patches were provided:</p>

<ul>
  <li>the <a href="https://salsa.debian.org/installer-team/installation-guide/-/merge_requests/23">installation guide</a>;</li>
  <li>the <a href="https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/138">release notes</a>;</li>
  <li>and the <a href="https://wiki.debian.org/DebianInstaller/NetbootFirmware">DebianInstaller/NetbootFirmware</a>
wiki page.</li>
</ul>

<p>The <a href="https://www.debian.org/devel/debian-installer/">Debian Installer</a> landing
page in the Debian Developers’ Corner of the website was updated when
<a href="https://www.debian.org/devel/debian-installer/News/2023/20230219">Debian Installer Bookworm Alpha 2</a>
was published.</p>

<p>The rest of the website contains many references to the usual <code class="language-plaintext highlighter-rouge">main contrib
non-free</code> triplet, and a <a href="https://bugs.debian.org/1030009">heads-up bug report</a>
was filed accordingly, so that Cyril’s colleagues could take over. Meanwhile,
<a href="https://packages.debian.org/">https://packages.debian.org/</a> was updated to also index packages in the new
<code class="language-plaintext highlighter-rouge">non-free-firmware</code> archive area (with <a href="https://salsa.debian.org/webmaster-team/packages/-/merge_requests/28">this merge
requested</a>
getting deployed live so that <code class="language-plaintext highlighter-rouge">testing</code> and <code class="language-plaintext highlighter-rouge">unstable</code> users could locate
packages that seemed to have disappeared).</p>

<p>Meanwhile, an
<a href="https://salsa.debian.org/apt-team/apt/-/merge_requests/282">APT patch is cooking</a>,
to help users migrating from <code class="language-plaintext highlighter-rouge">bullseye</code> to <code class="language-plaintext highlighter-rouge">bookworm</code>.</p>

<p>Upload:
<a href="https://tracker.debian.org/news/1421627/accepted-installation-guide-20230215-source-into-unstable/">installation-guide 20230215</a>.</p>

<p>Online documentation:</p>

<ul>
  <li><a href="https://www.debian.org/releases/bookworm/amd64/">Installation guide for Bookworm</a>.</li>
  <li>Release notes for Bookworm: <a href="https://www.debian.org/releases/bookworm/amd64/release-notes/ch-whats-new#archive-areas">What’s new in Debian 12</a>, <a href="https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information#non-free-split">Issues to be aware of for bookworm</a>.</li>
  <li>Debian Policy hasn’t been updated to define <code class="language-plaintext highlighter-rouge">non-free-firmware</code> yet,
see <a href="https://bugs.debian.org/1029211">#1029211</a>.</li>
</ul>

<h1>Stitching everything together: Debian Installer Bookworm Alpha 2</h1>

<p>We’ve published the first installer release with hopefully all the bits in
place:
<a href="https://www.debian.org/devel/debian-installer/News/2023/20230219">Debian Installer Bookworm Alpha 2</a>.</p>

<p>You can join the fun, by installing with
<a href="https://www.debian.org/devel/debian-installer/">the brand new images</a>,
and by also <a href="https://www.debian.org/releases/bookworm/amd64/ch05s04.en.html#problem-report">submitting installation reports</a>.
Those reports are always welcome, even if the installation process was uneventful. With
those important changes regarding firmware support, it’s very important for
us to understand if that’s indeed helping as much as we hoped, or if we have to
go back to the drawing board because we missed important things…</p>

<p>The importance of CPU microcode updates was underlined earlier, and
<a href="https://salsa.debian.org/installer-team/hw-detect/-/commit/1f54fb8dd8e208a978207a0259d3bc245634bc49">support is getting prepared</a>
for the next release. <a href="https://salsa.debian.org/installer-team/hw-detect/-/commit/7498281605ad951b6e29f284a00fe00fe219d649">Some</a>
<a href="https://salsa.debian.org/installer-team/hw-detect/-/commit/ead9c6a980ad179b7c6f43889ab3512345fde389">optimizations</a>
are lined up already, and we still need to
<a href="https://bugs.debian.org/1029849">generate the promised report</a>, since some
people might want to keep a close eye on the firmware packages getting installed.</p>

<p>On a personal note, Cyril would like to thank Steve McIntyre for all the support
during the last 10 years! There are so many moving pieces involved in publishing
Debian Installer releases, it’s always easier when a fellow developer has got
your back! 💯</p>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[Making Debian installation easier, finally!]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/debian-firmware.png" /><media:content medium="image" url="https://debamax.com/debian-firmware.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Groundhog Day: reflashing system images made easy</title><link href="https://debamax.com/blog/2023/02/06/groundhog-day-reflashing-made-easy/" rel="alternate" type="text/html" title="Groundhog Day: reflashing system images made easy" /><published>2023-02-06T06:00:00+01:00</published><updated>2023-02-06T06:00:00+01:00</updated><id>https://debamax.com/blog/2023/02/06/groundhog-day-reflashing-made-easy</id><content type="html" xml:base="https://debamax.com/blog/2023/02/06/groundhog-day-reflashing-made-easy/"><![CDATA[<h2>Foreword: system images</h2>

<p>A recurring topic for <span class="small-caps">Debamax</span> is helping customers design and maintain
build systems for Debian-based system images. Such images are usually:</p>

<ul>
  <li>defined by an image size, a partition layout, and various filesystems;</li>
  <li>prepared by using <code class="language-plaintext highlighter-rouge">debootstrap</code> then installing customer-specific packages;</li>
  <li>deployed to target device, where some “first boot” logic runs self-tests,
registers the device, requests certificates, etc.</li>
</ul>

<p>The build system prepares those images and also saves metadata after each build
(manifests listing packages, versions, files, etc.), making it possible to run
consistency checks, compare successive builds, prepare upgrades, etc.</p>

<p>Building an image is usually easy, but deploying it might be a little more
tricky:</p>

<ul>
  <li>One might want to build or use an existing live system that can be booted off
from USB, and embed the system image in a specific partition, to be copied
over to internal storage from the live system. This approach works on all
x86-based systems.</li>
  <li>One might need to extract internal storage from target device, plug it onto a
different system, copy data over, and then insert it back. This approach might
be required for some ARM-based systems, and might require a specific host
configuration, e.g. using a Compute Module 4 IO Board to connect a Compute
Module 4, then using <a href="https://github.com/raspberrypi/usbboot">usbboot</a> to make
its internal storage accessible via USB mass storage.</li>
</ul>

<p>Those solutions are rather straightforward and usually acceptable when it comes
to assembling components when a product is getting manufactured, but they aren’t
really acceptable for developers or testers, because they lead to very long
feedback loops.</p>

<h2>Side-stepping the need to touch hardware</h2>

<p>A different approach would be deploying the system image via the network. For
<span class="small-caps">Debamax</span>’s use cases, the main question is the following:</p>

<blockquote>
  <p>Is the target device still accessible and in a sufficiently good shape?</p>
</blockquote>

<p>A positive answer would usually imply:</p>

<ul>
  <li>The device still boots: it’s possible to break the bootloader or the boot
sequence entirely.</li>
  <li>The network connectivity is fine enough for the device to be accessible via
the local network: a common setup involves a working DHCP client and root
access via SSH keypair, but there’s more than one way to do it.</li>
  <li>The package manager is happy to install some extra packages: sometimes, a
botched upgrade can leave some broken packages, which would prevent installing
other packages.</li>
</ul>

<p>If all those boxes are ticked, what about running <code class="language-plaintext highlighter-rouge">curl | dd</code> or <code class="language-plaintext highlighter-rouge">curl | xz |
dd</code> directly from the existing system? That would mean downloading the system
image from the build server (or from the developer’s machine), possibly
decompressing it on the fly, and then writing it to the target storage, without
needing to touch any hardware: no live system on USB, no dismantling, no
unscrewing, no storage adapter, etc.</p>

<p>The obvious problem is that programs are running, files are open, and the image
getting written to storage will likely get corrupted in the process. Instead of
trying to close each and every resource, the Groundhog Day approach is to tweak
the initramfs!</p>

<h2>Groundhog Day</h2>

<h3>Technical approach</h3>

<p>A bootloader is usually configured to boot a Linux kernel (e.g. <code class="language-plaintext highlighter-rouge">vmlinuz-5.10.0-21-amd64</code>)
alongside an initramfs (e.g. <code class="language-plaintext highlighter-rouge">initrd.img-5.10.0-21-amd64</code>). The latter is an archive that
contains scripts, binaries, kernel modules, and makes it possible to prepare the root
filesystem before passing control over to init (e.g. <code class="language-plaintext highlighter-rouge">systemd</code>).</p>

<p>In Debian, <code class="language-plaintext highlighter-rouge">initramfs-tools</code> is responsible for creating and updating those
archives, based on configuration files and hooks. Some packages integrate with
<code class="language-plaintext highlighter-rouge">initramfs-tools</code> to extend it. That’s the case for <code class="language-plaintext highlighter-rouge">cryptsetup-initramfs</code>, which
makes it possible to get passphrase prompts for LUKS-based storage, and for
<code class="language-plaintext highlighter-rouge">dropbear-initramfs</code>, which makes it possible to integrate a lightweight SSH
server into the initramfs, which in turn makes it possible to specify LUKS
passphrases over the network.</p>

<p>The idea behind Groundhog Day is to reuse the existing <code class="language-plaintext highlighter-rouge">dropbear-initramfs</code>
integration, and to tweak it a little so that the initramfs gets a <code class="language-plaintext highlighter-rouge">dropbear</code>
instance up and running, so that one can log in and do the <code class="language-plaintext highlighter-rouge">curl | dd</code> dance
before the system actually starts: this way, the image can be deployed without
risking any corruption!</p>

<p>The script found below is what we call the “groundhog payload”, which is only
one part of a larger tool that automates reflashing (that one is tailored to
customer needs and out of scope for this blog post). This payload is meant to be
used once on a given system, and to keep it short, it operates on some files
that shouldn’t be touched (below <code class="language-plaintext highlighter-rouge">/usr</code>). This was deemed acceptable as the
whole idea is to replace an existing system with a different image anyway!</p>

<h3>Implementation</h3>

<p>The <a href="groundhog-payload.sh">groundhog payload</a> is rather generic and works fine
with x86-based machines running <code class="language-plaintext highlighter-rouge">amd64</code> and featuring SSD-based storage, as well
as with Raspberry Pi Compute Module 3 and Compute Module 4 running <code class="language-plaintext highlighter-rouge">arm64</code> and
featuring eMMC-based storage. The Compute Module 3 requires some tweaks when it
comes to the networking stack though.</p>

<figure class="highlight">
  <pre><code class="language-sh" data-lang="sh"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
</pre></td><td class="code"><pre><span class="c">#!/bin/sh</span>
<span class="c"># © 2022-2023 Cyril Brulebois &lt;cyril@debamax.com&gt;</span>
<span class="c">#</span>
<span class="c"># This helps prepare a system to be reflashed via `curl | dd` from the</span>
<span class="c"># initramfs, over SSH.</span>
<span class="nb">set</span> <span class="nt">-e</span>

<span class="nv">codename</span><span class="o">=</span><span class="si">$(</span>lsb_release <span class="nt">-sc</span><span class="si">)</span>
<span class="k">case</span> <span class="s2">"</span><span class="nv">$codename</span><span class="s2">"</span> <span class="k">in
  </span>buster|bullseye<span class="p">)</span>
    <span class="nb">echo</span> <span class="s2">"Supported: </span><span class="nv">$codename</span><span class="s2">"</span>
    <span class="p">;;</span>
  <span class="k">*</span><span class="p">)</span>
    <span class="nb">echo</span> <span class="s2">"Unsupported: </span><span class="nv">$codename</span><span class="s2">"</span>
    <span class="nb">exit </span>1
  <span class="p">;;</span>
<span class="k">esac</span>

<span class="c"># Sanity check:</span>
<span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-s</span> /root/.ssh/authorized_keys <span class="o">]</span><span class="p">;</span> <span class="k">then
  </span><span class="nb">echo</span> <span class="s2">"E: missing or empty /root/.ssh/authorized_keys"</span>
  <span class="nb">exit </span>1
<span class="k">fi</span>

<span class="c"># Configure APT:</span>
<span class="nv">SOURCES</span><span class="o">=</span>/etc/apt/sources.list
<span class="nv">URLS</span><span class="o">=</span><span class="s2">"http://internal-mirror.example.com/debian http://deb.debian.org/debian"</span>
: <span class="o">&gt;</span> <span class="s2">"</span><span class="nv">$SOURCES</span><span class="s2">"</span>
<span class="k">for </span>url <span class="k">in</span> <span class="nv">$URLS</span><span class="p">;</span> <span class="k">do
  if </span>curl <span class="nt">-k</span> <span class="nt">-s</span> <span class="s2">"</span><span class="nv">$url</span><span class="s2">"</span> <span class="o">&gt;</span>/dev/null 2&gt;&amp;1<span class="p">;</span> <span class="k">then
    </span><span class="nb">echo</span> <span class="s2">"I: Adding </span><span class="nv">$url</span><span class="s2">"</span>
    <span class="nb">echo</span> <span class="s2">"deb </span><span class="nv">$url</span><span class="s2"> </span><span class="nv">$codename</span><span class="s2"> main"</span> <span class="o">&gt;&gt;</span> <span class="s2">"</span><span class="nv">$SOURCES</span><span class="s2">"</span>
  <span class="k">else
    </span><span class="nb">echo</span> <span class="s2">"W: Skipping </span><span class="nv">$url</span><span class="s2">"</span>
  <span class="k">fi
done
if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-s</span> <span class="s2">"</span><span class="nv">$SOURCES</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
  </span><span class="nb">echo</span> <span class="s2">"E: empty </span><span class="nv">$SOURCES</span><span class="s2">"</span>
  <span class="nb">exit </span>1
<span class="k">fi</span>

<span class="c"># Divert u-i to speed up initial installation:</span>
dpkg-divert <span class="nt">--divert</span> /usr/sbin/update-initramfs.disabled <span class="nt">--rename</span> /usr/sbin/update-initramfs
<span class="nb">ln</span> <span class="nt">-s</span> /bin/true /usr/sbin/update-initramfs

<span class="c"># Install packages:</span>
apt-get update
<span class="nv">DEBIAN_FRONTEND</span><span class="o">=</span>noninteractive apt-get <span class="nb">install</span> <span class="nt">-y</span> dropbear-initramfs cryptsetup-

<span class="c"># Trust root's keys:</span>
<span class="nb">cp</span> /root/.ssh/authorized_keys /etc/dropbear-initramfs/authorized_keys

<span class="c"># This doesn't seem required on amd64, but let's make sure everywhere:</span>
<span class="nb">sed</span> <span class="s1">'s/^DEVICE=.*/DEVICE=eth0/'</span> <span class="se">\</span>
    <span class="nt">-i</span> /etc/initramfs-tools/initramfs.conf

<span class="c"># The order is important, otherwise the link doesn't get up automatically!</span>
<span class="k">if </span><span class="nb">grep</span> <span class="nt">-qs</span> 3-compute-module /proc/device-tree/compatible<span class="p">;</span> <span class="k">then
  </span><span class="nb">cat</span> <span class="o">&gt;</span>/etc/initramfs-tools/modules <span class="o">&lt;&lt;</span><span class="no">EOF</span><span class="sh">
mii
phy_generic
smsc95xx
</span><span class="no">EOF
</span><span class="k">fi</span>

<span class="c"># Drop condition on /etc/crypttab (seen on buster, not on bullseye):</span>
<span class="nb">sed</span> <span class="s1">'/^\[ -r \/etc\/crypttab \] || exit 0$/d'</span> <span class="se">\</span>
    <span class="nt">-i</span> /usr/share/initramfs-tools/hooks/dropbear

<span class="c"># Inject curl:</span>
<span class="nb">sed</span> <span class="s1">'/^copy_exec.*dropbear/i copy_exec /usr/bin/curl /bin'</span> <span class="se">\</span>
    <span class="nt">-i</span> /usr/share/initramfs-tools/hooks/dropbear

<span class="c"># Wait for eth0 to come up before trying to run dropbear, which would</span>
<span class="c"># fail immediately otherwise. Make sure to match on the call, not the</span>
<span class="c"># function definition:</span>
<span class="nb">sed</span> <span class="s1">'/^run_dropbear /i max=20; for try in $(seq 1 $max); do if [ -e /sys/class/net/$DEVICE ]; then break; fi; if [ "$try" = "$max" ]; then exit 0; fi; sleep 1; done'</span> <span class="se">\</span>
    <span class="nt">-i</span> /usr/share/initramfs-tools/scripts/init-premount/dropbear

<span class="c"># Remember MAC address:</span>
<span class="nv">mac</span><span class="o">=</span><span class="si">$(</span>ip <span class="nb">link </span>show dev eth0 | <span class="nb">awk</span> <span class="s1">'/link\/ether/ {print $2}'</span><span class="si">)</span>
<span class="nb">sed</span> <span class="s2">"/^run_dropbear /i ip link set address </span><span class="nv">$mac</span><span class="s2"> dev eth0"</span> <span class="se">\</span>
    <span class="nt">-i</span> /usr/share/initramfs-tools/scripts/init-premount/dropbear

<span class="c"># Add 15-minute delay before trying to mount everything:</span>
<span class="nb">echo</span> <span class="s1">'sleep 15m'</span> <span class="se">\</span>
    <span class="o">&gt;&gt;</span> /usr/share/initramfs-tools/scripts/init-premount/dropbear

<span class="c"># Remove diversion:</span>
<span class="nb">rm</span> <span class="nt">-f</span> /usr/sbin/update-initramfs
dpkg-divert <span class="nt">--rename</span> <span class="nt">--remove</span> /usr/sbin/update-initramfs

<span class="c"># Rebuilding for the current ABI is sufficient:</span>
update-initramfs <span class="nt">-u</span> <span class="nt">-k</span> <span class="si">$(</span><span class="nb">uname</span> <span class="nt">-r</span><span class="si">)</span>
</pre></td></tr></tbody></table></code></pre>
</figure>

<table>
  <thead>
    <tr>
      <th>Lines</th>
      <th>Comments</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>8-17</td>
      <td>The payload modifies various files shipped by Debian packages. They didn’t change much between Debian 10 and Debian 11, but a few tweaks were needed. Small changes might be required for further versions too, so it makes sense to make it clear what is supported and what is not.</td>
    </tr>
    <tr>
      <td>25-40</td>
      <td>Depending on development vs. production builds, the existing system might be configured to use a specific repository that only knows about the packages that are actually useful for a given product. This part ensures that a full mirror (either internal to the company or a public Debian mirror) is configured, making it possible to install extra packages.</td>
    </tr>
    <tr>
      <td>42-44</td>
      <td>This saves some I/O by skipping all <code class="language-plaintext highlighter-rouge">update-initramfs</code> calls that happen automatically via <code class="language-plaintext highlighter-rouge">dpkg</code> triggers when some packages are installed.</td>
    </tr>
    <tr>
      <td>46-48</td>
      <td>This saves some I/O by installing dropbear without cryptsetup: they are usually useful together but all we need here is dropbear.</td>
    </tr>
    <tr>
      <td>50-51</td>
      <td>This configures dropbear with root’s <code class="language-plaintext highlighter-rouge">authorized_keys</code>.</td>
    </tr>
    <tr>
      <td>53-55</td>
      <td>Setting the network device might not be necessary but for the avoidance of doubt, this sets <code class="language-plaintext highlighter-rouge">eth0</code> explicitly — these system images use <code class="language-plaintext highlighter-rouge">net.ifnames=0</code> to force the old naming scheme.</td>
    </tr>
    <tr>
      <td>57-64</td>
      <td>The Compute Module 3 requires some specific modules to be included in the initramfs for the network interface to be usable.</td>
    </tr>
    <tr>
      <td>66-68</td>
      <td>As mentioned above, dropbear and cryptsetup are usually useful together, and some version of that dropbear hook would exit early if there’s no cryptsetup configuration. This ensures the hook continues.</td>
    </tr>
    <tr>
      <td>70-72</td>
      <td>This sed call locates where <code class="language-plaintext highlighter-rouge">dropbear</code> gets copied into the initramfs via <code class="language-plaintext highlighter-rouge">copy_exec</code>, and inserts a line that copies <code class="language-plaintext highlighter-rouge">curl</code>. The <code class="language-plaintext highlighter-rouge">copy_exec</code> function copies an executable but also the libraries it depends on. If needed, it’s possible to use the same logic to inject tools to tweak partitioning or filesystems within the initramfs (e.g. <code class="language-plaintext highlighter-rouge">sfdisk</code>, <code class="language-plaintext highlighter-rouge">partprobe</code>, <code class="language-plaintext highlighter-rouge">e2fsck</code>, <code class="language-plaintext highlighter-rouge">resize2fs</code>).</td>
    </tr>
    <tr>
      <td>74-78</td>
      <td>This sed call locates where the <code class="language-plaintext highlighter-rouge">run_dropbear</code> function is called, and inserts a waiting loop right before: the idea is to give <code class="language-plaintext highlighter-rouge">eth0</code> a little time to come up, otherwise dropbear would exit immediately.</td>
    </tr>
    <tr>
      <td>80-83</td>
      <td>This is needed for the Compute Module 3, as it doesn’t set the MAC address automatically, which would result in the system’s getting a different IP address via DHCP. Instead of implementing parsing <code class="language-plaintext highlighter-rouge">/proc/cmdline</code>, looking for the <code class="language-plaintext highlighter-rouge">smsc95xx.macaddr</code> parameter, and using it to set the right MAC address… save the current MAC address from the running system, and restore it from the initramfs.</td>
    </tr>
    <tr>
      <td>85-87</td>
      <td>With the usual dropbear and cryptsetup integration, the initramfs would wait for filesystems to be unlocked before continuing. In our case, there are no such filesystems that need to be unlocked, so the boot would continue. Sleeping for 15 minutes makes sure users have time to log in, run their <code class="language-plaintext highlighter-rouge">curl | dd</code> command, and reboot into the newly-deployed image. The system continues booting if users didn’t connect during that timeframe.</td>
    </tr>
    <tr>
      <td>89-94</td>
      <td>This reverts the <code class="language-plaintext highlighter-rouge">update-initramfs</code> optimization, and builds a single initramfs, for the kernel version that’s currently running.</td>
    </tr>
  </tbody>
</table>

<h3>Usage</h3>

<p>The focus of this blog post is really the groundhog payload detailed in the
previous section, and how it gets used really depends on developer or tester
needs. The following steps should be rather common anyway:</p>

<ol>
  <li>copy the payload to the target system, e.g. via scp;</li>
  <li>run the script and make sure it exits successfully;</li>
  <li>collect dropbear fingerprints;</li>
  <li>reboot the target system;</li>
  <li>connect to the target system (now waiting in its initramfs), using collected dropbear fingerprints;</li>
  <li>detect the target storage (e.g. <code class="language-plaintext highlighter-rouge">/dev/sda</code> vs <code class="language-plaintext highlighter-rouge">/dev/mmcblk0</code>);</li>
  <li>run <code class="language-plaintext highlighter-rouge">curl http://server/product-arch.img.xz | xz -d | dd of=/dev/sda-or-mmcblk0</code>;</li>
  <li>adjust partitioning and/or filesystems if desired;</li>
  <li>reboot the target.</li>
</ol>

<p>At this point the target should be running on the freshly-deployed image!</p>

<p>Some tips:</p>

<ul>
  <li>Fingerprints: Since dropbear uses a specific key format, new fingerprints will
show up when connecting to the target system, when it’s waiting in its
initramfs. A <code class="language-plaintext highlighter-rouge">known_hosts</code> entry can be crafted in advance, e.g. by using the
<code class="language-plaintext highlighter-rouge">ecdsa-sha2-nistp256</code> prefix, followed by the output of this command:<br />
<code class="language-plaintext highlighter-rouge">dropbearkey -f /etc/dropbear-initramfs/dropbear_ecdsa_host_key -y</code></li>
  <li>Deploying the image: It might be a good idea to fetch both image and
fingerprint (e.g. <code class="language-plaintext highlighter-rouge">image.img.xz</code> and <code class="language-plaintext highlighter-rouge">image.img.shasum</code>), to verify what was
just written to storage.</li>
  <li>Adjusting partitioning and/or filesystems would usually require shipping more
tools in the initramfs (see comments in the previous section about
<code class="language-plaintext highlighter-rouge">copy_exec</code>). It is common for a system image to feature some “first boot”
logic that extends partitions and filesystems to use the whole storage though.</li>
  <li>Rebooting from the initramfs might be tricky: the <code class="language-plaintext highlighter-rouge">poweroff</code> or <code class="language-plaintext highlighter-rouge">shutdown</code>
commands usually work by talking to init, which is not possible at this stage.
Using <code class="language-plaintext highlighter-rouge">/proc/sysrq-trigger</code> might work depending on the target hardware, but
for some ARM-based devices, it might be much easier to use power sockets that
can be switched on and off (e.g. EnerGenie EG-PMS2, with the <code class="language-plaintext highlighter-rouge">sispmctl</code>
package and <a href="https://bugs.debian.org/929612">#929612</a> fixed).</li>
</ul>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[Flashing system images without ever touching hardware again? Not impossible!]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/groundhog-day.png" /><media:content medium="image" url="https://debamax.com/groundhog-day.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Helping with PiRogue Tool Suite</title><link href="https://debamax.com/blog/2022/06/21/helping-with-pirogue-tool-suite/" rel="alternate" type="text/html" title="Helping with PiRogue Tool Suite" /><published>2022-06-21T14:30:00+02:00</published><updated>2022-06-21T14:30:00+02:00</updated><id>https://debamax.com/blog/2022/06/21/helping-with-pirogue-tool-suite</id><content type="html" xml:base="https://debamax.com/blog/2022/06/21/helping-with-pirogue-tool-suite/"><![CDATA[<h2>PiRogue Tool Suite</h2>

<p>Here’s how the project is presented
<a href="https://pts-project.org/">on its website</a>:</p>

<blockquote>
  <p>PiRogue tool suite (PTS) is an open-source tool suite that provides a
comprehensive mobile forensic and network traffic analysis platform targeting
mobile devices both Android and iOS, internet of things devices (devices that
are connected to the user mobile apps), and in general any device using Wi-Fi
to connect to the Internet.</p>
</blockquote>

<p>The concept is to provide a tool suite that leverages a Raspberry Pi based
device, the PiRogue, to conduct a number of analyses.</p>

<p>The PiRogue operates as a network router, analyzing network traffic in real
time. It can operate in different modes, detailed under <a href="https://pts-project.org/docs/pirogue/overview/#what-a-pirogue-is">“What a PiRogue
is”</a>:</p>

<ul>
  <li>a kiosk mode for anyone who wants to know which servers a mobile device is
communicating with;</li>
  <li>an on-the-field mode;</li>
  <li>an expert mode.</li>
</ul>

<p>The PiRogue itself consists of:</p>

<ul>
  <li>a regular Pi 3 or Pi 4;</li>
  <li>a <abbr title="Hardware Attached on Top">HAT</abbr>;</li>
  <li>a case.</li>
</ul>

<p>The conception follows an open hardware approach, with KiCad files available in
the <a href="https://github.com/PiRogueToolSuite/pirogue-hat">pirogue-hat</a> repository.
The HAT makes it possible to get information on a small LCD screen, to control a
fan, and to get an <abbr title="real-time clock">RTC</abbr> (to keep track of
time when the PiRogue is shut down). For discovery and development purposes,
it’s very fine to use just a Pi 3 or Pi 4, without the HAT and the case.</p>

<h2><span class="small-caps">Debamax</span>’s initial involvement</h2>

<p>PiRogue tool suite is mainly developed and maintained by Defensive Lab Agency in
the person of <a href="https://esther.codes/">Esther Onfroy</a>. Some development sessions
are <a href="https://www.twitch.tv/u039b">streamed live</a>, and Esther got in touch to see
if it would be possible to talk about Debian packaging, looking at some
practical use cases. This seemed very interesting, and a very nice way to learn
more about this project, so Cyril jumped right in.</p>

<p>The entry point for this improvised workshop was the <a href="https://github.com/PiRogueToolSuite/deb-packages"><code class="language-plaintext highlighter-rouge">deb-packages</code>
repository</a>, which contains
the source of a few Debian packages. The resulting binary packages are published
in a Debian repository (or <abbr title="Personal Package Archive">PPA</abbr>).</p>

<p>The PiRogue OS is built using a fork of the <a href="https://github.com/RPI-Distro/pi-gen">official toolchain to create
Raspberry Pi OS images</a>, published in the <a href="https://github.com/PiRogueToolSuite/pirogue-os"><code class="language-plaintext highlighter-rouge">pirogue-os</code>
repository</a>. The modifications
are basically about configuring the package manager so that it knows about the
additional repository, and about installing the <code class="language-plaintext highlighter-rouge">pirogue-base</code> metapackage that
pulls all required <code class="language-plaintext highlighter-rouge">pirogue-*</code> packages and their dependencies.</p>

<p>The idea is to keep build time modifications as minimal as possible, which means
that users don’t need to download and deploy PiRogue OS images: it should be
possible to start from a Raspberry OS image (possibly even from a Debian image)
that’s already deployed, then configure the package manager and install a few
packages, to get an equivalent PiRogue OS image.</p>

<p>During the live session, we looked at the set of packages prepared by Esther,
mostly reviewing metadata in the <code class="language-plaintext highlighter-rouge">debian/control</code> files, checking build
instructions in <code class="language-plaintext highlighter-rouge">debian/rules</code>, and double-checking the contents of maintainer
scripts (e.g. <code class="language-plaintext highlighter-rouge">preinst</code> and <code class="language-plaintext highlighter-rouge">postinst</code>).</p>

<p>Some packages were moved from <code class="language-plaintext highlighter-rouge">Architecture: any</code> (meaning they have to be built
for each target architecture) to <code class="language-plaintext highlighter-rouge">Architecture: all</code> (meaning the exact same
package can be used across all target architectures). Using Python for most
things allows for such an optimization, which helps support building 32-bit and
64-bit images down the line (<code class="language-plaintext highlighter-rouge">armhf</code> and <code class="language-plaintext highlighter-rouge">arm64</code> respectively). Some
explanations were given about part of the <code class="language-plaintext highlighter-rouge">debhelper</code> suite,
e.g. <code class="language-plaintext highlighter-rouge">dh_installsystemd</code> which deals with systemd units, and about overrides in
<code class="language-plaintext highlighter-rouge">debian/rules</code>: those tools should have a sane default behavior, but sometimes
developers might need to add some tweaks.</p>

<p>We also alluded to tools facilitating clean builds, like <code class="language-plaintext highlighter-rouge">pbuilder</code>,
<code class="language-plaintext highlighter-rouge">cowbuilder</code>, and <code class="language-plaintext highlighter-rouge">sbuild</code>. Those are particularly helpful to check that the
expressed build dependencies are correct, and to shield the build process from
possible system configuration tweaks that might only be present on a given
developer’s system. This helps getting reproducible results when packages are
built on different systems, and it’s been considered best practice in the Debian ecosystem for many
years anyway.</p>

<p>All packages were in a very decent shape already, and this work session ended up
being an opportunity to give an overview of packaging concepts with an hands-on
approach.</p>

<h2>Follow-up work</h2>

<p>This live session really piqued Cyril’s interest, as two main improvements had
been identified:</p>

<ul>
  <li>by Esther from the get-go: NFStream;</li>
  <li>by Cyril: the <code class="language-plaintext highlighter-rouge">pip</code> calls in maintainer scripts.</li>
</ul>

<p>Helping with a project that aims at defending people’s rights seemed very
appealing, and was a nice return to some form of hacktivism, so <span class="small-caps">Debamax</span>
dedicated some days to doing all the heavy lifting for some non-trivial
packages, alongside answering any questions Esther might have about existing or
forthcoming packages, and participating in general brainstorming.</p>

<h3>Packaging NFStream</h3>

<p>As explained in the previous section, the PiRogue OS concept is:</p>

<ul>
  <li>publish a few packages published in a PPA;</li>
  <li>either add this PPA during a PiRogue OS image build, or add it to an
already-deployed Raspberry OS or Debian system;</li>
  <li>install all <code class="language-plaintext highlighter-rouge">pirogue-*</code> packages and their dependencies.</li>
</ul>

<p>That wasn’t possible initially, as <a href="https://www.nfstream.org/">NFStream</a> (a
flexible network data analysis framework) was being built during the image
build, and deployed under <code class="language-plaintext highlighter-rouge">/usr/local</code>… which invalidated the “you can deploy
PiRogue on top of your system” use case. That also meant that further upgrades
would be harder: those files weren’t tracked by the package manager, and we
would need to think about some custom system to support upgrades.</p>

<p>Additionally, NFStream requires specific tags or development branches for a few
of its build dependencies (<code class="language-plaintext highlighter-rouge">libgpg-error</code>, <code class="language-plaintext highlighter-rouge">libgcrypt</code>, <code class="language-plaintext highlighter-rouge">libpcap</code>, and <code class="language-plaintext highlighter-rouge">nDPI</code>), making
the challenge even more interesting.</p>

<p>Cyril suggested creating an extra <a href="https://github.com/PiRogueToolSuite/deb-nfstream"><code class="language-plaintext highlighter-rouge">deb-nfstream</code>
repository</a>, where all the
required components would be aggregated. A <code class="language-plaintext highlighter-rouge">bullseye</code> branch was forked from the
6.3.5 upstream tag, and the aforementioned components were declared as Git
submodules under the <code class="language-plaintext highlighter-rouge">vendor/</code> directory.</p>

<p>Working inside a development <code class="language-plaintext highlighter-rouge">chroot</code> environment, it became clear that each
build resulted in the deployment of development files (C headers and static
libraries), and that the NFStream built was using those static libraries to
generate a C extension (shared library) that can be used by the Python
interpreter. After that, those development files weren’t needed, so it seemed
feasible to ship NFStream’s Python files and its C extension, leaving all
intermediary files behind.</p>

<p>That’s how we ended up with a <a href="https://github.com/PiRogueToolSuite/deb-nfstream/blob/bullseye/debian/rules"><code class="language-plaintext highlighter-rouge">debian/rules</code>
file</a>
that performs a build of each component with the appropriate options, then
performs a <code class="language-plaintext highlighter-rouge">make install</code> step that targets a staging area (<code class="language-plaintext highlighter-rouge">vendor/staging</code>),
via the usual <code class="language-plaintext highlighter-rouge">DESTDIR</code> variable. This means we stash everything inside a
designated area of the build directory, without risking polluting system paths
(even if permission issues would stop us, as building is usually done as a
non-privileged user).
Then, the <code class="language-plaintext highlighter-rouge">NFStream</code> build system can be pointed to that staging area, which is
done by adding <code class="language-plaintext highlighter-rouge">-Ivendor/staging/usr/include</code> and
<code class="language-plaintext highlighter-rouge">-Ivendor/staging/usr/include/nDPI</code> for the <code class="language-plaintext highlighter-rouge">gcc</code> call.</p>

<p>The final touch was asking <code class="language-plaintext highlighter-rouge">dh_python3</code> not to rename the C extension. By
default, <code class="language-plaintext highlighter-rouge">engine_cc.so</code> would be renamed to embed information about the
interpreter (CPython version 3.9 on <code class="language-plaintext highlighter-rouge">bullseye</code>, our base distribution) and about
the architecture (the usual arch triplet) in its filename:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">engine_cc.cpython-39-x86_64-linux-gnu.so</code> for <code class="language-plaintext highlighter-rouge">amd64</code>;</li>
  <li><code class="language-plaintext highlighter-rouge">engine_cc.cpython-39-aarch64-linux-gnu.so</code> for <code class="language-plaintext highlighter-rouge">arm64</code>;</li>
  <li><code class="language-plaintext highlighter-rouge">engine_cc.cpython-39-arm-linux-gnueabihf.so</code> for <code class="language-plaintext highlighter-rouge">armhf</code>.</li>
</ul>

<p>This renaming is probably meant to support co-installability of packages from
various architectures on the same system. That’s not something that would make
sense for PiRogue OS, that’s why it was decided to skip the renaming altogether. Other
possibilities included shipping an <code class="language-plaintext highlighter-rouge">engine_cc.so</code> symlink pointing at the right
filename (this was tested successfully), or adjusting the <code class="language-plaintext highlighter-rouge">ffi.dlopen()</code> call
(this was not investigated).</p>

<p>The resulting source package can be built inside a clean <code class="language-plaintext highlighter-rouge">cowbuilder</code> chroot,
once for <code class="language-plaintext highlighter-rouge">armhf</code> and once for <code class="language-plaintext highlighter-rouge">arm64</code>, and the resulting <code class="language-plaintext highlighter-rouge">python3-nfstream</code>
binary packages are published in the <code class="language-plaintext highlighter-rouge">pirogue-3rd-party</code> section of the PPA.</p>

<h3>Packaging more Python components</h3>

<p>The next topic that could be vastly improved was the manual installation of
several Python packages, via <code class="language-plaintext highlighter-rouge">pip install</code> calls in maintainer scripts. It
seemed a little hazardous to rely on the contents of the <a href="https://pypi.org/">Python Package
index</a>, even more so when no versions were being specified:
the set of packages installed from PyPI could change at any time.</p>

<p>Having proper Debian packages for all dependencies would streamline both the
build process and further upgrade scenarios, meaning we would only have
to care about the contents of our PPA. And incidentally, while Cyril was working
on creating those packages, the PiRogue image build process starting to break
during release candidate preparations! That reinforced the initial “we should
try and provide proper packages for every component” feeling.</p>

<p>This explains how we ended up with a new <a href="https://github.com/PiRogueToolSuite/deb-python"><code class="language-plaintext highlighter-rouge">deb-python</code>
repository</a>, which aggregates a
few Python packages: <code class="language-plaintext highlighter-rouge">adb-shell</code>, <code class="language-plaintext highlighter-rouge">communityid</code>, <code class="language-plaintext highlighter-rouge">geoip2</code>, <code class="language-plaintext highlighter-rouge">iosbackup</code>,
<code class="language-plaintext highlighter-rouge">maxminddb</code>, <code class="language-plaintext highlighter-rouge">mvt</code>, and <code class="language-plaintext highlighter-rouge">nskeyedunarchiver</code>. Those packages were drafted using
the <code class="language-plaintext highlighter-rouge">py2dsp</code> command (found in the <code class="language-plaintext highlighter-rouge">pypi2deb</code> package). This “Python source
package to Debian source package converter” made it easy to prepare the
<code class="language-plaintext highlighter-rouge">debian/</code> directory for each module, even if it’s still based on some old
<code class="language-plaintext highlighter-rouge">debhelper</code> compatibility level (10 instead of 13).</p>

<p>A Python package required some extra care: <a href="https://frida.re/">Frida</a>. It’s a
particularly important one in the PiRogue Tool Suite ecosystem, and the upstream
repository is a little trickier as it relies on several Git submodules. Instead
of incorporating a specific tarball in the common <a href="https://github.com/PiRogueToolSuite/deb-python"><code class="language-plaintext highlighter-rouge">deb-python</code>
repository</a>, it made more sense
to have a separate <a href="https://github.com/PiRogueToolSuite/deb-frida"><code class="language-plaintext highlighter-rouge">deb-frida</code>
repository</a>, forked from the
<a href="https://github.com/frida/frida">upstream Frida repository</a>, with a
<code class="language-plaintext highlighter-rouge">debian/bullseye</code> packaging branch. Contrary to all other packages in the
PiRogue Tool Suite project, which follow best practices regarding Debian
packaging, that one requires specific settings when getting built in a clean
environment (like <code class="language-plaintext highlighter-rouge">cowbuilder</code>): it requires network access since the
appropriate embedded toolchain gets downloaded during the build process.</p>

<p>To work around an (unfortunately undocumented) issue with the NumPy package, a
dependency on a higher version was declared in the <code class="language-plaintext highlighter-rouge">pip install</code> call. The
specified version being only available in Debian <code class="language-plaintext highlighter-rouge">experimental</code>, it was decided
to fetch the packages from there, and repack them a little. It’s indeed built
for several versions of the Python interpreter, while all we care about is 3.9
for our <code class="language-plaintext highlighter-rouge">bullseye</code> base distribution. More details can be found in <a href="https://github.com/PiRogueToolSuite/ppa/commit/a1ee864c99c22a37ea3fb30688e05522495cabce">commit
a1ee864c99</a>,
where the tweaked packages were <a href="https://github.com/PiRogueToolSuite/ppa">merged into the
PPA</a>.</p>

<h2>Piecing everything together</h2>

<h3>Third party components</h3>

<p>In the interest of having as much control as possible over the set of packages
being deployed on the PiRogue, and for simplicity’s sake as well, Cyril
suggested merging the remaining third party packages into the PPA. This means
not having to maintain extra configuration for Grafana, InfluxDB, etc.</p>

<p>That’s how we ended up with the following packages in the <code class="language-plaintext highlighter-rouge">pirogue-3rd-party</code>
directory of the PPA:</p>

<table>
  <thead>
    <tr>
      <th>Filename</th>
      <th>Origin</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>chronograf_1.9.1-1_arm64.deb</td>
      <td>Merged from <a href="https://repos.influxdata.com/">https://repos.influxdata.com/</a></td>
    </tr>
    <tr>
      <td>chronograf_1.9.1-1_armhf.deb</td>
      <td>Merged from <a href="https://repos.influxdata.com/">https://repos.influxdata.com/</a></td>
    </tr>
    <tr>
      <td>frida_15.1.17~pirogue1_arm64.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-frida">deb-frida</a></td>
    </tr>
    <tr>
      <td>frida_15.1.17~pirogue1_armhf.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-frida">deb-frida</a></td>
    </tr>
    <tr>
      <td>grafana_8.1.5_arm64.deb</td>
      <td>Merged from <a href="https://grafana.com/grafana/download">https://grafana.com/grafana/download</a></td>
    </tr>
    <tr>
      <td>grafana_8.1.5_armhf.deb</td>
      <td>Merged from <a href="https://grafana.com/grafana/download">https://grafana.com/grafana/download</a></td>
    </tr>
    <tr>
      <td>influxdb_1.8.10-1_arm64.deb</td>
      <td>Merged from <a href="https://repos.influxdata.com/">https://repos.influxdata.com/</a></td>
    </tr>
    <tr>
      <td>influxdb_1.8.10-1_armhf.deb</td>
      <td>Merged from <a href="https://repos.influxdata.com/">https://repos.influxdata.com/</a></td>
    </tr>
    <tr>
      <td>python3-nfstream_6.3.5~ds2_arm64.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-nfstream">deb-nfstream</a></td>
    </tr>
    <tr>
      <td>python3-nfstream_6.3.5~ds2_armhf.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-nfstream">deb-nfstream</a></td>
    </tr>
    <tr>
      <td>python/python3-adb-shell_0.4.2-1~pirogue1_all.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-python">deb-python</a></td>
    </tr>
    <tr>
      <td>python/python3-communityid_1.4-1~pirogue1_all.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-python">deb-python</a></td>
    </tr>
    <tr>
      <td>python/python3-geoip2_4.5.0-1~pirogue1_all.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-python">deb-python</a></td>
    </tr>
    <tr>
      <td>python/python3-iosbackup_0.9.923-1~pirogue1_all.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-python">deb-python</a></td>
    </tr>
    <tr>
      <td>python/python3-maxminddb_2.2.0-1~pirogue1_arm64.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-python">deb-python</a></td>
    </tr>
    <tr>
      <td>python/python3-maxminddb_2.2.0-1~pirogue1_armhf.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-python">deb-python</a></td>
    </tr>
    <tr>
      <td>python/python3-mvt_1.5.4-1~pirogue1_all.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-python">deb-python</a></td>
    </tr>
    <tr>
      <td>python/python3-nskeyedunarchiver_1.5-1~pirogue1_all.deb</td>
      <td>Built from <a href="https://github.com/PiRogueToolSuite/deb-python">deb-python</a></td>
    </tr>
    <tr>
      <td>python/python3-numpy_1.22.1-1+pirogue1_arm64.deb</td>
      <td>Borrowed from Debian <code class="language-plaintext highlighter-rouge">experimental</code></td>
    </tr>
    <tr>
      <td>python/python3-numpy_1.22.1-1+pirogue1_armhf.deb</td>
      <td>Borrowed from Debian <code class="language-plaintext highlighter-rouge">experimental</code></td>
    </tr>
  </tbody>
</table>

<h3>PiRogue components</h3>

<p>While Cyril mainly worked on trying to get a reasonable set of extra packages to
complement what’s found in the Debian distribution, let’s look at the set of
packages proposed by PiRogue Tool Suite developers, published in the <code class="language-plaintext highlighter-rouge">pirogue</code>
directory of the PPA:</p>

<table>
  <thead>
    <tr>
      <th>Filename</th>
      <th>Purpose</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>pirogue-ap_1.0.0_all.deb</td>
      <td>Configures Wi-Fi Access Point</td>
    </tr>
    <tr>
      <td>pirogue-base_1.0.1_all.deb</td>
      <td>Depends on all other packages (metapackage)</td>
    </tr>
    <tr>
      <td>pirogue-cli_1.0.0-beta_all.deb</td>
      <td>Provisions Command Line Interface tools</td>
    </tr>
    <tr>
      <td>pirogue-dashboard_1.0.1_all.deb</td>
      <td>Provisions PiRogue dashboards</td>
    </tr>
    <tr>
      <td>pirogue-eve-collector_1.0.3_all.deb</td>
      <td>Provisions the Suricata alarm collector</td>
    </tr>
    <tr>
      <td>pirogue-flow-inspector_1.0.2_all.deb</td>
      <td>Provisions Deep Packet Inspection</td>
    </tr>
    <tr>
      <td>pirogue-hat_1.0.1_all.deb</td>
      <td>Manages hardware configuration for the HAT</td>
    </tr>
    <tr>
      <td>pirogue-maintenance_1.0.1_all.deb</td>
      <td>Updates Suricata rules</td>
    </tr>
    <tr>
      <td>pirogue-screen-st7789-240x240_1.0.2_arm64.deb</td>
      <td>Drives the LCD on the HAT</td>
    </tr>
    <tr>
      <td>pirogue-screen-st7789-240x240_1.0.2_armhf.deb</td>
      <td>Drives the LCD on the HAT</td>
    </tr>
    <tr>
      <td>pirogue-tools_1.0.2_all.deb</td>
      <td>Installs additional tools like <a href="https://frida.re/">Frida</a> or <a href="https://mvt.re/">MVT</a></td>
    </tr>
  </tbody>
</table>

<p>Most of them can be found in the <a href="https://github.com/PiRogueToolSuite/deb-packages"><code class="language-plaintext highlighter-rouge">deb-packages</code>
repository</a>, while
<code class="language-plaintext highlighter-rouge">pirogue-cli</code> lives <a href="https://github.com/PiRogueToolSuite/pirogue-cli">in its own
repository</a>.</p>

<h3>PPA overview</h3>

<p>Let’s check what the dependency graph looks like, between all aforementioned
components (leaving aside any packages that are found in the <code class="language-plaintext highlighter-rouge">bullseye</code> base
distribution):</p>

<ul>
  <li>in light green, the first two dependency levels are from the <code class="language-plaintext highlighter-rouge">pirogue</code>
directory;</li>
  <li>in light blue, dependency levels three and above are from the
<code class="language-plaintext highlighter-rouge">pirogue-3rd-party</code> directory.</li>
</ul>

<p><a href="ppa-dependencies.svg"><img src="ppa-dependencies.svg" alt="" title="Dependency graph for packages in our PPA" /></a></p>

<h2>Conclusion</h2>

<p>What started as an impromptu “let’s talk about Debian packaging for a few hours”
live session turned into a strong (even if informal) commitment to helping
PiRogue Tool Suite developers. Defending people’s rights is not a new topic for
<span class="small-caps">Debamax</span>, which worked on <a href="https://tails.boum.org/">Tails</a> previously,
and which provides some financial support to <a href="https://exodus-privacy.eu.org/">Exodus
Privacy</a>. It’s a pleasure to share some
technical expertise with such an important project, and to have helped reach the
1.0.0 milestone!</p>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[When mobile forensic and network analysis meet Raspberry Pi devices.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/pirogue-logo.png" /><media:content medium="image" url="https://debamax.com/pirogue-logo.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">systemd: RequiredBy versus WantedBy</title><link href="https://debamax.com/blog/2020/06/14/systemd-requiredby-versus-wantedby/" rel="alternate" type="text/html" title="systemd: RequiredBy versus WantedBy" /><published>2020-06-14T17:00:00+02:00</published><updated>2020-06-14T17:00:00+02:00</updated><id>https://debamax.com/blog/2020/06/14/systemd-requiredby-versus-wantedby</id><content type="html" xml:base="https://debamax.com/blog/2020/06/14/systemd-requiredby-versus-wantedby/"><![CDATA[<h2>Introduction</h2>

<p><span class="small-caps">Debamax</span> helps several customers build and maintain system
    images based on Debian: those are deployed on target devices
    (regular servers, workstations, embedded devices), and then
    upgraded to new versions of the custom operating system using
    secure channels, and validated upgraded paths.</p>

<p>Such systems usually require some system-level integration,
    making sure all required packages work well when installed
    together, along with some configuration daemon to ensure those
    sytems can be tweaked as required. Such configuration daemons can
    be controlled remotely over a so-called “business application”
    that takes care of establishing a link to some remote backoffice,
    or exposed on a local console for operators to configure the
    system in a controlled manner (as opposed to exposing a full
    <code>root</code> shell).</p>

<p>This article focuses on the implementation details of such a
    configuration daemon, which is split into two parts: the actual
    daemon is exposing a REST interface, which is used by the business
    application to trigger configuration updates, and another part
    which is handling upgrades. To abstract this article from internal
    names, those two parts are called <code>debamax-daemon</code>
    and <code>debamax-upgrade</code> respectively. The following graph
    highlights the possible interactions.</p>

<div class="container-with-margins">
  <table class="images">
    <tr>
      <th>Interactions between daemon and upgrade components</th>
    </tr>
    <tr>
      <td>
        <img src="unit-interactions.png" alt="Interactions between units" width="500" height="115" />
      </td>
    </tr>
  </table>
</div>

<p>Each part is managed by a <a href="https://www.freedesktop.org/software/systemd/man/systemd.service.html">systemd service unit</a>:</p>

<ul>
    <li><code>debamax-daemon</code> is spawning a daemon which manages
      REST requests, so uses <code>Type=simple</code> in
      the <code>[Service]</code> section.</li>
    <li><code>debamax-upgrade</code> is spawning a shell script which
      determines whether there's something to do on the upgrade side,
      so uses <code>Type=oneshot</code> in the <code>[Service]</code>
      section. It is started once at boot-up, triggering an upgrade if
      required (maybe resuming some interrupted download), and it can
      be started again at any later point in time, when the
      configuration daemon requests an upgrade (blue, left-to-right
      arrow in the graph above). In turn, during an upgrade,
      the <code>debamax-daemon</code> package might get upgraded, in
      which case its service gets restarted (magenta, right-to-left
      arrow in the graph above), as is usual in Debian
      environments.</li>
  </ul>

<p>Through some heavily simplified code, this article highlights the
    need for accurate metadata in those two systemd units,
    namely <code>RequiredBy=</code> versus <code>WantedBy=</code> in
    the <code>[Install]</code> section.</p>

<p>The playground is a minimal Debian 10 (Buster) virtual
    machine. To simplify tests, no actual <code>debamax-*</code> Debian
    packages are involved (even though that’s the case for customer
    systems), and directories for local administrators are used
    instead of system-wide directories (which are for proper
    packages). This means:</p>

<ul>
    <li>Using <code>/etc/systemd/system/&lt;unit&gt;.service</code>
      instead of <code>/lib/systemd/system/&lt;unit&gt;.service</code>
      for systemd service unit definitions.</li>
    <li>Using <code>/usr/local/sbin/&lt;daemon-or-script&gt;</code>
      instead of <code>/usr/sbin/&lt;daemon-or-script&gt;</code> for
      the configuration daemon and upgrade script.</li>
  </ul>

<p><b>Important:</b> In the whole article, the following convention will be
    used: when quoting commands being run, a line with 4 hyphens
    (<code>----</code>) will separate the output of the command from the
    system logs (as seen through <code>journalctl</code>). Letting
    a <code>journalctl -f</code> run in the background or in a different
    console makes it easy to follow what’s happening.</p>

<h2>Looking at the upgrade script</h2>

<h3>Naive attempt</h3>

<p>When starting from scratch, rather than patching an existing
    systemd unit, it might seem appealing to look around in
    all <code>/lib/systemd/system/*.service</code>, find something
    that resembles what we’d like to achieve, and adapt it. A minimal
    systemd service unit
    (<code>/etc/systemd/system/debamax-upgrade.service</code>) might
    look like this:
  </p>

<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[Service]</span>
<span class="py">Type</span><span class="p">=</span><span class="s">oneshot</span>
<span class="py">ExecStart</span><span class="p">=</span><span class="s">/usr/local/sbin/debamax-upgrade</span>

<span class="nn">[Install]</span>
<span class="py">RequiredBy</span><span class="p">=</span><span class="s">multi-user.target</span>
</code></pre></div></div>

<p>and a minimal upgrade script
    (<code>/usr/local/sbin/debamax-upgrade</code>, don’t forget
    the <code>+x</code> flag) might be:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="nb">echo</span> <span class="s2">"starting debamax-upgrade"</span>
<span class="nb">echo</span> <span class="s2">"stopping debamax-upgrade"</span>
</code></pre></div></div>

<p>Let’s enable it <i>now</i>, meaning <code>enable</code>
    and <code>start</code> in a single command:</p>

<pre>
root@demo:~# systemctl enable --now debamax-upgrade
Created symlink /etc/systemd/system/multi-user.target.requires/debamax-upgrade.service \\
  → /etc/systemd/system/debamax-upgrade.service.
----
Jun 13 17:01:16 demo systemd[1]: Reloading.
Jun 13 17:01:16 demo systemd[1]: Starting debamax-upgrade.service...
Jun 13 17:01:16 demo debamax-upgrade[881]: starting debamax-upgrade
Jun 13 17:01:16 demo debamax-upgrade[881]: stopping debamax-upgrade
Jun 13 17:01:16 demo systemd[1]: debamax-upgrade.service: Succeeded.
Jun 13 17:01:16 demo systemd[1]: Started debamax-upgrade.service.
</pre>

<p>The <code>RequiredBy=multi-user.target</code> is translated to a
    symlink pointing to the appropriate systemd service unit, systemd
    reloads its configuration on its own (no need for a
    separate <code>systemctl daemon-reload</code>), and the unit is
    started.</p>

<p>All good? Not quite!</p>

<h3>Actual testing</h3>

<p>Now we have a <code>debamax-upgrade</code> script that was started once,
    and that will start at boot-up, which is rather good. But what if the
    daemon needs to request an upgrade? Restarting the
    <code>debamax-upgrade.service</code> unit seems the right thing to
    do:</p>

<pre>
root@demo:~# systemctl restart debamax-upgrade
----
<i>Jun 13 17:04:05 demo systemd[1]: Stopped target Graphical Interface.</i>
Jun 13 17:04:05 demo systemd[1]: Stopping Graphical Interface.
<i>Jun 13 17:04:05 demo systemd[1]: Stopped target Multi-User System.</i>
Jun 13 17:04:05 demo systemd[1]: Stopping Multi-User System.
Jun 13 17:04:05 demo systemd[1]: Starting debamax-upgrade.service...
Jun 13 17:04:05 demo debamax-upgrade[892]: starting debamax-upgrade
Jun 13 17:04:05 demo debamax-upgrade[892]: stopping debamax-upgrade
Jun 13 17:04:05 demo systemd[1]: debamax-upgrade.service: Succeeded.
Jun 13 17:04:05 demo systemd[1]: Started debamax-upgrade.service.
<i>Jun 13 17:04:05 demo systemd[1]: Reached target Multi-User System.</i>
<i>Jun 13 17:04:05 demo systemd[1]: Reached target Graphical Interface.</i>
Jun 13 17:04:05 demo systemd[1]: Starting Update UTMP about System Runlevel Changes...
Jun 13 17:04:05 demo systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
<i>Jun 13 17:04:05 demo systemd[1]: Started Update UTMP about System Runlevel Changes.</i>
</pre>

<p>Wait a minute! What’s with those Graphical and Multi-User targets? Even
    if there are no display managers, <code>graphical.target</code> is the
    default target, which depends on <code>multi-user.target</code>,
    as can be seen in
    the <a href="https://www.freedesktop.org/software/systemd/man/bootup.html">bootup</a>
    manpage. But why are those two targets getting exited from?</p>

<p>It was requested to <b>restart</b> a service unit
    that’s <b>RequiredBy</b> the <code>multi-user.target</code>, which means
    their requirements are no longer met. Once the service unit starts again, those
    targets can be re-entered, and some side-effects can be seen (UTMP, System
    Runlevel Changes).</p>

<p>Since this is about a shell script that runs once and exits, as defined
    through <code>Type=oneshot</code>, one might wonder what happens if only
    a new <code>start</code> would be requested:</p>

<pre>
root@demo:~# systemctl start debamax-upgrade
----
Jun 13 17:05:09 demo systemd[1]: Starting debamax-upgrade.service...
Jun 13 17:05:09 demo debamax-upgrade[899]: starting debamax-upgrade
Jun 13 17:05:09 demo debamax-upgrade[899]: stopping debamax-upgrade
Jun 13 17:05:09 demo systemd[1]: debamax-upgrade.service: Succeeded.
Jun 13 17:05:09 demo systemd[1]: Started debamax-upgrade.service.
</pre>

<p>That looks way better, and closer to what was intended. To be
    entirely honest, an early “fix” was to just spawn
    <code>systemctl start debamax-upgrade</code> from
    the <code>debamax-daemon</code> code when requesting an upgrade,
    but that looked fishy. Can we fix the <code>restart</code> case
    that looked awkward?</p>

<h3>Towards a real fix</h3>

<p>Let’s switch the systemd service unit from <code>RequiredBy=</code>
    to <code>WantedBy=</code>. Beware, while doing so, it’s best to first
    disable the current unit, edit it, and then enable it again, so that the
    proper symlinks can be removed and created:</p>

<pre>
root@demo:~# systemctl disable --now debamax-upgrade
Removed /etc/systemd/system/multi-user.target.requires/debamax-upgrade.service.

root@demo:~# sed 's/RequiredBy=/WantedBy=/' -i /etc/systemd/system/debamax-upgrade.service

root@demo:~# systemctl enable --now debamax-upgrade
Created symlink /etc/systemd/system/multi-user.target.wants/debamax-upgrade.service \\
  → /etc/systemd/system/debamax-upgrade.service.
----
Jun 13 17:06:12 demo systemd[1]: Reloading.
…
Jun 13 17:06:42 demo systemd[1]: Reloading.
Jun 13 17:06:43 demo systemd[1]: Starting debamax-upgrade.service...
Jun 13 17:06:43 demo debamax-upgrade[936]: starting debamax-upgrade
Jun 13 17:06:43 demo debamax-upgrade[936]: stopping debamax-upgrade
Jun 13 17:06:43 demo systemd[1]: debamax-upgrade.service: Succeeded.
Jun 13 17:06:43 demo systemd[1]: Started debamax-upgrade.service.
</pre>

<p>For the avoidance of doubt, here’s the new version of the systemd
    service unit for the upgrade component
    (<code>/etc/systemd/system/debamax-upgrade.service</code>):</p>

<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[Service]</span>
<span class="py">Type</span><span class="p">=</span><span class="s">oneshot</span>
<span class="py">ExecStart</span><span class="p">=</span><span class="s">/usr/local/sbin/debamax-upgrade</span>

<span class="nn">[Install]</span>
<span class="py">WantedBy</span><span class="p">=</span><span class="s">multi-user.target</span>
</code></pre></div></div>

<p>Let’s compare starting:</p>

<pre>
root@demo:~# systemctl start debamax-upgrade
----
Jun 13 17:07:48 demo systemd[1]: Starting debamax-upgrade.service...
Jun 13 17:07:48 demo debamax-upgrade[941]: starting debamax-upgrade
Jun 13 17:07:48 demo debamax-upgrade[941]: stopping debamax-upgrade
Jun 13 17:07:48 demo systemd[1]: debamax-upgrade.service: Succeeded.
Jun 13 17:07:48 demo systemd[1]: Started debamax-upgrade.service.
</pre>

<p>with restarting:</p>

<pre>
root@demo:~# systemctl restart debamax-upgrade
----
Jun 13 17:08:16 demo systemd[1]: Starting debamax-upgrade.service...
Jun 13 17:08:16 demo debamax-upgrade[944]: starting debamax-upgrade
Jun 13 17:08:16 demo debamax-upgrade[944]: stopping debamax-upgrade
Jun 13 17:08:16 demo systemd[1]: debamax-upgrade.service: Succeeded.
Jun 13 17:08:16 demo systemd[1]: Started debamax-upgrade.service.
</pre>

<p>OK, this is way better!</p>

<h2>Looking at the daemon</h2>

<h3>More context</h3>

<p>Let’s backpedal a bit: it was confessed earlier that an early “fix” was
    just to switch from <code>restart</code> to <code>start</code> for the
    <code>debamax-upgrade.service</code> unit. It was noted in a ticket “for
    later” that some strange interactions across targets would otherwise be
    happening, but an easy fix was available. Why was it so important to get
    back to this topic to find a proper fix?</p>

<p>The problem was spotted again in a different area… This custom operating
    system features 3 critical components:</p>

<ul>
    <li>the configuration daemon (including upgrade management);</li>
    <li>a “business application” which is provided by the customer, talks to
      remote servers, talks REST with the configuration daemon to deploy
      configuration updates, and talks with other applications;</li>
    <li>a “support application” which is also provided by the customer, which
      has an important but less central role than the “business application”,
      and reports back to the “business application”.</li>
  </ul>

<p>The internal names for the business and support applications have been
    replaced with <code>demo-service-a</code> and <code>demo-service-b</code>
    in this article.</p>

<p>To free up some resources during upgrades, and to avoid possible ups and
    downs as those applications get restarted during upgrades, it was decided
    to just shut them down prior to upgrades. A successful upgrade would
    trigger a reboot and let the usual boot-up sequence start everyone again,
    while failed upgrades would restart services manually.</p>

<h3>Naive attempt</h3>

<p>Let’s see what minimal systemd unit configuration could look like for the
    configuration daemon and for the two demo services:</p>

<ul>
    <li><code>/etc/systemd/system/debamax-daemon.service</code> (note
      <code>RequiredBy=</code> is still used at this point):

<figure class="highlight"><pre><code class="language-ini" data-lang="ini"><span class="nn">[Service]</span>
<span class="py">Type</span><span class="p">=</span><span class="s">simple</span>
<span class="py">ExecStart</span><span class="p">=</span><span class="s">/usr/local/sbin/debamax-daemon</span>

<span class="nn">[Install]</span>
<span class="py">RequiredBy</span><span class="p">=</span><span class="s">multi-user.target</span></code></pre></figure>

    </li>
    <li><code>/etc/systemd/system/demo-service-a.service</code>:

<figure class="highlight"><pre><code class="language-ini" data-lang="ini"><span class="nn">[Service]</span>
<span class="py">Type</span><span class="p">=</span><span class="s">simple</span>
<span class="py">ExecStart</span><span class="p">=</span><span class="s">/usr/local/sbin/demo-service-a</span>

<span class="nn">[Install]</span>
<span class="py">WantedBy</span><span class="p">=</span><span class="s">multi-user.target</span></code></pre></figure>

    </li>
    <li><code>/etc/systemd/system/demo-service-b.service</code>:

<figure class="highlight"><pre><code class="language-ini" data-lang="ini"><span class="nn">[Service]</span>
<span class="py">Type</span><span class="p">=</span><span class="s">simple</span>
<span class="py">ExecStart</span><span class="p">=</span><span class="s">/usr/local/sbin/demo-service-b</span>

<span class="nn">[Install]</span>
<span class="py">WantedBy</span><span class="p">=</span><span class="s">multi-user.target</span></code></pre></figure>

    </li>
  </ul>

<p>Let’s keep all three daemons to something very minimal, waiting a full
    day for something to happen, ensuring they keep running for a while. The
    aim is <b>simulating</b> long-running daemons (<code>Type=simple</code>)
    with trivial code, instead of just having a single shell script exit early
    (which would usually be marked as <code>Type=oneshot</code>):</p>

<ul>
    <li><code>/usr/local/sbin/debamax-daemon</code>:

<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c">#!/bin/sh</span>
<span class="nb">trap</span> <span class="s1">'echo stopping debamax-daemon'</span> TERM
<span class="nb">echo</span> <span class="s2">"starting debamax-daemon"</span>
<span class="nb">sleep </span>24h <span class="o">||</span> <span class="nb">true</span></code></pre></figure>

    </li>
    <li><code>/usr/local/sbin/demo-service-a</code>:

<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c">#!/bin/sh</span>
<span class="nb">trap</span> <span class="s1">'echo stopping service A'</span> TERM
<span class="nb">echo</span> <span class="s2">"starting service A"</span>
<span class="nb">sleep </span>24h <span class="o">||</span> <span class="nb">true</span></code></pre></figure>

    </li>
    <li><code>/usr/local/sbin/demo-service-b</code>:

<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c">#!/bin/sh</span>
<span class="nb">trap</span> <span class="s1">'echo stopping service B'</span> TERM
<span class="nb">echo</span> <span class="s2">"starting service B"</span>
<span class="nb">sleep </span>24h <span class="o">||</span> <span class="nb">true</span></code></pre></figure>

    </li>
  </ul>

<p>Now, let’s check a new version of the upgrade script, that implements the
    policy described above: stop demo services first, then trigger an
    upgrade. To simply things further, the upgrade is simulated as well. What
    would normally happen when a package containing a daemon is upgraded would
    be putting new files in place, and restarting the daemon. Here, the
    upgrade simulation is only about restarting
    the <code>debamax-daemon</code> service unit.</p>

<ul>
    <li><code>/usr/local/sbin/debamax-upgrade</code>:

<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c">#!/bin/sh</span>
<span class="nb">echo</span> <span class="s2">"starting debamax-upgrade"</span>

<span class="nb">echo</span> <span class="s2">"1. stopping some services during the upgrade"</span>
systemctl stop demo-service-a
systemctl stop demo-service-b
<span class="nb">sleep </span>1

<span class="nb">echo</span> <span class="s2">"2. simulating the upgrade: new package gets installed, daemon gets restarted"</span>
systemctl restart debamax-daemon
<span class="nb">sleep </span>1

<span class="nb">echo</span> <span class="s2">"stopping debamax-upgrade"</span></code></pre></figure>

    </li>
  </ul>

<p><b>Note:</b> At this stage, both versions of
    the <code>debamax-upgrade.service</code> unit would give the same
    results. The <code>debamax-upgrade</code> script could even be started
    manually from the shell (without involving <code>systemctl</code> at
    all). For simplicity’s sake, the <code>systemctl start debamax-upgrade</code>
    call, that works with both versions, was chosen.</p>

<h3>Actual testing</h3>

<p>What happens when we start the upgrade process?</p>

<pre>
root@demo:~# systemctl start debamax-upgrade
----
Jun 13 17:28:47 demo systemd[1]: Starting debamax-upgrade.service...
Jun 13 17:28:47 demo debamax-upgrade[1272]: starting debamax-upgrade
Jun 13 17:28:47 demo debamax-upgrade[1272]: 1. stopping some services during the upgrade
Jun 13 17:28:47 demo demo-service-a[1263]: Terminated
Jun 13 17:28:47 demo demo-service-a[1263]: stopping service A
Jun 13 17:28:47 demo systemd[1]: Stopping demo-service-a.service...
Jun 13 17:28:47 demo systemd[1]: demo-service-a.service: Succeeded.
<i>Jun 13 17:28:47 demo systemd[1]: Stopped demo-service-a.service.</i>
Jun 13 17:28:47 demo demo-service-b[1262]: Terminated
Jun 13 17:28:47 demo demo-service-b[1262]: stopping service B
Jun 13 17:28:47 demo systemd[1]: Stopping demo-service-b.service...
Jun 13 17:28:47 demo systemd[1]: demo-service-b.service: Succeeded.
<i>Jun 13 17:28:47 demo systemd[1]: Stopped demo-service-b.service.</i>
Jun 13 17:28:48 demo debamax-upgrade[1272]: 2. simulating the upgrade: new package gets installed, daemon gets restarted
<i>Jun 13 17:28:48 demo systemd[1]: Stopped target Graphical Interface.</i>
Jun 13 17:28:48 demo systemd[1]: Stopping Graphical Interface.
<i>Jun 13 17:28:48 demo systemd[1]: Stopped target Multi-User System.</i>
Jun 13 17:28:48 demo systemd[1]: Stopping Multi-User System.
<i>Jun 13 17:28:48 demo systemd[1]: Started demo-service-b.service.</i>
Jun 13 17:28:48 demo debamax-daemon[1266]: Terminated
Jun 13 17:28:48 demo debamax-daemon[1266]: stopping debamax-daemon
Jun 13 17:28:48 demo systemd[1]: Stopping debamax-daemon.service...
<i>Jun 13 17:28:48 demo systemd[1]: Started demo-service-a.service.</i>
Jun 13 17:28:48 demo demo-service-b[1277]: starting service B
Jun 13 17:28:48 demo systemd[1]: debamax-daemon.service: Succeeded.
Jun 13 17:28:48 demo systemd[1]: Stopped debamax-daemon.service.
Jun 13 17:28:48 demo demo-service-a[1278]: starting service A
Jun 13 17:28:48 demo systemd[1]: Started debamax-daemon.service.
Jun 13 17:28:48 demo debamax-daemon[1280]: starting debamax-daemon
Jun 13 17:28:49 demo debamax-upgrade[1272]: stopping debamax-upgrade
Jun 13 17:28:49 demo systemd[1]: debamax-upgrade.service: Succeeded.
Jun 13 17:28:49 demo systemd[1]: Started debamax-upgrade.service.
<i>Jun 13 17:28:49 demo systemd[1]: Reached target Multi-User System.</i>
<i>Jun 13 17:28:49 demo systemd[1]: Reached target Graphical Interface.</i>
Jun 13 17:28:49 demo systemd[1]: Starting Update UTMP about System Runlevel Changes...
Jun 13 17:28:49 demo systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Jun 13 17:28:49 demo systemd[1]: Started Update UTMP about System Runlevel Changes.
</pre>

<p>Basically, both demo services are stopped as expected in the first
    place, but when the <code>debamax-daemon</code> unit is restarted, the
    same target dance (as seen in the previous section) happens. As a side
    effect of exiting and re-entering targets, services that were purposefully
    stopped are started again!</p>

<p>Spotting this while testing the upgrade process was the final incentive
    to move from the early “let’s use <code>start</code> instead
    of <code>restart</code>” workaround mentioned in the previous section… to
    a real fix, and that’s the point where the differences
    between <code>RequiredBy=</code> and <code>WantedBy=</code> were
    analyzed.</p>

<h3>Fixing metadata</h3>

<p>Let’s see what happens with proper metadata, remembering to first disable
    the service unit (to get the “wrong” symlink removed), before switching
    from <code>RequiredBy=</code> to <code>WantedBy=</code>, and enabling the
    service unit again:</p>

<pre>
root@demo:~# systemctl disable --now debamax-daemon
Removed /etc/systemd/system/multi-user.target.requires/debamax-daemon.service.

root@demo:~# sed 's/RequiredBy=/WantedBy=/' -i /etc/systemd/system/debamax-daemon.service

root@demo:~# systemctl enable --now debamax-daemon
Created symlink /etc/systemd/system/multi-user.target.wants/debamax-daemon.service \\
  → /etc/systemd/system/debamax-daemon.service.
----
Jun 13 17:30:59 demo systemd[1]: Reloading.
Jun 13 17:30:59 demo debamax-daemon[1280]: Terminated
Jun 13 17:30:59 demo debamax-daemon[1280]: stopping debamax-daemon
Jun 13 17:30:59 demo systemd[1]: Stopping debamax-daemon.service...
Jun 13 17:30:59 demo systemd[1]: debamax-daemon.service: Succeeded.
Jun 13 17:30:59 demo systemd[1]: Stopped debamax-daemon.service.
…
Jun 13 17:32:02 demo systemd[1]: Reloading.
Jun 13 17:32:02 demo systemd[1]: Started debamax-daemon.service.
Jun 13 17:32:02 demo debamax-daemon[1317]: starting debamax-daemon
</pre>

<p>For the avoidance of doubt, here’s the new version of the systemd
    service unit for the daemon:
    (<code>/etc/systemd/system/debamax-daemon.service</code>):</p>

<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[Service]</span>
<span class="py">Type</span><span class="p">=</span><span class="s">simple</span>
<span class="py">ExecStart</span><span class="p">=</span><span class="s">/usr/local/sbin/debamax-daemon</span>

<span class="nn">[Install]</span>
<span class="py">WantedBy</span><span class="p">=</span><span class="s">multi-user.target</span>
</code></pre></div></div>

<p>Let’s run the upgrade scenario again:</p>

<pre>
root@demo:~# systemctl start debamax-upgrade
----
Jun 13 17:32:37 demo systemd[1]: Starting debamax-upgrade.service...
Jun 13 17:32:37 demo debamax-upgrade[1321]: starting debamax-upgrade
Jun 13 17:32:37 demo debamax-upgrade[1321]: 1. stopping some services during the upgrade
Jun 13 17:32:37 demo demo-service-a[1278]: Terminated
Jun 13 17:32:37 demo demo-service-a[1278]: stopping service A
Jun 13 17:32:37 demo systemd[1]: Stopping demo-service-a.service...
Jun 13 17:32:37 demo systemd[1]: demo-service-a.service: Succeeded.
Jun 13 17:32:37 demo systemd[1]: Stopped demo-service-a.service.
Jun 13 17:32:37 demo demo-service-b[1277]: Terminated
Jun 13 17:32:37 demo demo-service-b[1277]: stopping service B
Jun 13 17:32:37 demo systemd[1]: Stopping demo-service-b.service...
Jun 13 17:32:37 demo systemd[1]: demo-service-b.service: Succeeded.
Jun 13 17:32:37 demo systemd[1]: Stopped demo-service-b.service.
Jun 13 17:32:38 demo debamax-upgrade[1321]: 2. simulating the upgrade: new package gets installed, daemon gets restarted
Jun 13 17:32:38 demo debamax-daemon[1317]: Terminated
Jun 13 17:32:38 demo debamax-daemon[1317]: stopping debamax-daemon
Jun 13 17:32:38 demo systemd[1]: Stopping debamax-daemon.service...
Jun 13 17:32:38 demo systemd[1]: debamax-daemon.service: Succeeded.
Jun 13 17:32:38 demo systemd[1]: Stopped debamax-daemon.service.
Jun 13 17:32:38 demo systemd[1]: Started debamax-daemon.service.
Jun 13 17:32:38 demo debamax-daemon[1326]: starting debamax-daemon
Jun 13 17:32:39 demo debamax-upgrade[1321]: stopping debamax-upgrade
Jun 13 17:32:39 demo systemd[1]: debamax-upgrade.service: Succeeded.
Jun 13 17:32:39 demo systemd[1]: Started debamax-upgrade.service.
</pre>

<p>This time, demo services are stopped as previously, but there’s no more
    target dance, which means that demo services are not started
    again. <code>\o/</code></p>

<h2>Conclusion</h2>

<p>Our main takeaway would be: be extra careful before thinking about
    using the <code>RequiredBy=</code> keyword!</p>

<p>Of course, when in doubt, checking the documentation
    (<a href="https://www.freedesktop.org/software/systemd/man/systemd.unit.html">systemd.unit</a>)
    might have saved some troubles, and looking at
    the <code>RequiredBy=</code> and <code>WantedBy=</code> sections, plus
    their redirections to <code>Requires=</code> and <code>Wants=</code> would
    likely have yielded the same outcome, but without being entirely clear about
    the differences between both approaches:</p>

<blockquote>
  <p>Often, it is a better choice to use Wants= instead of Requires= in
  order to achieve a system that is more robust when dealing with
  failing services.</p>
</blockquote>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[How using the wrong keyword in systemd metadata can lead to unexpected results…]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/systemd.png" /><media:content medium="image" url="https://debamax.com/systemd.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Installing Jitsi behind a reverse proxy</title><link href="https://debamax.com/blog/2020/03/18/installing-jitsi-behind-a-reverse-proxy/" rel="alternate" type="text/html" title="Installing Jitsi behind a reverse proxy" /><published>2020-03-18T10:15:00+01:00</published><updated>2025-05-30T04:00:00+02:00</updated><id>https://debamax.com/blog/2020/03/18/installing-jitsi-behind-a-reverse-proxy</id><content type="html" xml:base="https://debamax.com/blog/2020/03/18/installing-jitsi-behind-a-reverse-proxy/"><![CDATA[<div class="box has-text-white has-background-danger">
  <p><b>Update (May 2025):</b> Since the initial deployment of a local instance
    (advertised as a beta, and basically running a stock configuration), Jitsi
    upstream has seen lots of development and <span class="small-caps">Debamax</span> hasn’t been able
    to spend enough time to keep the service running optimally. The instance has
    been shut down accordingly.
  </p>
</div>

<h2>Introduction</h2>

<p>Videoconferencing with the
    official <a href="https://meet.jit.si/">meet.jit.si</a>
    instance has always been a pleasure, but it seemed a good idea to
    research how to install a <a href="https://jitsi.org">Jitsi</a>
    instance locally, that could be used by customers, by members of
    the local Linux Users Group
    (<a href="http://coagul.org">COAGUL</a>), or by anyone else.</p>

<div class="box has-background-warning">
  <p><b>Update (April 2020):</b> Since the first publication of this
    article, the <a href="#jitsi-configuration">“Jitsi
    configuration”</a> section has been updated to reflect changes
    upstream. A <a href="#more-about-stun-servers">“More about STUN
    servers</a> section has been added as well.
  </p>
</div>

<h2>Networking vs. virtualization host</h2>

<p>One host was already set up as a virtualization environment,
    featuring <a href="https://libvirt.org/">libvirt</a>, managing LXC
    containers and QEMU/KVM virtual machines. In this article, we
    focus on IPv4 networking. Basically, the TCP/80 and TCP/443 TCP ports are
    exposed on the public IP, and NAT’d to one particular container,
    which acts as a reverse proxy. The running Apache server defines
    as many VirtualHosts as there are services, and acts as a reverse
    proxy for the appropriate LXC container or QEMU/KVM virtual
    machine.</p>

<p>Schematically, here’s what happens:</p>

<ul>
    <li>Both TCP/80 and TCP/443 ports on <code>163.172.19.80</code> (the
      public IP) are NAT’d to the same ports
      on <code>192.168.122.100</code>, the reverse proxy’s IP address
      on the default libvirt bridge
      (<code>192.168.122.0/24</code>).</li>
    <li>Depending on the server name in the HTTP request,
      the <code>apache2</code> service running in that container will proxy
      requests to the appropriate service in another container or
      virtual machine. This could be something like
      <code>http://192.168.122.110:8080</code> for a Jenkins service
      or something like <code>https://192.168.122.120/</code> for a Jitsi
      service.</li>
    <li>As a general rule, the <code>apache2</code> configuration only
      returns an HTTP redirection pointing at the same service over HTTPS,
      except for the <code>.well-known/acme-challenge</code> path, to make the
      certificate dance for
      <a href="https://letsencrypt.org/">Let’s Encrypt</a> possible.</li>
  </ul>

<p>What does that mean for the Jitsi installation? Well, Jitsi
    expects those ports to be available:</p>

<ul>
    <li>TCP/443</li>
    <li>TCP/4443</li>
    <li>UDP/10000</li>
  </ul>

<p>For this specific host, TCP/4443 and UDP/10000 were available,
    and have been NAT’d as well to the Jitsi virtual machine
    directly. Given the existing services, the same couldn’t be done
    for the TCP/443 port, which explains the need for the following
    section.</p>

<p><br /></p>

<div class="container-with-margins">
  <table class="images">
    <tr>
      <th>NAT and reverse proxy for Jitsi (click for full view)</th>
    </tr>
    <tr>
      <td>
        <a href="jitsi-nat.png">
          <img src="jitsi-nat.png" alt="NAT and reverse proxy for Jitsi" width="561" height="224" />
        </a>
      </td>
    </tr>
  </table>
</div>

<p><i>Note:</i> A summary of the host’s <code>iptables</code> configuration
    is available in the annex at the bottom of this article.</p>

<h2>Apache as a reverse proxy</h2>

<p> A new VirtualHost was defined on the <code>apache2</code>
    service running as reverse proxy. The important parts are quoted
    below:</p>

<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;VirtualHost</span> <span class="err">*:80</span><span class="nt">&gt;</span>
    ServerName jitsi.debamax.com
    RedirectMatch permanent ^(?!/\.well-known/acme-challenge/).* https://jitsi.debamax.com/
<span class="nt">&lt;/VirtualHost&gt;</span>

<span class="nt">&lt;VirtualHost</span> <span class="err">*:443</span><span class="nt">&gt;</span>
    SSLProxyEngine on
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off

    ProxyPass        / https://192.168.122.120/
    ProxyPassReverse / https://192.168.122.120/
<span class="nt">&lt;/VirtualHost&gt;</span>
</code></pre></div></div>

<p>The redirections set up on the TCP/80 port were already mentioned
    in the previous section, so let’s concentrate on the TCP/443 port
    part.</p>

<p>The <code>ProxyPass</code> and <code>ProxyPassReverse</code>
    directives act on <code>/</code>, meaning every path will be
    proxied to the Jitsi virtual machine. If one wasn’t using
    VirtualHost directives to distinguish between services, one could
    be dedicating some specific paths (“subdirectories”) to Jitsi, and
    proxying only those to the Jitsi instance. But let’s concentrate
    on the simpler “the whole VirtualHost is proxied” case.</p>

<p>The first <code>SSLProxyEngine on</code> directive is needed
    for <code>apache2</code> to be happy with proxying requests to a
    server using HTTPS, instead of plain HTTP.</p>

<p>All other <code>SSLProxy*</code> directives aren’t too nice as
    they disable all checks! Why do that, then? The answer is that
    Jitsi’s default installation is setting up an NGINX
    server with HTTP-to-HTTPS redirections, and it seemed easier to
    directly forward requests to the HTTPS port, disabling all checks
    since that NGINX server was installed with a self-signed
    certificate. One could deploy a suitable certificate there
    instead and enable the checks again, instead of using this
    “StackOverflow-style heavy hammer” (some directives might not even
    be needed).</p>

<h2><a name="jitsi-configuration">Jitsi configuration</a></h2>

<p>Jitsi itself was installed on a QEMU/KVM virtual machine, running
    a basic Debian 10 (buster) system, initially provisioned with 2
    CPUs, 4 GB RAM, 3 GB virtual disk. Its IP address
    is <code>192.168.122.120</code>, which is what was configured as
    the target of the <code>ProxyPass*</code> directives in the
    previous section.</p>

<p>The installation was done using
    the <a href="https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md">quick-install.md</a>
    documentation, entering <code>jitsi.debamax.com</code> as the
    FQDN, and opting for a self-signed certificate (letting the
    reverse proxy in charge of the Let’s Encrypt certificate dance,
    like it does for all VirtualHosts).</p>

<div class="box has-background-warning">
  <p><b>Update (April 2020):</b> Since late March 2020, upstream
    switched from videobridge to videobridge2. Another important
    change is that the <code>jitsi-meet-turnserver</code> package is
    pulled through <code>jitsi-meet</code>’s <code>Recommends</code>,
    as can be seen in APT metadata (wrapped for readability):</p>

  <pre>
# apt-cache show jitsi-meet=1.0.4335-1
Depends:
 jitsi-videobridge2 (= 2.1-157-g389b69ff-1),
 jicofo (= 1.0-539-1),
 jitsi-meet-web (= 1.0.3928-1),
 jitsi-meet-web-config (= 1.0.3928-1),
 jitsi-meet-prosody (= 1.0.3928-1)
Recommends:
 jitsi-meet-turnserver (= 1.0.3928-1) | apache2
</pre>

  <p>
    <a href="https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_NAT">
      <abbr title="Traversal Using Relays around NAT">TURN</abbr></a>
    servers make it possible for clients to exchange streams in a peer
    to peer fashion when there are only two of them, by finding a way to
    traverse NATs. In the setup being documented here, the easiest is to
    <b>not install</b> the <code>jitsi-meet-turnserver</code> package
    (as documented recently in <code>quick-install.md</code>).</p>
</div>

<p>Now, a <b>very important</b> point needs to be addressed (no pun intended), which
    isn’t so much related to the fact one is running behind a reverse
    proxy, but related to the fact TCP/4443 and UDP/10000 ports are
    NAT’d: the videobridge component needs to know about that, and
    needs to know about the public IP and the local IP.

    In this context, the local IP is the Jitsi virtual machine’s local
    IP, where the NAT for TCP/4443 and UDP/10000 points to, and it
    is <b>not</b> the reverse proxy’s local IP.

    That’s why those lines have to be added to
    the <code>/etc/jitsi/videobridge/sip-communicator.properties</code>
    configuration file:</p>

<pre>
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=192.168.122.120
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=163.172.19.80
</pre>

<p>[ <i>Hint:</i> Beware, there’s
    another <code>sip-communicator.properties</code> configuration file, for
    the <code>jicofo</code> component! ]</p>

<div class="box has-background-warning">
  <p>Additionally, a default setting needs to be commented out (in the same file),
    because the TURN server isn’t installed:</p>

  <pre>
#org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=meet-jit-si-turnrelay.jitsi.net:443
</pre>
</div>

<p>Remember to restart the service:</p>

<pre>
systemctl restart jitsi-videobridge2
</pre>

<div class="box has-background-warning">
  <p><b>Update (April 2020):</b> Until late March 2020, this systemd
    server unit used to be called <code>jitsi-videobridge</code>
    instead.</p>
</div>

<h2><a name="more-about-stun-servers">More about STUN servers</a></h2>

<p>A privacy-concerned user was kind enough to inform a number of
    Jitsi instance administrators (including us) that the default
    Jitsi configuration uses Google’s STUN servers. This was fixed
    through a recent pull request:
    <a href="https://github.com/jitsi/jitsi-meet/pull/5433/files/e72960a782bc6268f8f15ccd2b4a7de880f3c2f1">config: use Jitsi’s STUN servers by default, instead of Google’s</a>.</p>

<p>Without waiting for a new upstream release, administrators can
    tweak their local configuration
    (in <code>/etc/jitsi/meet/F.Q.D.N-config.js</code>). This can be
    checked client-side by running <code>tcpdump</code> and checking
    packets are seen when a 2-participant conversation is set up:</p>

<pre>
tcpdump host meet-jit-si-turnrelay.jitsi.net
</pre>

<p>For completeness: Jitsi’s own infrastructure relies on Amazon Web
    Services at the moment.</p>

<h2>Annex: host networking configuration</h2>

<p>The relevant <code>iptables</code> rules on the host are the following
    (leaving aside the usual MASQUERADING which is required when using
    NAT):</p>

<pre>
Chain FORWARD (filter table)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            192.168.122.100      tcp dpt:80
ACCEPT     tcp  --  0.0.0.0/0            192.168.122.100      tcp dpt:443
ACCEPT     tcp  --  0.0.0.0/0            192.168.122.120      tcp dpt:4443
ACCEPT     udp  --  0.0.0.0/0            192.168.122.120      udp dpt:10000

Chain PREROUTING (nat table)
target     prot opt source               destination
DNAT       tcp  --  0.0.0.0/0            163.172.19.80        tcp dpt:80 to:192.168.122.100:80
DNAT       tcp  --  0.0.0.0/0            163.172.19.80        tcp dpt:443 to:192.168.122.100:443
DNAT       tcp  --  0.0.0.0/0            163.172.19.80        tcp dpt:4443 to:192.168.122.120:4443
DNAT       udp  --  0.0.0.0/0            163.172.19.80        udp dpt:10000 to:192.168.122.120:10000
</pre>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[What if you already have an Apache set up as a reverse proxy on your infrastructure?]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/jitsi-logo.png" /><media:content medium="image" url="https://debamax.com/jitsi-logo.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry xml:lang="en"><title type="html">Fixing faulty synchronization in Nextcloud</title><link href="https://debamax.com/blog/2020/02/29/fixing-faulty-synchronization-in-nextcloud/" rel="alternate" type="text/html" title="Fixing faulty synchronization in Nextcloud" /><published>2020-02-29T01:00:00+01:00</published><updated>2020-02-29T01:00:00+01:00</updated><id>https://debamax.com/blog/2020/02/29/fixing-faulty-synchronization-in-nextcloud</id><content type="html" xml:base="https://debamax.com/blog/2020/02/29/fixing-faulty-synchronization-in-nextcloud/"><![CDATA[<h2>Introduction</h2>

<p>Some problems were detected on a
    customer’s <a href="https://nextcloud.com/">Nextcloud</a>
    instance: disk space filled up all of a sudden, leading to service
    disruptions. Checking
    the <a href="http://munin-monitoring.org/">Munin</a> graphs, it
    seemed that disk usage gently increased from 80% to 85% over an
    hour, before spiking at 100% in a few extra minutes.</p>

<p>Looking at the “activity feed” (<code>/apps/activity</code>),
    there were a few minor edits by some users, but mostly many
    “<i>SpecificUser</i> has modified [lots of files]” occurrences. After
    allocating some extra space and making sure the service was
    behaving properly again, it was time to check with that
    <i>SpecificUser</i> whether those were intentional changes, or
    whether there might have been some mishap…</p>

<p>It turned out to be the unfortunate consequence of some disk
    maintenance operations that led to file system corruption. After
    some repair attempts, it seems the Nextcloud client triggered a
    synchronization that involved a lot of files, until it got
    interrupted because of the disk space issue on the server side. The question became:
    How many of the re-synchronized files might have been corrupted in
    the process? For example, <code>/var/lib/dpkg/status</code> had
    been replaced by a totally different file on the affected
    client.</p>

<h2>Searching for a solution</h2>

<p>Because of the possibly important corruption, a way to
    get back in time was to take note of all the “wanted” changes by
    other users, put them aside, restore from backups (previous
    night), and replay the changes. But then it was feared that any
    Nextcloud client having seen the new files could attempt to
    re-upload them, replacing the files that would have been just
    restored.</p>

<p>That solution wasn’t very appealing, that’s
    why <a href="https://twitter.com/CyrilBrulebois/status/1233405253569126400">Cyril tried his luck on Twitter</a>,
    asking whether there would be a way to revert all modifications
    from a given user during a given timeframe.</p>

<p>Feedback from the <a href="https://twitter.com/Nextclouders">@Nextclouders</a>
      account <a href="https://twitter.com/Nextclouders/status/1233419544116461569">was received shortly after that</a>,
    pointing out that such issues could have been caught client-side
    and a warning might have been displayed before replacing so many
    files, but that wasn’t the case unfortunately, and we were already
    in an after-the-fact situation.</p>

<p>
    The <a href="https://twitter.com/Nextclouders/status/1233419912477118464">second lead</a>
    could be promising if such an issue would be to happen more than
    once. All the required information is in the database already, and
    there’s already a malware app that knows how to detect files that
    could have been encrypted by a cryptovirus, and which would help
    restore them by reverting to the previous version. It should be
    possible to create a new application, implementing the missing
    feature by adjusting the existing malware app…</p>

<h2>Diving into the actual details</h2>

<p>At this stage, the swift reply from Nextcloud upstream seemed to
    indicate that early research didn’t miss any obvious solutions, so
    it was time to assess what happened on the file system, and see if
    that would be fixable without resorting to either restoring from
    backups or creating a new application…</p>

<p>It was decided to look at the last 10 hours, making sure to catch
    all files touched that day (be it before, during, or after the
    faulty synchronization):</p>

<pre>
cd /srv/nextcloud
find -type f -mmin -600 | sort &gt; ~/changes
</pre>

<p>Searching for patterns in those several thousand files, one could
    spot those sets:</p>

<ul>
    <li><code>data/commonuser/files/…</code><br />→ files that were
      indeed modified by various users; files belonging to, and shared
      by <i>CommonUser</i>.</li>
    <li><code>data/commonuser/files_versions/…</code><br />→
      versions of files that might or might not have been modified by
      the <i>SpecificUser</i>; files also belonging to, and shared
      by <i>CommonUser</i>.</li>
    <li><code>data/appdata_…/preview/…/….png</code><br />→
      previews/thumbnails likely generated on the fly when the
      activity stream was inspected.</li>
    <li><code>data/specificuser/uploads/…/…</code><br />→
      temporary files used during the synchronization, which were left
      around.</li>
  </ul>

<p>Good news: Provided there are no more running Nextcloud clients
    trying to synchronize things with the server for
    that <i>SpecificUser</i>, all those files
    under <code>data/specificuser/uploads</code> could go away
    entirely, freeing up 10 GiB.</p>

<p>Next: preview files were only 100 MiB, meaning spending more time
    on them didn’t seem worth it.</p>

<p>The remaining parts were of course the interesting ones: what
    about those <code>files/</code> versus <code>files_versions/</code> entries?</p>

<h2>Versions in Nextcloud</h2>

<p><b>Important note:</b> The following is based on observation on
    this specific Nextcloud 16 instance, which wasn’t heavily
    customized; exercise caution, and use at your own risk!</p>

<p>Without checking either code or documentation, it seemed pretty
    obvious how things work: when a given <code>foo/bar.baz</code>
    file gets modified, the previous copy is kept by Nextcloud, moving
    it from under <code>files/</code> to
    under <code>files_versions/</code>, adding a suffix. It is
    constructed this way: <code>.vTIMESTAMP</code>,
    where <code>TIMESTAMP</code> is expressed in seconds since
    epoch. Here’s an example:</p>

<pre>
./data/commonuser/files/foo/bar.baz
./data/commonuser/files_versions/foo/bar.baz.v1564577937
</pre>

<p>To convert from a given timestamp:</p>

<pre>
$ date -d '@1564577937'
Wed 31 Jul 14:58:57 CEST 2019

$ date -d '@1564577937' --rfc-2822
Wed, 31 Jul 2019 14:58:57 +0200

$ date -d '@1564577937' --rfc-3339=seconds
2019-07-31 14:58:57+02:00
</pre>

<p>Given there’s a direct mapping (same path, except under different
    directories) between an old version and its most recent file,
    this opened the way for a very simple check: “For each of those
    versions, does that version match the most recent file?”</p>

<p>If that version has the exact same content, one can assume that
    the Nextcloud client re-uploaded the exact same file (as a new
    version, though), and didn’t re-upload a corrupted file instead; which
    means that the old version can go away. If that version has a
    different content, it has to be kept around, and users notified
    so that they can check whether the most recent file is desired, or
    if a revert to a previous version would be better (Cyril acting as
    a system administrator here, rather than as an end-user).</p>

<p>Here’s a tiny shell script consuming the <code>~/changes</code>
    file containing the list of recently-modified files (generated
    with <code>find</code> as detailed in the previous section),
    filtering and extracting each version
    (called <code>$snapshot</code> in the script for clarity),
    determining the path to its most recent file by dropping the
    suffix and adjusting the parent directory, and checking for
    identical contents with <code>cmp</code>:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="nb">set</span> <span class="nt">-e</span>

<span class="nb">cd</span> /srv/nextcloud
<span class="nb">grep</span> <span class="s1">'/files_versions/'</span> ~/changes | <span class="se">\</span>
<span class="k">while </span><span class="nb">read </span>snapshot<span class="p">;</span> <span class="k">do
  </span><span class="nv">current</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$snapshot</span><span class="s2">"</span> | <span class="nb">sed</span> <span class="s1">'s/\.v[0-9][0-9]*$//'</span> | <span class="nb">sed</span> <span class="s1">'s,/files_versions/,/files/,'</span><span class="si">)</span>
  <span class="k">if </span>cmp <span class="nt">-s</span> <span class="s2">"</span><span class="nv">$snapshot</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$current</span><span class="s2">"</span><span class="p">;</span> <span class="k">then
    </span><span class="nb">echo</span> <span class="s2">"I: match for </span><span class="nv">$snapshot</span><span class="s2">"</span>
  <span class="k">else
    </span><span class="nb">echo</span> <span class="s2">"E: no match for </span><span class="nv">$snapshot</span><span class="s2">"</span>
  <span class="k">fi
done</span>
</code></pre></div></div>

<p>At this point, it became obvious that most files were indeed
    re-uploaded without getting corrupted, and it seemed sufficient to
    turn the first <code>echo</code> call into an <code>rm</code> one
    to get rid of their old, duplicate versions and regain an extra 2
    GiB. Cases without a match seemed to resemble the list of files
    touched by other users, which seemed like good news as well. To be
    on the safe side, that list was mailed to all involved users, so
    that they could check that current files were the expected ones,
    possibly reverting to some older versions where needed.</p>

<h2>Conclusion</h2>

<p>Fortunately, there was no need to develop an extra application to
    implement a new “let’s revert all changes from this user during
    that timeframe” feature to solve this specific case. Observation
    plus automation shrank the list of 2500+ modified files to just a
    handful that needed manual (user) checking. Some time lost, and
    some space that was reclaimed in the end. Not too bad for a Friday
    afternoon…</p>

<p>Many thanks to the Nextcloud team and community for a great piece
    of software, and for a very much appreciated swift reply!</p>]]></content><author><name></name></author><category term="blog" /><summary type="html"><![CDATA[Sometimes a client sync in Nextcloud goes haywire, here’s a little workaround.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://debamax.com/nextcloud-logo-background.png" /><media:content medium="image" url="https://debamax.com/nextcloud-logo-background.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>