Packaging CrowdSec for Debian: Bookworm (Debian 12)

13 June 2023

Introduction

What CrowdSec is, how Debamax got involved, and what happened during the Bullseye release cycle can be found in the previous blog post: Packaging CrowdSec for Debian: Bullseye (Debian 11).

There were two main goals during the Bookworm release cycle: upgrading the existing crowdsec package (Security Engine) to catch up with the new upstream releases (from 1.0.x to 1.4.x), and packaging some bouncers (Remediation Components). Bouncers are responsible for taking action when needed.

Build dependencies

Bouncers

It was mentioned in the previous blog post that crowdsec was an end-user program, packaged accordingly. That means the generated executables (crowdsec and cscli) are shipped in the crowdsec binary package, alongside configuration files and a copy of hub data (under /usr/share/crowdsec/hub).

Since one of the goals is to add support for bouncers, that means building a library package as well, since parts of the crowdsec source code are required. Now, the crowdsec source package builds the crowdsec binary package (as it did before) and also the golang-github-crowdsecurity-crowdsec-dev one. This makes it possible to have packages that include github.com/crowdsecurity/crowdsec in the require part of their go.mod file. This addition was first staged in an upload to experimental so that it could be reviewed from NEW by the FTP team whenever they had time, without disrupting ongoing work in unstable/testing.

With that new package, it was possible to introduce golang-github-crowdsecurity-go-cs-bouncer which is the main building block for bouncers, and is required by both the firewall and custom bouncers. The firewall one also needed an extra package, to deal with firewall rules, hence golang-github-google-nftables on the bouncer-related graph below (important packages in magenta, new dependencies in green).

crowdsec

A bigger amount of work was required for the crowdsec package itself, since its go.mod grew many new dependencies between the 1.0.x versions and the 1.4.x ones. The plan was published as a request for comments: Updating crowdsec, adding and updating other packages.

As usual, there were some rather easy additions, but also some packages to rename:

The ent ecosystem was shuffled around a little and since the upstream locations are directly linked to module names, that meant introducing new packages that weren’t entirely new (e.g. golang-entgo-ent-dev replacing golang-github-facebook-ent-dev).
Some of the packages introduced in the previous release cycle for the initial crowdsec upload were also forked/adopted officially by the CrowdSec team (e.g. golang-github-crowdsecurity-grokky-dev replacing golang-github-logrusorgru-grokky-dev).

Regarding existing packages, Cyril could count on Shengjing Zhu (Debian Go Packaging Team) to take care of updating the golang-github-apparentlymart-go-textseg package which was a little tricky. There were some inconsistencies between documentation and actual usage when it comes to including a trailing /vN suffix in the go.mod files versus the actual contents of the package, where there’s no vN directory. Long story short: there was no need to introduce a new golang-github-apparentlymart-go-textseg-v13 package.

The golang-github-hashicorp-hcl-v2 package was an entirely different story: the existing golang-github-hashicorp-hcl has 98 reverse dependencies and it seemed much safer to just introduce a new package, even if only used by crowdsec initially. Adding that package makes it possible for the aforementioned reverse dependencies to migrate progressively, but it should also help other packages that ship a copy of hashicorp/hcl/v2 in a vendor directory (like golang-github-hashicorp-nomad-dev).

There were two extra existing packages to update: golang-github-gin-gonic-gin and golang-github-zclconf-go-cty. The ratt-powered process described in the previous blog post confirmed the updates were reasonable, as the only packages that wouldn’t build from source with the updated packages were already riddled with release critical bugs, so they were no factor.

Here is a complete graph of all relevant packages: 16 new packages (green) and 3 updated packages (red), in addition to crowdsec itself (magenta).

Complete dependency graph

Adding the bouncer-related packages to the previous graph gives the full picture for the Bookworm release cycle: 18 new packages (green) and 3 updated packages (red), in addition to crowdsec and its two bouncers (magenta).

Updating crowdsec

Upstream patches

This is the same story as for Bullseye: software must be tailored to the distribution’s needs or best practices, which is usually implemented via a series of patches. Here is a quick summary of the patches found in the Bookworm version:

0003: Adjusts the crowdsec.service systemd unit to match Debian’s needs and best practices.
0004: Drops support for geoip-enrich which relies on non-free GeoIP data.
0005: Tweaks paths in the config file to point at /var/lib/crowdsec instead of /etc/crowdsec for the hub.
0007: Enables the online hub transparently when cscli hub update is called.
0008: Adjusts the require and import directives, replacing github.com/r3labs/diff/v2 with github.com/r3labs/diff/v3.
0009: Disables an acquisition module which would require adding many more packages, which isn’t warranted since it’s not indispensable.
0010: Disables some tests that would require deploying a significant test infrastructure. Those tests are run on upstream’s CI infrastructure anyway.
0011: Refreshes code generated from Protobuf specifications, to make sure we don’t run into version skews between build-time and run-time. This patch was provided by upstream very quickly when we understood where the test failures were coming from. This is definitely the kind of strong commitment Debian Developers like to see from their upstream!
0013: Disables some tests that are supposed to check performance. Unfortunately, those error out at random, be it on the build daemon network while building the package (when the test suite is being run, which leads to a build failure), and/or on Debian’s CI infrastructure (where autopkgtest-based testing happens). Since those tests are being picky about what the results should look like, and since test failures don’t really indicate actual problems, disabling them is best.
0014: Silences some log related to YAML patching. That’s a feature that was introduced in the 1.4.x versions, that can be used by crowdsec itself and by bouncers as well. Details can be found in later sections.
0015: Silences some messages about the version’s being outdated. We’re shipping the package in a stable distribution, it’s totally fine not to try and update all the time! Even more so since upstream is committed to providing long term support for versions that end up in Debian stable releases.
0016: Adjusts some tests around syslog parsing. No longer hardcoding the reference values for year-less syslog messages means two things. First, rebuilding the package next year is not going to fail (due to an off-by-one error), which could affect security updates or point releases. Second, that means happier reproducible builds: one of the common variations there is to build a given package in the past or in the future, which would trigger discrepancies and test failures without that patch.

Packaging separate resources

There are still two additional original tarballs for the new upstream release:

the data tarball got refreshed from the various sources it aggregates;
the hub tarball received many changes as the hub kept growing, and bundling the contents of the upstream v1.4.6 branch meant a significant bump: from 279 files in crowdsec_1.0.9.orig-hub1.tar.gz to 1304 files in crowdsec_1.4.6.orig-hub1.tar.gz!

While the logic behind those additional original tarballs stayed exactly the same as it was before, the fact there are many more items being shipped in the offline hub meant we had to adjust the strategy when it comes to deploying collections: not all of them are useful for a default Linux setup, and some of them can be very resource-intensive!

Plugging everything together: maintainer scripts

Foreword: the crowdsec.postinst script is called after installation, but it can be called in many ways as documented in the Debian Policy.

The main focus in this section is the configure action, which is used when configuring the package for the first time (fresh installation) but also when upgrading it. In the latter case, an extra parameter is provided by dpkg, so that the script knows which version of the package it is upgrading from, which helps make informed decisions.

Case 1: Initial installation

For a fresh installation, the new strategy regarding collections is to only deploy a few of them, following upstream’s recommendations:

crowdsecurity/linux
crowdsecurity/apache2
crowdsecurity/nginx

Since there’s no tooling around enabling collections from the local filesystem (offline hub), the crowdsec.postinst script has to create symlinks on its own, and to do that correctly, it also needs to consider all items each collection requires (i.e. parsers, scenarios, postoverflows, and other collections…) and enable them as well.

When everything is fine, cscli collections list would return something like this:

COLLECTIONS
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                                📦 Status   Version   Local Path
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
 crowdsecurity/apache2               ✔️ enabled   0.1       /etc/crowdsec/collections/apache2.yaml
 crowdsecurity/base-http-scenarios   ✔️ enabled   0.6       /etc/crowdsec/collections/base-http-scenarios.yaml
 crowdsecurity/http-cve              ✔️ enabled   1.9       /etc/crowdsec/collections/http-cve.yaml
 crowdsecurity/linux                 ✔️ enabled   0.2       /etc/crowdsec/collections/linux.yaml
 crowdsecurity/nginx                 ✔️ enabled   0.2       /etc/crowdsec/collections/nginx.yaml
 crowdsecurity/sshd                  ✔️ enabled   0.2       /etc/crowdsec/collections/sshd.yaml
──────────────────────────────────────────────────────────────────────────────────────────────────────────────

A missing item (e.g. after failing to create a symlink to a required parser) would lead to the following warning:

crowdsecurity/linux                 ⚠️ enabled,tainted   0.2       /etc/crowdsec/collections/linux.yaml

Enabling upstream-recommended collections and their dependencies is implemented via two lists: UPSTREAM_COLLECTIONS and UPSTREAM_ITEMS.

Case 2: Upgrading from an earlier version

Since there’s also no tooling to maintain existing collections on the local filesystem (offline hub), it seemed reasonable to concentrate on the three collections mentioned in the previous section: they were deployed initially, and they would only need to get a few more items (which could be called “new dependencies”) enabled to make sure they appear as enabled and not enabled,tainted.

Since it’s important to respect the admin’s decisions, that only happens if all three collections are still enabled when performing the upgrade. If one of them is missing, it’s assumed the admin took control, and will deal with collections on their own.

This is also implemented by using the same two lists as mentioned above: UPSTREAM_COLLECTIONS and UPSTREAM_ITEMS.

New topic: SQLite-related warnings

Upstream is really careful about many things, and the default settings are meant to be safe. The default configuration relies on a small SQLite database, and to avoid problems on some filesystems, the WAL feature is disabled by default. This is fine except it generates warnings in various places, and it seemed best to address it.

That’s why crowdsec.postinst detects whether a local configuration is present. If there’s none, it will try and detect whether the database backend is sqlite, whether WAL is enabled, and the path to the database. If the answers to the first two questions is yes and no respectively, it will check which filesystem is used for the database, and it will automatically enable the WAL feature unless nfs* is detected.

To avoid modifying the package-provided main configuration file, /etc/crowdsec/config.yaml (which would trigger prompts during further upgrades), the db_config.use_wal boolean is stored in what upstream calls a YAML patch, in a new /etc/crowdsec/config.yaml.local file, that gets layered on top of the main configuration file. Again, to respect any changes that might have been deployed by the admin, the auto-detection is skipped if that file is already present.

Starting to use this YAML patch feature is why the 0014 patch mentioned above was introduced: without it, some warning would be printed when the daemon starts, and whenever a command reads the config (i.e. each and every cscli call).

Checking the upgrade path from Bullseye to Bookworm

At that point, it looked like everything should be in place: upstream patches, updated data and hub files, updated crowdsec.postinst code to manage initial collection deployment and also upgrading from a previous version.

Since fresh installations were tested very easily and regularly (purging the package ensures starting from a clean slate every time), it was time to focus on verifying the upgrade path from the version found in Bullseye. This time, not just making sure collections are handled properly, but also ensuring the runtime would work fine as well.

Spoiler alert: there were a few more changes to the crowdsec.postinst script!

Problem #1: delays

The first bad surprise was huge delays during the upgrade, for no obvious reasons at first, filed as #1031326. Upon investigation, it became clear the old daemon had troubles stopping, and one needed to wait until the default TimeoutStopSec before systemd switched from SIGTERM to SIGKILL. Upstream confirmed there were various issues with the event loops in the 1.0.x versions, and that it was unlikely a solution could be found and deployed via a point release. This meant we couldn’t fix the version getting upgraded from, so that it wouldn’t exhibit this problem.

To avoid waiting 90 seconds, it was agreed with upstream to implement a workaround, lowering the timeout to 20 seconds, which should leave coroutines plenty of time to commit any pending changes to the database, even if a clean exit couldn’t be reached afterwards. This is done only when upgrading from 1.0.x, by setting a temporary override for the systemd unit (crowdsec.service), that’s only valid until the next reboot (temporary storage under /run/systemd/system).

Problem #2: SQLite incompatibilities

Another upgrade issue is related to one of the few native libraries used by the crowdsec package. While most of the code is written in Go and statically linked, there’s an external dependency on libsqlite3-0. Unfortunately, two incompatibilities were discovered:

The old crowdec can’t work with the new libsqlite3-0 since assumptions in the ent framework would no longer be valid. This was filed as #1033029 against new libsqlite3-0, so that its maintainer could consider adding a Breaks relationship against the old crowdsec package to avoid this particular combination. László Böszörményi quickly uploaded the package with the proposed patch, and it ended up in testing a few days later.
Conversely, the new crowdsec can’t work with the old libsqlite3-0 since it relies on syntax that was only added in a version between Bullseye’s version and Bookworm’s, tracked in #1033132. What happens here is that the dependency on the native library is computed using what’s called the shlibs/symbols mechanism, which determines the minimal version that’s needed to ensure the libsqlite3-0 package contains all the ELF-level symbols used by crowdsec. Unfortunately, that doesn’t account for SQLite syntax support, and some higher dependency was hardcoded to reflect the actual runtime needs regarding SQLite language support, and not just the required symbols.

Combining both fixes ensures a correct upgrade path from Bullseye to Bookworm, making sure partial upgrades are fine: even when the package manager is performing a full upgrade from oldstable to stable, it splits work into sets of packages, and it was critical to avoid having both packages in different sets!

Problem #3: missing decisions

Yet another upgrade issue was missing decisions after the upgrade, for several hours, filed as #1033138. Most users are likely to expect to get decisions from the Central API (CAPI), via the Community blocklist alert. Unfortunately, the format changed between 1.0.x and 1.4.x, and even if the relevant entries in the database would still exist, they wouldn’t be taken into account by the new version. Even worse, since the last CAPI pull is considered recent enough, this issue would persist for several hours, leaving users without any kind of protection.

An easy fix would have been to purge all alerts before starting the new version, but that would also lose local alerts (e.g. manual additions by the admin). Another easy fix would have been to purge the Community blocklist before restarting, but the old API didn’t know how to perform specific deletions.

That’s why the following was implemented:

when upgrading from 1.0.x;
once restarted into 1.4.x;
if there are alerts matching Community blocklist, delete them;
if deletions happened, restart once more.

The deletion and restart logic is conditioned by a flag set when upgrading from 1.0.x. It makes sure there’s an immediate CAPI pull, instead of waiting for the next scheduled one, which would default to waiting 2 hours without any kind of protection via the Community blocklist.

Problem #4: missing decisions, again

Seeing how decisions could go missing during upgrades, it seemed important to inspect what happens on fresh installations as well. That’s how it was spotted that there are no guarantees regarding the first time crowdsec starts, with many coroutines running in parallel, and there might not be any successful CAPI pull. This also means leaving users without any kind of protection right after the initial installation, for several hours.

That led to implementing another workaround:

when installing;
poll the logs once every second;
as soon as a message about added/deleted entries appears, the CAPI pull is confirmed, and we can move on;
if there’s no such message after 20 seconds, force a CAPI pull by restarting the daemon.

Bouncers

Introduction

The official term in the upstream documentation is Remediation Components, but they’re called bouncers most of the time. They are in charge of acting upon decisions provided by the Security Engine. Again, after discussion with upstream, it was decided to focus on two bouncers.

The firewall bouncer will periodically fetch new, expired, and deleted decisions, and adjust the firewall configuration to reflect those changes. This is how one deploys a crowd-powered fail2ban-like setup.

The custom bouncer is a rather simple yet very flexible bouncer, which must be configured with a command via the bin_path configuration key. It will also periodically fetch new, expired, and deleted decisions, but it will pass them as arguments to the configured command. There are no requirements on that command, it can be a shell script or anything else.

There are various ways to deploy crowdsec (the Security Engine) and bouncers, and this flexibility is reflected in the packaging as well. Both bouncers have crowdsec only in Recommends, which is basically an optional dependency (as opposed to Depends), allowing for different setups:

One can deploy crowdsec on a particular machine of the infrastructure, dealing both with Central API exchanges and exposing the Local API for use by other machines on the network. Bouncer(s) can then be deployed on other machines, configured to use the Local API exposed by the first machine, without needing a crowdsec package installed locally. In this case, each bouncer would require at least the api_url and api_key keys to be set in their configuration file.
One can opt for a simpler, all-in-one solution: installing bouncer(s) on a machine will automatically install crowdsec (unless the admin requested otherwise), which by default configures its Local API to listen on the loopback. In that case, bouncers will detect that crowdsec is present on the same machine, and will automatically register themselves, meaning no extra steps for the admin (no need for api_url or api_key at least).

This auto-registration seemed to be working fine initially but the order in which packages are configured when Recommends are involved isn’t guaranteed like it would be when using Depends. Thankfully there was still some time left in the Bookworm release cycle to fix those issues with minimal changes, that were swiftly reviewed and approved by the release team. The updated logic is the following:

Bouncers can still auto-register as they did before, if crowdsec is installed and if it is configured already.
If crowdsec is only installed but hasn’t been configured yet, they can queue their registration. In this case, crowdsec.postinst is responsible for an extra task: processing the registration queue.

The configuration for the bouncers is similar to what happens on the crowdsec side: an upstream-provided configuration file is shipped under /etc/crowdsec/crowdsec-<NAME>-bouncer.yaml. The YAML patching is also supported, that’s why the postinst script of each bouncer creates a .local file instead of touching the main configuration file. This includes the api_key when auto-registering, but also other parameters that are detected on the local system, and/or that are expected to be changed by the admin. Additionally, a .id is left behind with the bouncer’s name, so that the postrm script (run by dpkg when removing a package) can also proceed to auto-unregistration.

Firewall bouncer

It can be trivially installed on Debian 12 with just the following command (which will pull and configure crowdsec automatically as described above):

apt-get install crowdsec-firewall-bouncer

The crowdsec-firewall-bouncer script detects which firewall system is currently deployed, and can deal with nftables (the default since Debian 11), or iptables. In the latter case, ipset must also be installed. Some warning message is shown in the tricky case: nftables and iptables are both installed, with the iptables alternative pointing to iptables-legacy (instead of the now-default iptables-nft)… while ipset isn’t installed. Unfortunately, the Depends syntax can’t really express nftables or iptables+ipset, especially with iptables listing nftables in Recommends to provide an upgrade path for older systems.

The installation triggers those messages on successful installation:

Setting up crowdsec-firewall-bouncer (0.0.25-1+b2) ...
I: Configuring nftables [see README.Debian]
To adjust the config: editor /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.local && systemctl restart crowdsec-firewall-bouncer
Created symlink /etc/systemd/system/multi-user.target.wants/crowdsec-firewall-bouncer.service -> /lib/systemd/system/crowdsec-firewall-bouncer.service.

The README.Debian file can be found on disk under /usr/share/doc/crowdsec-firewall-bouncer/ as usual, and can also be viewed online.

After installation, with the default, upstream-recommended collections mentioned earlier for crowdsec, a few thousand lines are expected when asking for the ruleset list:

nft list ruleset | wc -l

Custom bouncer

The installation is almost as trivial as the firewall bouncer one:

apt-get install crowdsec-custom-bouncer

But this time around, it must be configured to start doing something useful, as suggested when the configuration happens:

Setting up crowdsec-custom-bouncer (0.0.15-1+b2) ...
W: Using /usr/bin/true as default custom script
To adjust the config: editor /etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml.local && systemctl restart crowdsec-custom-bouncer
Created symlink /etc/systemd/system/multi-user.target.wants/crowdsec-custom-bouncer.service -> /lib/systemd/system/crowdsec-custom-bouncer.service.

The crowdsec-custom-bouncer.postinst script initially provisions the local configuration with bin_path: /usr/bin/true, which gives a nice no-op placeholder that lets the service start successfully, even before the admin starts adjusting the configuration.

While this bouncer is very flexible and can run anything, one should probably keep in mind that using it means spawning many processes. For example, after simply setting bin_path: /usr/bin/logger and restarting the bouncer, one can see the following:

# journalctl | grep 'root\[' | head -1
Jun 13 08:00:00 crowdsec root[14812]: add 1.2.3.4 666 crowdsecurity/ssh-bf {/* JSON1 */}

# journalctl | grep 'root\[' | tail -1
Jun 13 08:00:05 crowdsec root[21905]: add 5.6.7.8 666 crowdsecurity/http-probing {/* JSON2 */}

(IP addresses are redacted, and JSON metadata is skipped for readability.)

This means that the custom command was spawned more than 7000 times!

One should also note that the bin_path parameter expects an absolute path to an executable (program or script), and that cannot include options (e.g. bin_path: "/usr/bin/logger -t custom-test" wouldn’t work) or rely on shell expansion. If the custom command is actually a well-known program that should be called with specific options, a standard solution would be to write a simple wrapper, store it under /usr/local/bin for example, and point bin_path to that wrapper.

Related tasks for the Debian Go team

Here are two additional pointers to related tasks:

RFC: Fixing golang-github-tidwall-gjson’s RC bugginess for bookworm: there were known security issues in one of the libraries used by crowdsec, Thorsten Alteholz (another member of the Debian Go Packaging Team) agreed with the plan, then the release team agreed as well, and everything happened a few months before the Bookworm release.
RFC: Removing some unneeded packages from unstable (and testing): since some packages introduced earlier are no longer needed, and haven’t become dependencies of other packages, some clean-up was suggested. Since there were many things to keep everyone busy, that didn’t happen before the Bookworm release. Removal requests are likely to be filed early in the Trixie (new testing and future Debian 13) release cycle.

Conclusion

While there were significant efforts during the Debian 11 release cycle, there wasn’t enough time to package both the Security Engine (crowdsec) and Remediation Components (bouncers). Thankfully, the initial packaging work proved to be appropriate groundwork for the crowdsec upgrade from 1.0.x to 1.4.x, even if a number of fixes or workarounds were needed to ensure a smooth upgrade experience. Adding crowdsec-firewall-bouncer and crowdsec-custom-bouncer along crowdsec itself completed the work started right before the Bullseye freeze, finally making it possible to take action on Bookworm!

Package	Version
crowdsec	1.4.6-4
crowdsec-custom-bouncer	0.0.15-3
crowdsec-firewall-bouncer	0.0.25-3

Version table for Debian 12 “Bookworm”

About upstream

Debamax as a company and Cyril as a Debian Developer are very pleased with all interactions with CrowdSec’s upstream. Developers are clearly interested in what is best for Debian users, and they’re willing to compromise and/or work a little more to make sure creating, then updating the Debian packages goes as smoothly as possible. This includes proposing solutions to avoid hard-to-solve, Protobuf-related version skews. Very nice cooperation!