In this blog post we will walk through how systemd integrated continuous fuzzing using the Fuzzit platform. Even though it was done with the Fuzzit platform we will talk about general design decisions and approaches in the process of setting a continuous fuzzing workflow that can be applicable to any platform as well as an in-house solution.
This post assumes basic knowledge and familiarity with fuzzing and specifically with libFuzzer fuzzer.
Writing the fuzz targets
The first step before setting up a continuous fuzzing will be to write the fuzz targets. The systemd team did a great job writing and maintaining more than 30 targets (Coverage is available here). We won’t go (at least not in this blog post) into how to choose which functions/targets to fuzz and how many of them to create and assume the targets are already written and now we have to decide When? How? And in what scale to fuzz them. Or in other words – how the continuous fuzzing workflow will look like.
Continuous Fuzzing Workflow
Given that we have a lot of fuzzing targets for this project and a limited amount of resources we need to use them smartly.
The first question that we asked is which version/branch we want to fuzz because obviously we can’t fuzz all branches but we do want to find new bugs as soon as possible when they are introduced.
We got to the decision of fuzzing the master continuously while updating the binary of the fuzzer on a daily basis to find new bugs and keeping the corpus between versions so they can learn from different runs and not start all over again each day. So, this looks something like this (for each target):
systemd is used by a wide range of Linux distribution when each of them is using a specific version and sometimes even a custom version with their own changes. So, from our point of view it would be smart for those distribution to fuzz a specific version as there might be unknown bugs which are fixed in master and thus not found by the fuzzers and can be found only by fuzzing a specific version deployed by the specific distribution.
Sanity Tests & PRs
We also wanted to take advantage of all the generated test-cases (the corpus) and the crashes. libFuzzer can run in two modes either fuzzing, generating and looking for new test-cases (which is an infinite process essentially) or running through a specific set of test-cases which is usually a very fast one – what we call a sanity test. For every merge and pull-request we run a fast sanity test with the generated corpus and past crashes to quickly check if old crashes are introduced again or new ones are introduced from the generated corpus. This way we can catch bugs quickly and early in the pull-request process. Running the fuzzing sanity tests can be looked at as a very good addition to unit-tests that usually run on every PR.
This is a sneak peak at how the integration looks on the fuzzit dashboard:
On the CI side you can see the additional steps in .travis.yml and the integration script itself can be found at fuzzit.sh which is essentially building the fuzz targets and sending them via the fuzzit CLI.
More information and ideas about continuous fuzzing can also be found in our docs.
Crashes which are found using the fuzzing process are usually a sensitive data, at least until they are fixed in master + some reasonable time so third-part vendors will have enough time pushing the fix. Due to it being an open-source project, only specific people in the systemd team has access and get notifications of new crashes introduced by the fuzzers so data won’t leak to external contributors before a fix is introduced.
This is also a big subject that will be addressed in a different post of how to handle bugs in general and specifically those found by fuzzers even though they are not assigned a CVE. An interesting discussion can be found at oss-sec.
Big thanks and kudos to the systemd team and especially to Evgeny Vereshchagin (@yevvers) who maintains the fuzzing targets and integration at systemd.
OSS Project? ping us at firstname.lastname@example.org and we will help you setup continuous fuzzing for free!