In this post we will focus on why continuous go fuzzing is needed and what are the challenges in implementing continuous fuzzing. Previous posts/papers regarding why go fuzzing in general is important as well as why it’s a good idea even to integrate it as part of the Go toolchain can be found here (written by Dmitry Vyukov and Romain Baugue – highly recommended).
A good walkthrough both for writing and building go-fuzz targets can be found on our example repo as well as in the go-fuzz repository. Our example contains also a basic example of how to integrate continuous fuzzing via Fuzzit.
Fuzzing Quick Recap
Essentially fuzzing consist of two types of jobs
- Fuzzing – which is essentially a job that can run infinitely. This job automatically generates interesting test-cases that cover more paths, as well as monitors for crashes and other memory related problems.
- Sanity (Not sure it’s the official term but we use it) – Run the same fuzzer through set of specific test-cases (Usually that were generated by the long running jobs). This is usually a very quick job.
Continuous Fuzzing Challenges
There are a few challenges/questions that arise from how to integrate fuzzing to the current CI. We will walk through some of them as there are a lot of other open questions that really depends on the development workflow and the specific project.
Challenge 1 – Long Running Jobs: Fuzzing is a long-running (infinite) job unlike a CI that we try to keep as short as possible to provide fast feedback for commits/PRs.
Solution 1 – Async Jobs: This where we need to spawn a different server or use a platform like Fuzzit to run the fuzzers asynchronously. The platform will notify the administrators or the relevant security people of any new vulnerabilities that the fuzzers find (This might take days-months).
Challenge 2 – Targets*Versions=$$$: or which version to fuzz? We need to decide wisely which versions to fuzz as if will blindly fuzz all possible versions in a project infinitely for X targets that might cost us a lot of money.
Solution 2 – Master + Stable: One approach that we saw popular with our users is fuzzing the development branch (master) and release branch. The development branch is fuzzed continuously and the fuzzer is updated every time new code is pushed to master. The updated fuzzers check the additional code but keeps the corpus from previous runs. This way the fuzzers can essentially always continue from where they stopped and only work on the additional code. Fuzzit helps with managing the corpus and keeping it in minimised state.
Challenge 3 – Learning from old mistakes: Once we setup continues fuzzing we aggregate very valuable test-cases and crashes that we fix along the way. We would love to use all those precious test-cases to check every PR before it get’s merged.
Solution 4 – Sanity Fuzz Tests: For every PR just like unit-test we run the fuzzers through all the generated test-cases and the fixed crashes which is usually a very quick process which fits a classic CI. Fuzzit helps running the fuzzers with the aggregated corpus from the Fuzzit servers, fail the CI and alert the developer immediately.
This was a quick walkthrough of the some of the challenges of integrating continuous fuzzing to C/C++/Go/Rust projects from our experience. If you want to join the continuous fuzzing revolution sign-up at https://app.fuzzit.dev follow us on twitter and join our slack .