3. GNATfuzz User’s Guide
3.1. Introduction
GNATfuzz provides fuzz-testing facilities for Ada projects. The main features it provides are:
Identifying subprograms that are suitable for automated fuzz-testing (referred to as automatically fuzzable subprograms).
Generating fuzz-test harnesses that receive binary encoded data, invoke the subprogram under test, and report run-time anomalies.
Generating fuzz-test initial test cases.
Building and executing fuzz-testing campaigns controlled by user-defined stopping criteria.
Real-time test coverage analysis through integration with GNATcoverage.
These features are available both via the command line and through integration with the GNAT Studio IDE.
The fuzzing engine in GNATfuzz is based on AFL++, a popular open-source and highly configurable fuzzer, and an instrumentation plugin shipped as part of the GNAT Pro toolchain.
The primary goal of GNATfuzz is to abstract a lot of the technical details needed to set up, run, and understand the outcome of a fuzzing session on an application.
3.1.1. Fuzz-testing in a nutshell
Note
In the context of GNATfuzz, we refer to the project or program that you’re targeting with fuzz-testing as the “system under test”.
Fuzzing (or fuzz-testing) in the context of software is an automated testing technique that, based on some initial input test cases (the corpus) for a program, will automatically and repeatedly run tests and generate new test cases at a very high frequency to detect faulty behavior of the system under test. Such erroneous behavior is captured by monitoring the system for triggered exceptions, failing built-in assertions, and signals such as SIGSEGV (which shouldn’t occur in Ada programs unless they’re erroneous).
Fuzz-testing is now widely associated with cybersecurity. It has proven to be an effective mechanism for finding corner-case code vulnerabilities that traditional human-driven verification mechanisms, such as unit and integration testing, can miss. Such code vulnerabilities can often lead to malicious exploitations.
Fuzz-testing comes in different flavors. Black-box fuzz-testing refers
to the approach of randomly generating test cases without taking into
account any information about the internal structure, functional behavior,
or execution of the system under test. AFL++’s native execution mode is
considered a grey box fuzzer because it obtains code coverage by using code
instrumentation information. This information helps the mutation and
generation phase of the fuzzer generate test cases that have a higher
probability of increasing code coverage than randomly generated test cases
would. This instrumentation is added automatically by GNATfuzz during the
fuzz mode, explained in detail later in this guide.
3.1.2. AFL++ basics
This section contains a very high-level overview of AFL++, a highly-configurable, instrumentation-guided genetic fuzzer. A GCC plugin, enabled via the AFL-GCC-fast compiler driver directive, generates efficient code instrumentation that enables AFL++ to process GCC-compiled code. Once the system under test is compiled with the AFL++ GCC plugin, you can use the “afl-fuzz” fuzzing driver to run a fuzzing session. The different basic elements and phases involved in an AFL++ fuzzing session are described in Fig. 3.1
Note
This document is not intended to be a substitute for the AFL++ documentation or that of its GCC plugin extensions, but rather to provide some information needed to understand AFL++ and how it works. In this document, we mostly focus on the added value of the GNATfuzz toolchain. For more information about AFL++, you can visit the AFL++ official site.
Fig. 3.1 Overview of an AFL++ fuzz-testing session
Prior to starting a fuzzing session, you must provide some initial test cases, called the corpus, each in a separate file. Starting a fuzzing session moves the corpus to what’s called the test cases queue, which isn’t really a queue, but instead a directory containing a list of files, each representing a test case. AFL++ adds tests to the queue, but never removes any from it.
The details are complex, but in general, for each test case in the queue, AFL++ uses one or more internal or external mutators to create multiple new test cases (it can create between hundreds and tens of thousands of new tests from each test in the queue), each of which it executes on the system under test. It monitors each execution for any faulty behavior, as describe above, and moves any test cases that cause such behavior to a directory called crashes. AFL++ provides a configuration parameter which sets a timeout for each test. If the test exceeds the timeout, it’s killed and stored in the session’s hangs directory. Saving these tests allows you to replay and investigate these problematic test cases.
While running each test, AFL++ collects all the instrumentation data and updates its bitmap, an array global to the session where each index represents an instrumentation point added to the system under test during its compilation by the GCC AFL++ plugin. Whenever an instrumentation point is executed, its corresponding index in the bitmap is incremented. The bitmap enables the fuzzing session to be path-coverage aware.
If a mutated test case executes a new path, it’s considered a good candidate to further increase the coverage when mutated and is added to the queue so AFL++ will generate further mutations of that test. The fuzzing session cycle counter is increased when all the test cases in the queue have been through the fuzzing process at least once. The cycle counter is part of the fuzzing statistics that GNATfuzz produces and stores during a session. An example file containing these statistics is shown below:
Fig. 3.2 Overview of an AFL++ fuzzer_stats file (produced by afl-fuzz++3.00a)
Some of these statistics are also included in the fuzzing session information terminal (which can be optionally shown during the execution of GNATfuzz):
Fig. 3.3 Fuzzing session information terminal example. (produced by afl-fuzz++3.00a)
Some of the most important statistics are:
start_time - start time of afl-fuzz (as a unix time)
last_update - last update of this file (as a unix time)
fuzzer_pid - PID of the fuzzer process
cycles_done - queue cycles completed so far
execs_done - number of execution calls to the system under test
execs_per_sec - current number of executions per second
corpus_count - total number of entries in the queue
corpus_found - number of entries discovered through local fuzzing
corpus_imported - number of entries imported from other instances
max_depth - number of levels in the generated data set
cur_item - entry number currently being processed
pending_favs - number of favored entries still waiting to be fuzzed
pending_total - number of all entries waiting to be fuzzed
stability - percentage of bitmap bytes that behave consistently
corpus_variable - number of test cases showing variable behavior
unique_crashes - number of unique crashes recorded
unique_hangs - number of unique hangs encountered
The complete list of all the statistics generated by AFL++ and the description of each can be found here. These statistics are meant to assist you in making informed decisions on when to stop a fuzzing session. Typically, based on the statistics, you can identify when a fuzzing session has plateaued, meaning that GNATfuzz has no further potential to explore new execution paths in the system under test.
Setting up, running, and stopping a fuzzing session can be challenging. You must build and execute a coverage campaign that allows you to understand the impact of fuzz-testing on your project. There are many steps to this. First, you must construct an adequate starting corpus. The quality of the starting corpus can significantly affect the fuzzing session’s results. The more meaningful the starting corpus is as an input to the system under test, the more coverage GNATfuzz can achieve. After providing the corpus, you need to wrap the system under test in a suitable test harness. Next, you have to build the wrapped system with the GCC AFL++ plugin to add instrumentation to the code. Then you need to call the command that starts the fuzzing session with all the required configuration flags and actively monitor it to decide if it has performed enough fuzzing.
AFL++ fuzzing is primarily used to fuzz programs that accept files as inputs. This makes it more challenging to use AFL++ as a form of unit-level vulnerability testing. The current built-in mutators will mutate test cases at the binary level and are unaware of any structure within the test cases. This can be problematic when, for example, it mutates static data that should not be changed, such as an array’s upper and lower bounds - their mutation can result in corruption of the test-case data resulting in a wasted mutation cycle.
3.1.3. GNATfuzz overview
GNATfuzz utilizes several AdaCore technologies to provide ease of use and enhanced fuzzing capability for Ada and SPARK programs. Fuzzing Ada and SPARK code has a significant advantage over fuzzing other traditional, less memory-safe languages, like C. Ada’s extended runtime constraint checks can capture faulty behavior, such as overflows, which often go undetected in C. This provides a significant advantage because the software assurance and security confidence that can be achieved when fuzzing Ada and SPARK programs is higher than when fuzzing C programs.
In a nutshell, GNATfuzz abstracts away the complexity of the building, execution, and termination of fuzzing campaigns and enables unit-level-based fuzz-testing for Ada and SPARK programs by:
Automating the detection of fuzzable subprograms.
Automating the creation of the test-harness for a fuzzable subprogram.
Automating the creation and minimization of the starting corpus.
Allowing for simple specification of criteria for stopping the session and monitoring how those criteria are being met during the session.
Leveraging the GNATcoverage tool to provide coverage information during or after a fuzzing session.
Supporting multicore fuzzing.
Fig. 3.4 GNATfuzz overview
Fig. 3.4 gives an overview of the GNATfuzz toolchain. GNATfuzz offers several modes which you can use to largely automate the manual activities needed for setting up and running a fuzz-test campaign. The green boxes in Fig. 3.4 represent the four main operating modes, of GNATfuzz: analyze, generate, corpus-gen, and fuzz.
The gnatfuzz command line interface provides complete access to these features and the IDE integration in GNAT Studio provides a full UI workflow to drive the command line interface.
We start by giving a very high-level description of the GNATfuzz workflow and then provide details about the information footprints of each GNATfuzz mode and how each works in later sections.
The workflow shown in Fig. 3.4 is intended for unit-level fuzz-testing. In this scenario, you first identify fuzzable subprograms within the system under test. You do this by specifying the project file of the system under test as an argument to the analyze mode of GNATfuzz. This mode scans the project for all fuzzable subprograms and lists them in a JSON file, including the source file and line information for each. Subprograms considered to be fuzzable are those for which the GNATfuzz generate mode can automatically generate a starting corpus. See section GNATfuzz automation capabilities and limitations for the auto-generation capabilities of the current version of GNATfuzz.
You then select one of these fuzzable subprograms and specify it as an argument to the generate mode of GNATfuzz. This mode creates a test harness for that subprogram, including all the execution scripts needed to run the fuzzing campaign and the input-data specifications for the subprogram. Based on these input-data specifications, you can automatically generate the starting corpus using the corpus-gen mode. After you run that mode, all the prerequisites needed to execute a fuzz-testing session are in place. The two last steps you might optionally want to perform are creating stopping criteria for the fuzzing campaign and manually providing some additional test cases for the starting corpus.
You can then call the GNATfuzz fuzz mode. This builds the system under test with AFL++ code instrumentation, using the AFL-GCC-fast compiler plugin. The system under test is also built with support for GNATcoverage; see the GNATcoverage User’s Guide for more information about the GNATcoverage tool.
After the system is built, fuzz mode executes one or more AFL++ fuzzing sessions. By default, GNATfuzz handles the multicore aspects of the fuzz testing campaign, such as allocating parallel fuzzing sessions of the system under test to available cores. It also periodically synchronizes the test-case queues and coverage reporting and monitors the stopping criteria between all the parallel fuzzing sessions. For details of the internal processes of a test fuzzing session, see Fig. 3.3.
Finally, all fuzzing activities will stop when the stopping criteria (either specified by you or by default) are met, at which point the final results of the fuzzer are ready for your inspection, including the set of crashing test cases, the set of test cases causing the system to timeout (the hangs), the set of test cases that increased coverage, the coverage report, and the AFL++ fuzzing statistics.
In the following sections, we first provide detailed information on the various GNATfuzz modes, including their configuration, and then provide examples, first using the GNATfuzz command-line interface and then using the GNATstudio IDE GNATfuzz interface, to illustrate these modes.
3.2. GNATfuzz command line interface
This section describes the various possible parameters for each GNATfuzz mode. To see the detailed effect, the artifacts each switch can generate, and how these artifacts can be fed into the next mode of the GNATfuzz toolchain, see section GNATfuzz by example.
Note
For this documentation, the ‘$’ sign denotes the system prompt.
3.2.1. GNATfuzz “analyze” mode command line interface
In this mode GNATfuzz has the following command line interface:
$ gnatfuzz analyze [--help|-h] [-P <PROJECT>] [-S <SPEC-FILE>] [-X <NAME=VALUE>] [--disable-styled-output]
where
- --help, -h
Shows the usage information for the analyze mode
- -P <PROJECT>
Specifies the project file to be analyzed Mandatory
- -S <SPEC-FILE>
Targets the analysis at an Ada specification file within the given project
- -X <NAME=VALUE>
Sets the external reference
NAMEin the system-under-test’s GPR project toVALUE
- --disable-styled-output
By default GNATfuzz may style some of its output using ANSI escape sequences. This switch can be used to disable this behavior.
- --no-subprojects
By default GNATfuzz analyzes sources in all projects, to the exclusion of externally built projects. With this switch, GNATfuzz will only process sources in the root project.
3.2.2. GNATfuzz “generate” mode command line interface
In this mode GNATfuzz has the following command line interface:
$ gnatfuzz generate [--help|-h] [-P <PROJECT>] [-S <SOURCE-FILE>] [-L <LINE-NUMBER>]
[--analysis <ANALYZE-JSON>] [--subprogram-id <SUBPROGRAM-ID>]
[-o <OUTPUT-DIRECTORY>] [-X <NAME=VALUE>] [--disable-styled-output]
where
- --help, -h
Shows the usage information for the generate mode
- -P <PROJECT>
Specifies the project file to be analyzed Mandatory
- -S <SOURCE-FILE>
Specifies the source file of the subprogram under test Mandatory Requires: -L Mutually Exclusive: --analysis, --subprogram-id
- -L
Line number of the subprogram under test (indexed at 1) Mandatory Requires: -S Mutually Exclusive: --analysis, --subprogram-id
- --analysis
Path to the analysis json generated by the analyze mode Mandatory Requires: --subprogram-id Mutually Exclusive: -S, -L
- --subprogram-id
Id of the target fuzzable subprogram found in the analysis file Mandatory Requires: --analysis Mutually Exclusive: -S, -L
- -o
Path where the generated files are created. If not specified, the generated files will be placed under the default location
<PROJECT-OBJ-DIR>/gnatfuzz/harness.
- -X <NAME=VALUE>
Sets the external reference
NAMEin the system-under-test’s GPR project to the valueVALUE
- --disable-styled-output
By default GNATfuzz may style some of its output using ANSI escape sequences. This switch can be used to disable this behavior.
3.2.3. GNATfuzz “corpus-gen” mode command line interface
In this mode GNATfuzz has the following command line interface:
$ gnatfuzz corpus-gen [--help|-h] [-P <PROJECT>] [-o <output-directory>]
[-X <NAME=VALUE>] [--disable-styled-output]
where
- --help, -h
Shows the usage information for the corpus-gen mode
- -P <PROJECT>
Specifies the fuzz-test.gpr project file created during a GNATfuzz generate mode associated with the fuzzing campaign we want to generate a corpus for Mandatory
- -o
Path where the generated files are created Mandatory
- -X <NAME=VALUE>
Sets the external reference
NAMEin the system-under-test’s GPR project toVALUE
- --disable-styled-output
By default GNATfuzz may style some of its output using ANSI escape sequences. This switch can be used to disable this behavior.
3.2.4. GNATfuzz “fuzz” mode command line interface
In this mode GNATfuzz has the following command line interface:
$ gnatfuzz fuzz [--help|-h] [-P <PROJECT>] [-X <NAME=VALUE>]
[--cores CORES] [--corpus-path CORPUS-PATH] [--no-cmin]
[--seed SEED] [--no-cmplog] [--afl-mode AFL-MODE]
[--no-deterministic-phase] [--no-GNATcov]
[--stop-criteria STOP-CRITERIA] [--ignore-stop-criteria]
[--disable-styled-output]
where
- --help, -h
Shows the usage information for the fuzz mode
- -P <PROJECT>
Specifies the fuzz-test.gpr project file created during a GNATfuzz generate mode associated with the fuzzing campaign we want to launch Mandatory
- -X <NAME=VALUE>
Sets the external reference
NAMEin the system-under-test’s GPR project toVALUE
- --cores
The number of cores the fuzzing campaign can utilize. If 0 (also the default if no value is passed), the campaign will utilize the maximum number of available cores. The provided value must be in the range of 0 to 255
- --corpus-path
Path to the starting corpus to be used
- --no-cmin
Do not run the corpus minimizer tool “afl-cmin”; afl-cmin reduces the starting corpus down to a set of test cases that all find unique paths through the control flow of the system-under-test
- --seed
Use a fixed seed for AFL random number generators. This is mainly used for deterministic benchmarking. The provided value must be in the range of 0 to the maximum file size of the starting corpus test case files (in bytes)
- --no-cmplog
Disable the CMPLOG (aka RedQueen) feature (see section CMPLOG (aka RedQueen) support)
- --afl-mode
Specify the AFL_Mode to run. Possible alternatives: afl_defer_and_persist, afl_persist, afl_defer, afl_plain. Default: afl_persist
- --no-deterministic-phase
Do not run the deterministic (sequential bit flip) mutation phase
- --no-GNATcov
Do not run the (near) real-time dynamic coverage analysis using GNATcoverage (statement)
- --stop-criteria
Override the default stop criteria rules with the provided
- --ignore-stop-criteria
Ignore stop criteria rules (fuzzer must be manually stopped)
- --disable-styled-output
By default GNATfuzz may style some of its output using ANSI escape sequences. This switch can be used to disable this behavior.
3.3. GNATfuzz by example
This section provides several examples to demonstrate the use of the GNATfuzz tool. These examples incrementally increase in complexity and represent the main capabilities of GNATfuzz. You can run these cases on your machine. All examples are located under:
<install_prefix>/share/examples/gnatfuzz/
3.3.1. Prerequisites and possible configurations
However, as discussed below, you need to be aware of some AFL++ and GNATfuzz requirements and configurations prior to using the technology.
3.3.1.1. AFL++ requirements
To allow the AFL++ tool, and therefore GNATfuzz, to run successfully, you
need to execute the following commands as root (either by logging in as
root or running su or sudo, as appropriate for your system) before
running any fuzzing sessions:
$ echo core >/proc/sys/kernel/core_pattern
$ cd /sys/devices/system/cpu
$ echo performance | tee cpu*/cpufreq/scaling_governor
The first command avoids having crashes misinterpreted as timeouts and the second is required to improve the performance of a fuzz-testing session. For more information, see the relevant AFL++ implementation details.
3.3.1.2. GNATfuzz requirements
The current requirements for GNATfuzz are:
GNATfuzz is currently only supported by GNAT Pro Ada x86 64 bit Linux distributions.
GNATfuzz requires that Ada projects be built using GPR project files and the GPRbuild compilation tool.
Warning
GNATfuzz has no specific hardware requirements but is a highly resource-intensive application. Executing a fuzz-testing session could run thousands of simultaneous tests, which can occupy the majority of the resources of a computing system. Ensure you allocate the number of cores you feel comfortable with to a fuzz-testing session (see section GNATfuzz “fuzz” mode command line interface).
3.3.1.3. GNATfuzz configurations
GNATFUZZ_XTERM : by default, the display of the AFL++ terminals (see
Screenshot 3.3) is disabled during a GNATfuzz
testing session. To enable their display set this environment variable.
3.3.1.4. Accommodating linker flags required by the user project
If a user project requires any specific linker flags or to link to a library, the flags need to be added to the list of the linker switches of the test harness project file generated by GNATfuzz. Namely, the path_to_generated_test_harness/fuzz_testing/fuzz_test.gpr. For example, if the user’s project contains C++ code and it needs to link to the C++ runtime, the linker switch -lstdc++ needs to be included in the test harness generated project file:
1 package linker is
2 for Driver use common_driver;
3 -- add any linker switches required by the user project:
4 for Switches ("Ada") use ("-lstdc++");
5 end linker;
3.3.1.5. GNATfuzz and the AFL++ execution modes
AFL++ provides four fuzz-testing execution modes: plain,
persistent, defer, and defer-and-persistent. You can choose one
of these via the command line switch --afl-mode when running
GNATfuzz in fuzz mode . If you don’t specify this switch, GNATfuzz
uses the persistent mode.
Different AFL++ executions modes cause the GNATfuzz fuzz-testing session to behave in different ways. These behaviors are explained in the following sections.
Fig. 3.5 GNATFuzz AFL++ supported modes overview
3.3.1.5.1. GNATfuzz and the AFL++ plain mode
When GNATfuzz runs in AFL++ plain mode, a new process is spawned
for the execution of each test. This guarantees that all
application memory is reset between tests, however this has a
large overhead because, on top of the time required to spawn a new process,
all initialization code is rerun for every test case.
3.3.1.5.2. GNATfuzz and the AFL++ persistent mode
When GNATfuzz runs in AFL++ persistent mode, groups of up to 10,000
tests run within the same process before it spawns a new one
to run the the next set of tests. This is more efficient, as you only
have to pay the time cost of spawning a new process once every 10,000
test cases.
However, as many test cases are run in the same process, any dynamically allocated memory that is modified during execution of a test must be reset, as these changes will otherwise persist while subsequent test cases are executed. Test case failures may not be reproducible in cases where dynamically allocated memory is not fully reset, as the failure could be a result of changes made during previous test runs, instead of depending solely on the data from the current test.
For GNATfuzz, this applies both to any library resources that require you
to free memory, as well as to reset any dynamically allocated state, for example,
any memory allocated from a pool of available memory (often referred to as
a heap) via the Ada keyword new.
You can find an example of how to reset dynamically allocated memory in the program under test before each test execution in section Resetting dynamically allocated memory when fuzz-testing with the AFL++ persistent mode.
Note
One way to tell if changes are not being fully reset between test runs
is to observe the stability value. 100% stability indicates that when the same
input data is used, it produces the same result, whereas lower values indicate that
this is not always the case. Stability can be monitored through the GNATfuzz stopping
criteria, as well as through the AFL++ terminal. If it is not feasible to completely
reset state between test runs, consider using the plain, defer, or defer_and_persist modes.
3.3.1.5.3. GNATfuzz and the AFL++ defer mode
As illustrated in Fig. 3.5, when running in defer
mode, only one process is spawned to execute test cases (when running
with only one core), and then after some initial setup, this process
is forked to run each test case. The benefit of this is that any dynamically
allocated memory that is modified during the execution of a test case,
is effectively automatically reset before the next test case is run.
You do not have to manually clean up after the previous test case.
This is also useful as it allows you to perform large initializations once
before the process is forked, and you do not have to worry about re-doing
this work between test cases. This can be done using the Deferred_Mode_Setup
function in the User_Configuration package (see section
The “User_Configuration” package).
Deferred_Mode_Setup differs from the Setup function in the user
configuration package in two ways:
It can not depend on test case data
When running in
defermode,Deferred_Mode_Setupwill only be called once, whereasSetupwill be called once per test
That said, in most cases defer mode is unlikely to be faster than
running in persist mode with a optimized set of Setup and
Teardown functions.
Note that if you are running in plain mode, Deferred_Mode_Setup
will be called once per test cases. When running in persist mode
it will be called once per group of tests (at least every 10,000 test cases).
3.3.1.5.4. GNATfuzz and the AFL++ defer-and-persistent mode
AFL++ defer-and-persistent mode combines the AFL++ defer and
persist modes, as illustrated in Fig. 3.5. This
can be useful in cases where test stability is not 100%, but is close,
and so you still want to make use of the performance improvements afforded
by the persist mode. When running using defer-and-persistent mode,
a new process is forked to run groups of up to 10,000 test cases, but
unlike when just running in persist mode, the Deferred_Mode_Setup
function is only ever called once.
3.3.2. GNATfuzz automation capabilities and limitations
Section GNATfuzz overview provides an overview of the stages you need to use to get a fuzzing session up and running in an automated fashion, each corresponding to running one of the four operating modes of GNATfuzz; namely, analyze, generate, corpus-gen, and fuzz.
The degree to which GNATfuzz can automate a testing session is determined mainly by two factors: its ability to auto-generate test harnesses and its ability to auto-generate a starting corpus. We divide subprograms into three categories based on those two factors:
Fully-supported fuzzable subprograms: ones for which GNATfuzz can auto-generate both a test-harness and a starting corpus.
Partially-supported fuzzable subprograms: ones for which GNATfuzz can auto-generate a test-harness but not a starting corpus. In these cases, GNATfuzz provides a subprogram that allows you to provide your own seeds. GNATfuzz will encode the manually provided seeds and generate a starting corpus.
Non-supported fuzzable subprograms: subprograms for which GNATfuzz can neither auto-generate a test-harness nor a starting corpus.
Which subprograms belong to each of the three categories will evolve with future releases of GNATfuzz, with the goal of increasing automation in later releases. Also, the outcome of a GNATfuzz campaign targeting a non-compatible subprogram is dependent on the nature of the incompatibility. In some cases, the test harness will fail to compile. In other cases, the test harness will build and execute but not produce anything meaningful. In future versions of GNATfuzz, incompatible subprograms will be identified during the analysis mode.
In the current version, a test harness can be auto-generated for a subprogram unless one of the following are true:
Any of the subprogram’s “in” or “in out” mode parameters are of an Access type or contain a sub-component of an Access type.
Any of the subprogram’s “in” or “in out” mode parameters are Subprogram Access Types.
Any of the subprogram’s “in” or “in out” mode parameters are Limited types.
Any of the subprogram’s “out” mode parameters are discriminated or variant records with a discriminant without a default value.
Any of the subprogram’s “in” or “in out” mode parameters types are null records, derive from null records or contain a null record sub-components.
Any of the subprogram’s “in” or “in out” mode parameters types are tagged types.
Any of the subprogram’s “in” or “in out” mode parameters types are array types with more than 1000 components.
Any of the subprogram’s “in” or “in out” mode parameters types is a type composed by another type that that meets any of the rules above.
Any of the subprogram’s “in” or “in out” mode parameters types is a type with a Static_Predicate aspect.
GNATfuzz can automatically generate a starting corpus for a subprogram unless any of the following are true:
A test harness can’t be generated for the subprogram (see above).
Any of the subprogram’s “in” or “in out” mode parameters is a private type with a component whose type is declared in the package’s private declarative part.
Any of the subprogram’s “in” or “in out” mode parameters is an anonymous type or composed by one.
Section Fuzzing a simple Ada subprogram illustrates a case of a fully-supported fuzzable subprogram and section Fuzzing an autogenerated test-harness with user-provided seeds illustrates how you can fuzz a partially-supported fuzzable subprogram by providing your own seeds via a GNATfuzz-generated interface. Finally, section Fuzzing instantiations of generic subprograms illustrates how to fuzz instantiations of generic subprograms.
3.3.2.1. Fuzz-testing when corpus auto-generation is unsupported
Users can provide seeds to start a fuzz-testing campaign when corpus auto-generation is unsupported. In some cases, it could be challenging to determine such valid test cases.
While providing a fuzz-testing campaign with a valid set of starting test cases is considered best practice, GNATfuzz also allows seeding a campaign using arbitrary test cases that may or may not contain valid data. Suppose a user chooses to provide a starting corpus. GNATfuzz will utilize coverage-driven fuzzer mutations coupled with a highly fuzz-efficient marshaller technology called “Test Generator” (TGen) to convert invalid test cases into valid ones. This behavior holds even if all test cases in the starting corpus are invalid.
TGen is AdaCore’s dynamic marshalling technology that removes a lot of memory padding in test cases (typically present to ensure byte alignment of data components). TGen’s compact representation of values significantly increases the probability of a successful mutation that results in a new valid test case. This approach to corpus generation is practical when GNATfuzz cannot automatically generate a valid starting corpus.
Section Fuzzing with an arbitrary starting corpus illustrates how fuzz-testing campaigns can still be executed with a dummy starting corpus on subprograms whose parameters use Ada types that are unsupported by GNATfuzz’s corpus auto-generation.
3.3.3. Fuzzing a simple Ada subprogram
Let us consider this simple project:
1package body Simple is
2
3 function Is_Divisible (X : Integer; Y : Integer) return Boolean
4 is
5 begin
6 if X mod Y = 0 then
7 return true;
8 else
9 return false;
10 end if;
11 end Is_Divisible;
12end Simple;
and:
1package Simple is
2
3 function Is_Divisible (X : Integer; Y : Integer) return Boolean;
4 -- checks if X is divisible by Y
5
6end Simple;
This project consists of a single function that takes two integer arguments
and returns true if the first (X) is divisible by the second
(Y). The code contains a deliberate bug: division by 0 is possible. In
a few steps, let’s demonstrate how the GNATfuzz toolchain can automatically
detect this bug.
3.3.3.1. Using analyze mode
First, we run the GNATfuzz analyze mode on the project:
$ gnatfuzz analyze -P simple.gpr
Empowered by the Libadalang technology, this mode scans the project to
detect fuzzable subprograms and generate an analysis file under
<simple-project-obj-dir>/gnatfuzz/analyze.json, which contains a table
listing all the fuzzable subprograms detected. In the case of this project,
that file looks like:
1{
2 "user_project": "<install_prefix>/share/examples/gnatfuzz/simple/simple.gpr",
3 "scenario_variables": [],
4 "fuzzable_subprograms": [
5 {
6 "source_filename": "<install_prefix>/share/examples/gnatfuzz/simple/src/simple.ads",
7 "start_line": 3,
8 "id": 1,
9 "label": "Is_Divisible",
10 "corpus-gen-supported": true
11 }
12 ]
13}
Here we see that GNATfuzz determined that the subprogram declared in the
simple.ads specification file at line 3 is fuzzable. This corresponds
to the Is_Divisible function. You can now use the GNATfuzz generate
mode to autogenerate a test harness suitable for fuzzing.
3.3.3.2. Using generate mode
We can do this in one of two ways.
We can specify the location of the subprogram as given in analyze.json:
$ gnatfuzz generate -P simple.gpr -S ./src/simple.ads -L 3 -o generated_test_harness
Or we can use the unique identification number (id) for the subprogram as given in the analyze.json file:
$ gnatfuzz generate -P simple.gpr --analysis ./obj/gnatfuzz/analyze.json --subprogram-id 1 -o generated_test_harness
Either of these commands generates a new fuzz test-harness project and all
the artifacts needed to support a fuzz-test session for the selected
subprogram. These are placed under the specified output directory,
generated_test_harness in our case, with the following directory
structure:
├── generate_test_harness
│ ├── fuzz_testing -- subprogram-specific test-harness artifacts
│ │ ├── build -- corpus tools
│ │ ├── generated_src -- test-harness source code
│ │ └── user_configuration -- user configurable files
│ └── gnatfuzz_shared -- gnatfuzz lib artifacts
│ ├── ...
│ └── ...
In short:
fuzz_testing/fuzz_test.gpr- the new project file of the test-harness that wraps the subprogram-under-test to enable injecting of test cases, capturing crashes, and coverage analysis.fuzz_testing/generated_src- the test-harness source code that you should not modify under normal circumstances.fuzz_testing/user_configuration- the test-harness source code that we may modify when needed. For example you may provide its starting corpus (see section Fuzzing an autogenerated test-harness with user-provided seeds).fuzz_testing/fuzz_config.json- contains all the information/configurations needed by the GNATfuzz fuzz mode to enable the fuzz-testing of the generated test-harness.fuzz_testing/build- includes all the corpus-related tools needed for GNATfuzz to automatically generate the starting corpus as well as the encoding and decoding of test cases.
3.3.3.3. Using corpus-gen mode
The subprogram under test is suitable for starting corpus auto-generation
(see the "corpus-gen-support": true field at line 9 of the
<simple-project-obj-dir>/gnatfuzz/analyze.json). Therefore, we can use
the following GNATfuzz command to automatically generate the starting
corpus for our fuzzing session:
$ gnatfuzz corpus-gen -P ./generated_test_harness/fuzz_testing/fuzz_test.gpr -o starting_corpus
This generates multiple test_cases under the starting_corpus directory.
Each is suitable for testing the subprogram chosen using the fuzzing
mode. The test cases are in a binary format.
Note
Currently, starting corpus auto-generation support includes all the integral types and arrays of integral types (including strings). There are ongoing efforts to increase the supported types to the full scope of Ada types. Despite this, GNATfuzz generate can produce test-harnesses for a more extensive set of subprograms, beyond the ones for which their starting corpus can be auto-generated. GNATfuzz also automatically generates a framework to assist you in manually providing the starting corpus for these cases. We demonstrate this in section Fuzzing an autogenerated test-harness with user-provided seeds.
Note
The corpus-gen mode is currently in its preliminary stages. It includes only one default strategy of auto-generating a starting corpus (using the First, Middle, and Last values for scalar types). Thus, when it’s feasible to auto-generate the starting corpus for a subprogram, you can currently skip the corpus-gen mode. In this case, the fuzz mode calls the corpus-gen mode in the background before performing any fuzzing-relating activities.
3.3.3.4. Decoding an auto-generated test case
To inspect the contents of a generated binary test case, GNATfuzz’s decoding utility can be used. Access to this utility can be enabled by setting GNATFUZZ_BUILD_DECODER environment variable before the execution of GNATfuzz’s generate mode. Then, the utility can be found at:
<simple_example_generated_test_harness_path>/fuzz_testing/build/decode_test
For example:
$ ./decode_test ./starting-corpus/GNATfuzz_1
Parameter: Param 1 [Integer] = -2147483648
Parameter: Param 2 [Integer] = -2147483648
Here, the GNATfuzz_1 auto-generated test case is decoded and its values are printed. Param 1 and Param 2 correspond to the X and Y arguments of the Is_Divisible function, respectively.
Note that when decoding a CMPLOG generated test case the switch -CMPLOG can be used with decode_test:
$ ./decode_test ./cmplog_used_starting_corpus/GNATfuzz_1 -CMPLOG
3.3.3.5. Using fuzz mode
Now everything is in place for fuzz-testing the selected subprogram under test. We can use the following command to launch a fuzzing session:
$ gnatfuzz fuzz -P ./generated_test_harness/fuzz_testing/fuzz_test.gpr --corpus-path ./starting_corpus/ --cores=2
This command does the following:
Builds the test harness.
Builds the system-under-test with the AFL instrumentation pass.
Builds the system-under-test with GNATcoverage.
Minimizes the starting corpus using the afl-cmin tool to reduce the starting corpus to the test cases that execute unique paths of the system under test.
Launches the AFL++ fuzz-testing session.
Collects dynamic coverage and fuzzing statistics.
Monitors the stop criteria and stops the fuzzing session if they’re met.
Displays status information.
Important
If the optional –corpus-path fuzz mode switch
is not used, an internal GNATfuzz temporary directory will be
used to place the generated corpus. The generated corpus or its
minimized version (if corpus minimization is not disabled) will
then be transferred to the used_starting_corpus directories shown in
Fig. 3.10. These directories hold
the final corpus to be used for starting the fuzz-testing campaign.
Furthermore, the default behavior is to execute GNATfuzz’s
corpus generation. If the user wants only to use their provided
starting seeds, either through the –corpus-switch or through the
user-configuration package, then GNATfuzz’s corpus generation
can be disabled through the user’s configuration package
User_Provided_Seeds procedure (see
Source Code 3.16
lines 4 and 5).
You can see the status information displayed in this example during a fuzzing session for this system under test in Fig. 3.6. GNATfuzz displays various general information points, including the elapsed time since the fuzzing session was started, the number of anomalies detected, information on code coverage based on GNATcoverage statement coverage and the progress on meeting the predefined stopping criteria for the session. Both the coverage information and stopping criteria sections can be turned off using the fuzz-mode’s --no-GNATcov and --ignore-stop-criteria switches, respectively.
Fig. 3.6 GNATfuzz fuzzing session status overview
Important
The stopping criteria are defined in XML format. GNATfuzz uses a set
of default stopping criteria if you don’t provide them via the
--stop-criteria fuzz-mode switch. The generate mode also
creates a local copy of this default stopping criteria XML file
within the auto-generated test harness. You can inspect and modify
these default criteria and can point to them by specifying the switch
--stop-criteria=<path-to-generated-test-harness>/fuzz_testing/user_configuration/stop-criteria.xml.
If you want to avoid using the stopping criteria and manually stop a
fuzzing session using ctrl+c on the terminal, you must use the
--ignore-stop-criteria fuzz-mode switch. More information
on GNATfuzz’s stopping criteria can be found in section
GNATfuzz stopping criteria.
The fuzzing command generates a session directory under
<path-to-generated-test-harness>/fuzz_testing/session: with the
following structure:
session/
├ afl-fuzz.<N>.log -- The STDOUT from the AFL++ afl-fuzz node executing on core N
├ afl-fuzz.<N+1>.log -- The STDOUT from the AFL++ afl-fuzz node executing on core N+1
├ fuzz.log -- The diagnostics log file generated by GNATfuzz during fuzz mode
├ gnatfuzz_stats.log -- The STDOUT from GNATfuzz during fuzz mode
├── build -- The object-files of various compilations needed
│ ├── obj-<AFL mode> -- The build directory of the AFL++ instrumented test harness
│ │ ├ gnatfuzz_test_harness.verbose -- The verbose test harness that will print the test results - uses TGen encoding
│ ├── obj-<AFL mode>_CMPLOG -- The build directory of the AFL++ CMPLOG instrumented test harness
│ └ gnatfuzz-verbose_test_harness -- The verbose test harness that will print the test results - uses CMPLOG encoding
├── coverage_output -- Dynamic coverage reporting
├── fuzzer_output -- AFL++ generated artifacts
│ ├── gnatfuzz_1_master -- The "master" node's fuzzing-process generated artifacts
│ │ ├ fuzzer_setup -- CL and associated env vars used to launch afl-fuzz
│ │ ├ fuzzer_stats -- Latest information about this AFL fuzzing node
│ │ ├── crashes -- Test-cases found to crash the system-under-test
│ │ ├── hangs -- Test-cases found to hang the system-under-test
│ │ └── queue -- Test-cases found to exercise a unique execution path
│ └── gnatfuzz_<2..N>_slave -- A "slave" node's fuzzing-process generated artifacts
│ ├ fuzzer_setup -- CL and associated env vars used to launch afl-fuzz
│ ├ fuzzer_stats -- Latest information about this AFL fuzzing node
│ ├── crashes -- Test-cases found to crash the system-under-test
│ ├── hangs -- Test-cases found to hang the system-under-test
│ └── queue -- Test-cases found to exercise a unique execution path
├── used_starting_corpus -- The starting-corpus used by the fuzzing session
├── CMPLOG_fuzzer_output -- AFL++ CMPLOG generated artifacts (same structure as fuzzer_output)
├── CMPLOG_used_starting_corpus -- The starting-corpus used by the CMPLOG fuzzing session
├── minimized_starting_corpus -- The result of minimizing the starting corpus
├── symcc_output -- Holds all the SymCC-generated test cases
└── results -- The synchronized results of the GNATfuzz fuzz mode
├── crashes -- All test-cases found to crash the system-under-test
├── hangs -- All test-cases found to hang the system-under-test
└── ending_corpus -- All test-cases found to exercise a unique execution path
You should be aware of the following when evaluating or interpreting a fuzzing session:
Multiple fuzzing processes: The fuzz-mode switch --cores specifies the number of fuzzing processes to run. Each process is created on an individual core and generates its artifacts directory under the
fuzzer_outputdirectory. The first process is always the master (gnatfuzz_1_master) and the rest are slave processes. The more cores used, the more computing power is allocated to the fuzzing session and may speed up coverage and bug finding. But be careful not to exhaust your system resources since that will have a negative effect on fuzzing speed. In order to monitor the results of a fuzzing session, it is enough to inspect only the master’s process artifacts (“gnatfuzz_1_master”). This is because all the slave processes communicate their findings to the master process. Also note that CMPLOG requires a full core to execute and will therefore reduce the number of cores available to non-CMPLOG AFL++ nodes by 1.Coverage: The dynamic coverage collected during a fuzzing session is provided in the
coverage_outputdirectory. This is statement-based coverage using the source-code instrumentation coverage provided by GNATcoverage (see source-traces-based coverage by GNATcoverage). Anindex.htmlis generated under the coverage directory, which is accessible via a browser. This allows you to navigate the coverage-information annotated code of the system under test. The coverage percentages are currently reported for the entire compilation module and not a subprogram selected from that module for fuzzing. So even though the fuzzing can execute all the statements in the selected subprogram, if the compilation module contains more subprograms than the one chosen for fuzzing, the coverage percentage may never reach 100%. The exception is if the subprogram under test performs calls to all other subprograms. You should keep this in mind when using coverage as a stopping criterion.Used Starting Corpus: The used_starting_corpus directory of the session directory is the starting corpus used by the fuzzing session. This might be the same as the autogenerated or user-provided starting corpus or might be a reduced version if you enable afl-cmin. Note that the afl-cmin is currently enabled by default by the fuzz mode of GNATfuzz (use the --no-cmin to disable it.)
AFL++ raw fuzzing stats: You can inspect the raw AFL++ generated statistics during the fuzzing session by examining the fuzzer_stats file under the master’s process artifacts directory (gnatfuzz_1_master). An example of an AFL++ status file is shown in Fig. 3.2.
When the fuzz-testing session is completed (either because the stopping criteria are satisfied or we stopped it manually), we can check the results/crashes directory to examine if the fuzzer was able to capture the expected divide-by-zero bug within the subprogram under test. To inspect the values and the behavior of the crashing test case when executed with the subprogram under test, you can run it with the gnatfuzz_test_harness.verbose executable as follows:
$ cd <path-to-generated-test-harness>/fuzz_testing/session/results
$ ls crashes/
id:000000,sig:06,src:000000,time:64,execs:258,op:flip32,pos:4 README.txt
$ ../build/obj-AFL_PERSIST/gnatfuzz_test_harness.verbose crashes/id\:000000\,sig\:06\,src\:000000\,time\:64\,execs\:258\,op\:flip32\,pos\:4
@@@GNATFUZZ_OUTPUT_START@@@
{
"Decoded_In_Parameters": [
{
"Parameter_Name": "X",
"Parameter_Type": "Standard.Integer",
"Parameter_Value": "-1"
},
{
"Parameter_Name": "Y",
"Parameter_Type": "Standard.Integer",
"Parameter_Value": " 0"
}
],
"Subprogram_Under_Test": "Simple.Is_Divisible",
"Testcase_Exception": {
"Exception_Information": "raised CONSTRAINT_ERROR : simple.adb:6 divide by zero\n",
"Exception_Message": "simple.adb:6 divide by zero",
"Exception_Name": "CONSTRAINT_ERROR"
}
}
@@@GNATFUZZ_OUTPUT_END@@@
As you can see, GNATfuzz produced a single crashing test case (note that the naming of the crashing case can vary). The verbose execution of this crashing test case reveals the cause of the crash: the test case has a value of zero for the divisor (argument Y of Is_Divisible in Code Sample 3.7) causing a CONSTRAINT_ERROR.
Note
The gnatfuzz_test_harness.verbose execution of a crashing test case can be used within a debugger to understand further and correct the erroneous behavior of the subprogram under test.
3.3.4. Fuzzing an autogenerated test-harness with user-provided seeds
Let’s now try to fuzz the records example, which contains the following
source:
1 package Show_Date is
2
3 type Months is
4 (January, February, March, April,
5 May, June, July, August, September,
6 October, November, December);
7
8 type Date is record
9 Day : Integer range 1 .. 31;
10 Month : Months;
11 Year : Integer range 1 .. 3000 := 2032;
12 end record;
13
14 procedure Display_Date (D : Date);
15 -- Displays given date
16
17 end Show_Date;
1 with Ada.Text_IO; use Ada.Text_IO;
2
3 package body Show_Date is
4
5 procedure Display_Date (D : Date) is
6 begin
7 Put_Line ("Day:" & Integer'Image (D.Day)
8 & ", Month: "
9 & Months'Image (D.Month)
10 & ", Year:"
11 & Integer'Image (D.Year));
12 end Display_Date;
13
14 end Show_Date;
First, we run the GNATfuzz analyze mode on the project:
$ gnatfuzz analyze -P records_test.gpr
This mode scans the project to detect fuzzable subprograms and generates an
analysis file <records_test-project-obj-dir>/gnatfuzz/analyze.json,
which contains a table listing all the fuzzable subprograms detected. In
our case, the analysis information file looks like this:
1 {
2 "fuzzable_subprograms": [
3 {
4 "corpus_gen_supported": false,
5 "id": 1,
6 "label": "Display_Date",
7 "reason": "Corpus gen for <ConcreteTypeDecl [\"Date\"] show_date.ads:8:4-12:15> is not supported",
8 "source_filename": "<install_prefix>/share/examples/gnatfuzz/records/src/show_date.ads",
9 "start_line": 14
10 }
11 ],
12 "non_fuzzable_subprograms": [
13 ],
14 "scenario_variables": [
15 ],
16 "user_project": "<install_prefix>/share/examples/gnatfuzz/records/records_test.gpr"
17 }
By inspecting the generated analysis file, we can see there’s one fuzzable
subprogram that GNATfuzz can auto-generate a test harness for, namely, the
Display_Date. However, it can’t auto-generate a starting corpus for
it, as shown by the field corpus_gen_supported.
We then proceed to generate a test harness and try to auto-generate a starting corpus:
$ gnatfuzz generate -P records_test.gpr -S ./src/show_date.ads -L 14 -o generated_test_harness
$ gnatfuzz corpus-gen -P ./generated_test_harness/fuzz_testing/fuzz_test.gpr -o starting_corpus
INFO: * Building corpus creator.....INFO: Done
INFO: * Running GNATfuzz corpus creator...INFO: Done
ERROR: Failed to create a starting corpus - check if the User_Provided_Seeds subprogram needs completing within the User Configuration package
ERROR: ERROR_GENERATING_STARTING_CORPUS
ERROR: For more information about what the cause of this error could be, please see section 3.8 of the GNATDAS User Documentation.
ERROR: The full log can be found at ./generated_test_harness/fuzz_testing/build/gnatfuzz/corpus_gen.log
As we saw, GNATfuzz was able to generate a test harness for the given
subprogram but couldn’t auto-generate a starting corpus. This is because the
current GNATfuzz version doesn’t yet support starting-corpus
auto-generation for subprograms with any “in” or “in out” mode parameters
that are an anonymous type or composed by one. In our example, Date is
composed of Integer range 1 .. 31 and Integer range 1 .. 3000 := 2032
anonymous types. Future versions of GNATfuzz could allow this, but in cases
where automatic corpus generation is not yet supported, GNATfuzz generates
an easy interface to enable you to provide its seeds, which GNATfuzz then
uses to generate the starting corpus required for fuzzing:
generated_test_harness/fuzz_testing/user_configuration/
├── show_date-gnatfuzz_user_configuration.adb
├── show_date-gnatfuzz_user_configuration.ads
└── stop_criteria.xml
The actual package namespace is either a child of GNATfuzz or,
when available, a child of the package holding the subprogram under
test. In this case, the package will be generated as a child package of
Show_Date.
Now we can see the User_Provided_Seeds procedure in
show_date-gnatfuzz_user_configuration.adb:
1 procedure User_Provided_Seeds is
2 begin
3
4 -- Uncomment the next line to avoid the default corpus generation
5 -- GNATfuzz_Configurations.Generate_Default_Corpus := False;
6
7 -- Complete here with calls to Add_Corpus_Seed with each set of seed values.
8 --
9 -- For instance, add two calls
10 -- Add_Corpus_Seed (A, B);
11 -- Add_Corpus_Seed (C, D);
12 -- to generate two seeds, one with input values (A, B) and one with (C, D).
13
14 null;
15 end User_Provided_Seeds;
The above procedure allows turning off the default starting corpus auto-generation by uncommenting line number 5. We can now edit this procedure to add seeds for the subprogram under test, as follows:
1 procedure User_Provided_Seeds is
2 begin
3
4 Add_Corpus_Seed (Fuzz_Input_Param_1 => ((24, Show_Date.February, 1978)));
5 Add_Corpus_Seed (Fuzz_Input_Param_1 => ((29, Show_Date.March, 1978)));
6 Add_Corpus_Seed (Fuzz_Input_Param_1 => ((29, Show_Date.March, 1984)));
7 Add_Corpus_Seed (Fuzz_Input_Param_1 => ((11, Show_Date.August, 1994)));
8
9 end User_Provided_Seeds;
where Fuzz_Input_Param_1 is the parameter of the subprogram under test,
Display_Date (D : Date). Let’s try generating the starting corpus
again:
$ gnatfuzz corpus-gen -P ./generated_test_harness/fuzz_testing/fuzz_test.gpr -o starting_corpus
INFO: * Building corpus creator....INFO: Done
INFO: * Generating corpus...INFO: Done
The generated starting corpus is now located in the designated directory and ready to be used for a fuzz-testing session:
$ ls starting_corpus/
User_1 User_2 User_3 User_4
$ gnatfuzz fuzz -P ./generated_test_harness/fuzz_testing/fuzz_test.gpr --corpus-path ./starting_corpus/ --cores=1 --no-cmplog
3.3.5. Fuzzing instantiations of generic subprograms
For GNATfuzz to test a generic subprogram, we must provide an instantiation.
The GNATfuzz analyze mode can report the instantiation in its generated
analysis file if it consists of a fuzzable subprogram (see section
GNATfuzz automation capabilities and limitations for what is
considered a fuzzable subprogram). When multiple instantiations exist, the
analyze mode provides a list of all those that are fuzzable to allow you to
select the one to be fuzz-tested. When a chain of instantiations exists, the
analyze mode also reports that.
Let’s now try to fuzz the generics example, which has the following
source code:
1 generic
2 type Index_Type is range <>;
3 type Element_Type is private;
4 with function "<" (L, R : Element_Type) return Boolean is <>;
5 package Sorting_Algorithms is
6
7 type Array_To_Sort is array (Index_Type) of Element_Type;
8
9 procedure Selection_Sort (X : in out Array_To_Sort);
10
11 procedure Bubble_Sort (X : in out Array_To_Sort);
12
13 end Sorting_Algorithms;
1 with Sorting_Algorithms;
2
3 generic
4 type Index_Type is range <>;
5 type Element_Type is private;
6 with function "<" (L, R : Element_Type) return Boolean is <>;
7 package Sorting_Multiple_Arrays is
8
9 package Sort_Array is new Sorting_Algorithms (Index_Type, Element_Type);
10
11 type Sortable_Matrix is array (Index_Type) of Sort_Array.Array_To_Sort;
12
13 procedure Selection_Sort (Matrix : in out Sortable_Matrix);
14
15 procedure Bubble_Sort (Matrix : in out Sortable_Matrix);
16
17 end Sorting_Multiple_Arrays;
1 package body Sorting_Multiple_Arrays is
2
3 procedure Selection_Sort (Matrix : in out Sortable_Matrix) is
4 begin
5 for K in Matrix'Range loop
6 Sort_Array.Selection_Sort (Matrix (K));
7 end loop;
8 end Selection_Sort;
9
10 procedure Bubble_Sort (Matrix : in out Sortable_Matrix) is
11 begin
12 for K in Matrix'Range loop
13 Sort_Array.Bubble_Sort (Matrix (K));
14 end loop;
15 end Bubble_Sort;
16
17 end Sorting_Multiple_Arrays;
1 with Sorting_Algorithms;
2 with Sorting_Multiple_Arrays;
3
4 package Instantiations is
5
6 type Range_Array_5 is range 1 .. 5;
7
8 package Sort_Integer_Array_Of_Range_5 is new Sorting_Algorithms
9 (Range_Array_5, Integer);
10
11 package Sort_Integer_Arrays_Of_Range_5 is
12 new Sorting_Multiple_Arrays (Range_Array_5, Integer);
13
14 end Instantiations;
We didn’t display the Sorting_Algorithms.adb file, which implements the
two sorting algorithms, Selection_Sort, and Bubble_Sort. You can find this
file in the example’s source directory.
First, we run the GNATfuzz analyze mode on the project:
$ gnatfuzz analyze -P generics.gpr
This generates an analysis file generated in the
<generics_test-project-obj-dir>/gnatfuzz/analyze.json file, which
contains a table with all the fuzzable subprograms detected:
1 {
2 "user_project": "<install_prefix>/share/examples/gnatfuzz/generic/generics.gpr",
3 "scenario_variables": [],
4 "fuzzable_subprograms": [
5 {
6 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/sorting_algorithms.ads",
7 "start_line": 9,
8 "label": "Selection_Sort",
9 "instantiations": [
10 {
11 "id": 1,
12 "corpus_gen_supported": true,
13 "instantiation_chain": [
14 {
15 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/instantiations.ads",
16 "start_line": 8,
17 "label": "Sort_Integer_Array_Of_Range_5"
18 }
19 ]
20 },
21 {
22 "id": 2,
23 "corpus_gen_supported": true,
24 "instantiation_chain": [
25 {
26 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/instantiations.ads",
27 "start_line": 11,
28 "label": "Sort_Integer_Arrays_Of_Range_5"
29 },
30 {
31 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/sorting_multiple_arrays.ads",
32 "start_line": 9,
33 "label": "Sort_Array"
34 }
35 ]
36 }
37 ]
38 },
39 {
40 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/sorting_algorithms.ads",
41 "start_line": 11,
42 "label": "Bubble_Sort",
43 "instantiations": [
44 {
45 "id": 3,
46 "corpus_gen_supported": true,
47 "instantiation_chain": [
48 {
49 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/instantiations.ads",
50 "start_line": 8,
51 "label": "Sort_Integer_Array_Of_Range_5"
52 }
53 ]
54 },
55 {
56 "id": 4,
57 "corpus_gen_supported": true,
58 "instantiation_chain": [
59 {
60 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/instantiations.ads",
61 "start_line": 11,
62 "label": "Sort_Integer_Arrays_Of_Range_5"
63 },
64 {
65 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/sorting_multiple_arrays.ads",
66 "start_line": 9,
67 "label": "Sort_Array"
68 }
69 ]
70 }
71 ]
72 },
73 {
74 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/sorting_multiple_arrays.ads",
75 "start_line": 13,
76 "label": "Selection_Sort",
77 "instantiations": [
78 {
79 "id": 5,
80 "corpus_gen_supported": true,
81 "instantiation_chain": [
82 {
83 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/instantiations.ads",
84 "start_line": 11,
85 "label": "Sort_Integer_Arrays_Of_Range_5"
86 }
87 ]
88 }
89 ]
90 },
91 {
92 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/sorting_multiple_arrays.ads",
93 "start_line": 15,
94 "label": "Bubble_Sort",
95 "instantiations": [
96 {
97 "id": 6,
98 "corpus_gen_supported": true,
99 "instantiation_chain": [
100 {
101 "source_filename": "<install_prefix>/share/examples/gnatfuzz/generic/src/instantiations.ads",
102 "start_line": 11,
103 "label": "Sort_Integer_Arrays_Of_Range_5"
104 }
105 ]
106 }
107 ]
108 }
109 ]
110 }
Let’s now focus on fuzzing the Selection_Sort generic subprogram from
the Sorting_Algorithms.ads package. In analysis file 3.22 at line 9, we see a list with two possible
instantiations of this generic subprogram, with IDs 1 and 2, respectively.
By inspecting the source code, we can see the following two instantiations:
1. instantiations.ads:Sort_Integer_Array_Of_Range_5 -> sorting_algorithms.ads:Selection_Sort
2. instantiations.ads:Sort_Integer_Arrays_Of_Range_5 -> sorting_multiple_arrays.ads:Sort_Array
-> sorting_algorithms.ads:Selection_Sort
The first is a direct instantiation of the generic Selection_Sort
subprogram and the second is an indirect instantiation of the same generic
via a chain of instantiations. By using the IDs of the instantiations, we
can generate a test-harness for the selected instantiation of the
Selection_Sort and perform a fuzz-testing session. For example:
$ gnatfuzz generate -P generics.gpr --analysis obj/gnatfuzz/analyze.json --subprogram-id 2 -o generated_test_harness
$ gnatfuzz fuzz -P generated_test_harness/fuzz_testing/fuzz_test.gpr
3.3.6. Fuzzing global and private state
The current version of GNATfuzz doesn’t support the automatic generation of
test harnesses that can fuzz global and private states, which can also affect a
subprogram’s execution. Fortunately, there are easy ways to enable the fuzzing
of such a state (to understand if it’s considered fuzzable, see section
GNATfuzz automation capabilities and limitations). In this example,
we demonstrate a simple way of enabling the fuzzing of global and private
variables that affect the execution of a subprogram. Let’s consider the
global_and_private_state example with the following source:
1 package Global_And_Private_State_Fuzzing is
2
3 Operation_Increase_Enabled : Boolean := True;
4
5 function Increase_Or_Decrease_Value
6 (Value : Integer; Amount : Integer) return Integer;
7 -- Based on the "Operation_Increase_Enabled" this function will
8 -- either increase or decrease the given "Value" by the given "Amount",
9 -- and base on the "Absolute_Value_Enabled", it will return either the
10 -- absolute value or not of the selected operation's result
11
12 private
13 Absolute_Value_Enabled : Boolean := False;
14
15 end Global_And_Private_State_Fuzzing;
1 package body Global_And_Private_State_Fuzzing is
2
3 function Increase_Or_Decrease_Value (Value : Integer; Amount : Integer)
4 return Integer
5 is
6 Temp_Value : Integer := 0;
7 begin
8
9 if Operation_Increase_Enabled then
10 Temp_Value := Value + Amount;
11 else
12 Temp_Value := Value - Amount;
13 end if;
14
15 if Absolute_Value_Enabled then
16 return abs Temp_Value;
17 end if;
18
19 return Temp_value;
20 end Increase_Or_Decrease_Value;
21
22 end Global_And_Private_State_Fuzzing;
The operation performed by the Increase_Or_Decrease_Value subprogram is
controlled by the global parameter Operation_Increase_Enabled. The
result of the selected operation is conditionally returned as an absolute
value depending on the Absolute_Value_Enabled private variable. Let’s
try to fuzz this subprogram as is:
$ gnatfuzz analyze -P global_and_private_state.gpr
$ gnatfuzz generate -P global_and_private_state.gpr -S src/global_and_private_state_fuzzing.ads -L 5 -o generated_test_harness/
$ gnatfuzz fuzz -P generated_test_harness/fuzz_testing/fuzz_test.gpr
When we allow the fuzzing session to run for a while, we see that the coverage reaches a stable value and doesn’t increase any further:
Fig. 3.7 Coverage can’t reach 100% due to not fuzzing the global and private state that affect the subprogram’s execution.
We can also examine the coverage report for the source code of the subprogram under test by opening its HTML format in a browser (firefox in this example):
$ firefox generated_test_harness/fuzz_testing/session/coverage_output/global_and_private_state_fuzzing.adb.html
Because the GNATfuzz autogenerated test-harness doesn’t change the
Operation_Increase_Enabled global or the Absolute_Value_Enabled
private variable, we can see from the coverage report in Fig. 3.8 that the code at line 11 and line 15 is
never reached.
Fig. 3.8 Coverage report for the Increase_Or_Decrease_Value subprogram
We can add the following child package to the global_and_private_state
project to enable the fuzzing of the Operation_Increase_Enabled global
and the Absolute_Value_Enabled private variable:
1 package Global_And_Private_State_Fuzzing.Child is
2
3 function Fuzz_Global_And_Private_State
4 (Value : Integer; Amount : Integer; Oper_Increase_Enabled : Boolean;
5 Absolute_Val_Enabled : Boolean) return Integer;
6 -- Wrapper function for the "Increase_Or_Decrease_Value" function
7 -- that takes two extra arguments which will allow the fuzzing of
8 -- the global and private variables that affect the execution of
9 -- the wrapped function.
10
11 end Global_And_Private_State_Fuzzing.Child;
1 package body Global_And_Private_State_Fuzzing.Child is
2
3 function Fuzz_Global_And_Private_State
4 (Value : Integer; Amount : Integer; Oper_Increase_Enabled : Boolean;
5 Absolute_Val_Enabled : Boolean) return Integer
6 is
7 begin
8
9 Operation_Increase_Enabled := Oper_Increase_Enabled;
10 Absolute_Value_Enabled := Absolute_Val_Enabled;
11
12 return Increase_Or_Decrease_Value (Value, Amount);
13
14 end Fuzz_Global_And_Private_State;
15
16 end Global_And_Private_State_Fuzzing.Child;
This child package applies a wrapper function,
Fuzz_Global_And_Private_State, to the function we want to fuzz test.
Creating a child of the package containing the subprogram we want to fuzz
has two benefits: the child can access the global and private state of the
parent and we don’t need to modify our original code. The wrapper has the
same parameters as the wrapped function, plus two extra: one represents the
global and the other the private variable that affect the wrapped
function’s execution state. As we can see in Source Code 3.26 on lines 9 and 10, these
extra parameters are used to assign values to the
Operation_Increase_Enabled global variable and the
Absolute_Value_Enabled private variable, respectively, and thus can
drive their values during fuzz-testing of
Fuzz_Global_And_Private_State. The wrapped function,
Increase_Or_Decrease_Value, is called at line 12. This setup now
allows the fuzz-testing of the Increase_Or_Decrease_Value as before,
but it now also drives the global and local state that affects its
execution.
Let’s now try to fuzz the newly created wrapper function:
$ gnatfuzz analyze -P global_and_private_state.gpr
$ gnatfuzz generate -P global_and_private_state.gpr -S src/global_and_private_state_fuzzing-child.ads -L 3 -o generated_test_harness_2/
$ gnatfuzz fuzz -P generated_test_harness_2/fuzz_testing/fuzz_test.gpr
Fig. 3.9 Coverage reaches 100% when global and private variables that affect the execution of the subprogram under test are being fuzzed
We see that the coverage indeed reached 100% in a short time. By inspecting
the coverage report generated by GNATcoverage on the source code of the
Increase_Or_Decrease_Value subprogram, we can see that GNATfuzz now
executes all source-code lines of the Increase_Or_Decrease_Value
subprogram:
$ firefox generated_test_harness_2/fuzz_testing/session/coverage_output/global_fuzzing.adb.html
Fig. 3.10 Coverage report for the Increase_Or_Decrease_Value subprogram when
fuzzing the private and global variables that affect the subprogram’s
execution
3.3.7. Fuzzing Access Types
Currently, GNATfuzz doesn’t support the auto-generation of a fuzz-test harness
when a subprogram takes as an argument an Access Type of any kind (see section
GNATfuzz automation capabilities and limitations). Fortunately, we
can use the approach used to fuzz-test global and private states (see section
Fuzzing global and private state) to fuzz test some common forms of
Access Types. More specifically, we can do this when the type of the object
that the Access Type is granting access to is fuzzable (see section
GNATfuzz automation capabilities and limitations). We’ll use a
modified version of the example in section Fuzzing global and private state,
namely, the access_types_parameters example, where we update the parameters
of the Increase_Or_Decrease_Value function to be Integer` access types:
1 package Access_Types_Fuzzing is
2
3 Operation_Increase_Enabled : Boolean := True;
4 type Integer_Access is access Integer;
5
6 function Increase_Or_Decrease_Value
7 (Value_Access : Integer_Access; Amount_Access : Integer_Access)
8 return Integer;
9 -- Based on the "Operation_Increase_Enabled" this function will
10 -- either increase or decrease the given "Value" by the given "Amount",
11 -- and base on the "Absolute_Value_Enabled", it will return either the
12 -- absolute value or not of the selected operation's result
13
14 private
15 Absolute_Value_Enabled : Boolean := False;
16
17 end Access_Types_Fuzzing;
1 package body Access_Types_Fuzzing is
2
3 function Increase_Or_Decrease_Value (Value_Access : Integer_Access;
4 Amount_Access : Integer_Access)
5 return Integer
6 is
7 Temp_Value : Integer := 0;
8 begin
9 if Operation_Increase_Enabled then
10 Temp_Value := Value_Access.all + Amount_Access.all;
11 else
12 Temp_Value := Value_Access.all - Amount_Access.all;
13 end if;
14
15 if Absolute_Value_Enabled then
16 return abs Temp_Value;
17 end if;
18
19 return Temp_value;
20 end Increase_Or_Decrease_Value;
21
22 end Access_Types_Fuzzing;
By providing a wrapper function to the Increase_Or_Decrease_Value
function, we enable the auto-generation of a test-harness that allows
fuzz-testing of the wrapped function. This is done similarly to the
approach used for fuzz-testing global and private variables in section
Fuzzing global and private state:
1 package Access_Types_Fuzzing.Child is
2
3 function Fuzz_Access_Type_Parameters
4 (Value : Integer; Amount : Integer; Oper_Increase_Enabled : Boolean;
5 Absolute_Val_Enabled : Boolean) return Integer;
6 -- Wrapper function for the "Increase_Or_Decrease_Value" function
7 -- that takes two extra arguments which will allow the fuzzing of
8 -- the global and private variables that affect the execution of
9 -- the wrapped function.
10
11 end Access_Types_Fuzzing.Child;
1 package body Access_Types_Fuzzing.Child is
2
3 function Fuzz_Access_Type_Parameters
4 (Value : Integer; Amount : Integer; Oper_Increase_Enabled : Boolean;
5 Absolute_Val_Enabled : Boolean) return Integer
6 is
7 Value_Access : constant Integer_Access := new Integer'(Value);
8 Amount_Access : constant Integer_Access := new Integer'(Amount);
9 begin
10
11 Operation_Increase_Enabled := Oper_Increase_Enabled;
12 Absolute_Value_Enabled := Absolute_Val_Enabled;
13
14 return Increase_Or_Decrease_Value (Value_Access, Amount_Access);
15 end Fuzz_Access_Type_Parameters;
16
17 end Access_Types_Fuzzing.Child;
Now, we can proceed with the standard GNATfuzz modes for fuzz-testing the
Fuzz_Access_Type_Parameters wrapper function:
$ gnatfuzz analyze -P access_type_parameter.gpr
$ gnatfuzz generate -P access_type_parameter.gpr -S src/access_types_fuzzing-child.ads -L 3 -o generated_test_harness/
$ gnatfuzz fuzz -P generated_test_harness/fuzz_testing/fuzz_test.gpr
Looking at the coverage report generated, we can confirm that the function
Increase_Or_Decrease_Value was successfully fuzzed:
Fig. 3.11 Coverage report for the Increase_Or_Decrease_Value subprogram when fuzzing the private, global variables, and Access Type parameters that affect the subprogram’s execution
3.3.8. Filtering exceptions
A generated GNATfuzz fuzz-test harness captures all exceptions unhandled by
the system under test and adds the test cases that caused these exceptions
to the crashes directory. In some cases, you might want to filter some
known and expected exceptions (such as an exception raised by design to
notify the calling routine of an error when a library’s API is misused) to
stop them from being captured as software bugs. The GNATfuzz-generated
test-harness provides an easy mechanism to do this.
Warning
You should be careful if you decide to filter out Ada run-time exceptions. Such exceptions, if unhandled, probably indicate software bugs or potential security vulnerabilities.
Let’s look at the filtering_exceptions example that demonstrates how to
tell GNATfuzz not to add a crash to the crashes directory when a specific
exception is raised. The source code of the example is:
1 package Procedure_Under_Test is
2
3 Some_Expected_Exception : exception;
4
5 procedure Test (Some_Text : String);
6
7 end Procedure_Under_Test;
1 with Ada.Text_IO; use Ada.Text_IO;
2
3 package body Procedure_Under_Test is
4
5 procedure Test (Some_Text : String) is
6 begin
7 Put_Line (Some_Text);
8 raise Some_Expected_Exception;
9 end Test;
10
11 end Procedure_Under_Test;
As shown in Source Code 3.32, the
procedure Test raises an exception at line 8. Let’s generate a test
harness for this procedure and use it to filter out this exception:
$ gnatfuzz analyze -P filtering_exceptions.gpr
$ gnatfuzz generate -P filtering_exceptions.gpr -S src/procedure_under_test.ads -L 5 -o generated_test_harness
In the generated_test_harness/fuzz_testing/user_configuration/ directory
of the GNATfuzz autogenerated test-harness, there’s a file called
procedure_under_test-gnatfuzz_user_configuration.adb, with the
following procedure:
1 procedure Execute_Subprogram_Under_Test
2 (GNATfuzz_Some_Text : Standard.String)
3 is
4 begin
5
6 -- Call the subprogram under test. The user is not expected to change
7 -- this code.
8 Procedure_Under_Test.Test
9 (Some_Text => GNATfuzz_Some_Text);
10
11 -- Filter out any expected exception types here.
12 --
13 -- Ada run-time exceptions should not be filtered because they probably
14 -- indicate software bugs and potential security vulnerabilities if
15 -- captured. In such a case, GNATfuzz will place the test case
16 -- responsible for raising the exception under the "crashes" directory.
17 -- If an exception is expected, then an exception handler can be added
18 -- here to capture it and allow GNATfuzz to ignore it.
19
20 -- For example,
21 --
22 -- exception
23 -- when Occurrence : Some_Expected_Exception =>
24 -- null;
25
26 end Execute_Subprogram_Under_Test;
We can modify this in the following manner:
1 procedure Execute_Subprogram_Under_Test
2 (GNATfuzz_Some_Text : Standard.String)
3 is
4 begin
5
6 -- Call the subprogram under test. The user is not expected to change
7 -- this code.
8 Procedure_Under_Test.Test
9 (Some_Text => GNATfuzz_Some_Text);
10
11 exception
12 when Occurrence : Procedure_Under_Test.Some_Expected_Exception =>
13 null;
14
15 end Execute_Subprogram_Under_Test;
This modification allows catching the exception before GNATfuzz will record
it as a crash. So the test which led to the exception being raised won’t be
added to the crashes directory. By providing the following custom stop
criteria, we can confirm the assumption with a fuzz-test session:
1 <GNATfuzz>
2 <afl_stop_criteria>
3 <rule_grouper type="and">
4 <rule>
5 <state>coverage</state>
6 <operator>eq</operator>
7 <value>100</value>
8 <all_sessions>true</all_sessions>
9 </rule>
10 <rule>
11 <state>saved_crashes</state>
12 <operator>eq</operator>
13 <value>0</value>
14 <all_sessions>true</all_sessions>
15 </rule>
16 <rule>
17 <state>run_time</state>
18 <operator>gt</operator>
19 <value>60</value>
20 <all_sessions>true</all_sessions>
21 </rule>
22 </rule_grouper>
23 </afl_stop_criteria>
24 </GNATfuzz>
$ gnatfuzz fuzz -P generated_test_harness/fuzz_testing/fuzz_test.gpr --cores=1 --stop-criteria=stop_criteria.xml
This fuzz-testing session will stop if 100% coverage of the subprogram
under test is achieved and no crashes are found. This means that line 8 in
Source Code 3.32 will be executed, but
the raised exception won’t result in the associated test case being added
to the crashes directory. We also allow the fuzz-testing session to run
for one minute to ensure sufficient time was given to place any potential
crashing test cases in the crashes directory. Our fuzz-testing session
exits with this stopping criteria, meaning the raised exception was
successfully filtered out.
3.3.9. Fuzzing C functions through Ada bindings
Currently, GNATfuzz fuzz-testing automation is not supported for C programs. However, it’s possible to fuzz-test C functions through Ada bindings.
Let’s look at the testing_C_code_through_Ada_bindings example to
demonstrate how to allow GNATfuzz to fuzz-test a C function using an Ada
binding and detect a bug in the C code. The C source code of the example
is:
1 #include "calculations.h"
2
3 // simple function performing some basic math operations
4 // which allows a division by 0
5 int calculations_problematic (int a, int b)
6 {
7 int temp = 0;
8 temp = a * b;
9 temp = temp + a / b;
10 return temp;
11 }
12
13 // same as previous function but prohibits the division
14 // by zero
15 int calculations (int a, int b)
16 {
17 int temp = 0;
18 temp = a * b;
19 if (b != 0)
20 temp = temp + a / b;
21 return temp;
22 }
The two functions perform the same simple calculation but the
calculations_problematic function allows a division by zero. We want to
verify that GNATfuzz is able to detect this bug by fuzz-testing the Ada
binding for this function.
The Ada bindings for the two C functions are:
1 package Procedure_Under_Test is
2
3 function Calculations (A : Integer; B : Integer) return Integer
4 with Import => True,
5 Convention => C,
6 External_Name => "calculations";
7
8 function Calculations_Problematic (A : Integer; B : Integer) return Integer
9 with Import => True,
10 Convention => C,
11 External_Name => "calculations_problematic";
12
13 procedure Test
14 (Control : Integer;
15 In_A : Integer;
16 In_B : Integer);
17
18 end Procedure_Under_Test;
A possible driver procedure to test the two C functions could be:
1 with Ada.Text_IO; use Ada.Text_IO;
2
3 package body Procedure_Under_Test is
4
5 procedure Test
6 (Control : Integer;
7 In_A : Integer;
8 In_B : Integer)
9 is
10 Result : Integer := 0;
11 begin
12
13 if Control < 0 then
14 Result := Calculations_Problematic (In_A, In_B);
15 else
16 Result := Calculations (In_A, In_B);
17 end if;
18
19 Put_Line (Result'Image);
20
21 end Test;
22
23 end Procedure_Under_Test;
Note
By default, when using gprbuild we only compile Ada source files. To compile
C code files as well, we use the Languages attribute in the GPR file and
specify c as an option: for Languages use ("ada", "c");.
By running the GNATfuzz analyze mode:
$ gnatfuzz analyze -P testing_C_code.gpr
GNATfuzz generates the following analysis file:
1 {
2 "user_project": "<install_prefix>/share/examples/gnatfuzz/testing_C_code_through_Ada_bindings/testing_C_code.gpr",
3 "scenario_variables": [],
4 "fuzzable_subprograms": [
5 {
6 "source_filename": "<install_prefix>/share/examples/gnatfuzz/testing_C_code_through_Ada_bindings/src/procedure_under_test.ads",
7 "start_line": 3,
8 "id": 1,
9 "corpus_gen_supported": true
10 },
11 {
12 "source_filename": "<install_prefix>/share/examples/gnatfuzz/testing_C_code_through_Ada_bindings/src/procedure_under_test.ads",
13 "start_line": 8,
14 "id": 2,
15 "corpus_gen_supported": true
16 },
17 {
18 "source_filename": "<install_prefix>/share/examples/gnatfuzz/testing_C_code_through_Ada_bindings/src/procedure_under_test.ads",
19 "start_line": 13,
20 "id": 3,
21 "corpus_gen_supported": true
22 }
23 ]
24 }
Note
As we can see, the Ada bindings for the C functions can also be directly
targeted by GNATfuzz for fuzz-testing. For this example, we are using
the Test procedure as a driver.
We can now use the generate and fuzz mode to fuzz-test our driver function:
$ gnatfuzz generate -P testing_C_code.gpr --analysis obj/gnatfuzz/analyze.json --subprogram-id 3 -o generated_test_harness
$ gnatfuzz fuzz -P generated_test_harness/fuzz_testing/fuzz_test.gpr --cores=1 --stop-criteria=stop_criteria.xml
The stop-criteria specified only allow the fuzz-testing session to exit
when full coverage of the driver function is achieved (implying that both
Ada C bindings are called) and at least one test case causing a crash is found
(which should be the divide-by-zero allowed by the calculations_problematic C function):
1 <GNATfuzz>
2 <afl_stop_criteria>
3 <rule_grouper type="and">
4 <rule>
5 <state>coverage</state>
6 <operator>eq</operator>
7 <value>100</value>
8 <all_sessions>true</all_sessions>
9 </rule>
10 <rule>
11 <state>saved_crashes</state>
12 <operator>gt</operator>
13 <value>0</value>
14 <all_sessions>true</all_sessions>
15 </rule>
16 </rule_grouper>
17 </afl_stop_criteria>
18 </GNATfuzz>
By examining the crashes directory, we can see that GNATfuzz was able
to detect the expected divide-by-zero bug:
$ cd generated_test_harness/fuzz_testing/session/results/
$ ls crashes/
id:000000,sig:06,src:000000,time:10,execs:359,op:flip32,pos:8
id:000001,sig:06,src:000001,time:183,execs:6366,op:int32,pos:4,val:+0
$ ../build/obj-AFL_PERSIST/gnatfuzz-test_harness.verbose /id\:000000\,sig\:06\,src\:000000\,time\:10\,execs\:359\,op\:flip32\,pos\:8
@@@GNATFUZZ_OUTPUT_START@@@
{
"Decoded_In_Parameters": [
{
"Parameter_Name": "Control",
"Parameter_Type": "Standard.Integer",
"Parameter_Value": "-2147483648"
},
{
"Parameter_Name": "In_A",
"Parameter_Type": "Standard.Integer",
"Parameter_Value": "-1"
},
{
"Parameter_Name": "In_B",
"Parameter_Type": "Standard.Integer",
"Parameter_Value": " 0"
}
],
"Subprogram_Under_Test": "Procedure_Under_Test.Test",
"Testcase_Exception": {
"Exception_Information": "raised CONSTRAINT_ERROR : s-intman.adb:134 explicit raise\n",
"Exception_Message": "s-intman.adb:134 explicit raise",
"Exception_Name": "CONSTRAINT_ERROR"
}
}
@@@GNATFUZZ_OUTPUT_END@@@
Note
The test case filenames are not deterministic. The commands shown above are examples that need to be modified with actual filenames generated during the fuzz session.
As we can see, GNATfuzz detected two crashing test cases (note that the names of the cases may vary). By decoding one of the test cases, we can see that the value for the divisor (argument B passed to Calculations_Problematic) is zero while the value of the Control argument ensures the call to the problematic function. We can now use this test case and the gnatfuzz-test_harness.verbose executable to debug and correct the C function. Note that arithmetic overflows are also allowed by both C functions, so more crashes can be detected before we get to the division-by-zero crash.
3.3.10. Resetting dynamically allocated memory when fuzz-testing with the AFL++ persistent mode
In section GNATfuzz and the AFL++ persistent mode, we explained that
persistent mode is only suitable for programs whose state can be completely
reset between multiple test executions.
Let’s now look at the afl_persist_reset_mem example that
demonstrates how to reset the dynamically allocated memory of the
program under test when using the persistent execution mode:
persistent mode (test_afl_persist.ads) 1 package Test_AFL_Persist is
2
3 Procedure Initialize (Width : Integer; Length : Integer);
4
5 private
6
7 type Floor_Plan_Type is record
8 Width : Integer;
9 Length : Integer;
10 Ratio : Integer;
11 end record;
12
13 type Floor_Plan_Access is access all Floor_Plan_Type;
14
15 Floor_Plan : Floor_Plan_Access;
16
17 end Test_AFL_Persist;
persistent mode (test_afl_persist.adb) 1 package body Test_AFL_Persist is
2
3 Procedure Initialize (Width : Integer; Length : Integer)
4 is
5 Initialization_Error : exception;
6 begin
7 if Floor_Plan = null then
8 Floor_Plan := new Floor_Plan_Type;
9 Floor_Plan.Width := Width;
10 Floor_Plan.Length := Length;
11 Floor_Plan.Ratio := Width / Length;
12 else
13 raise Initialization_Error with
14 "Error! Object is already initialized";
15 end if;
16 end Initialize;
17
18 end Test_AFL_Persist;
The system under test initializes a record containing both the dimensions
of a floor and their ratio. The initialization is only supposed to be done
once and an exception is raised otherwise. Line 11 of the Initialize
subprogram is vulnerable to both division-by-zero and overflow. Setting the
following stopping criteria stops the fuzz-testing session when two crashes
are detected:
1 <GNATfuzz>
2 <afl_stop_criteria>
3 <rule_grouper type="single">
4 <rule>
5 <state>saved_crashes</state>
6 <operator>gt</operator>
7 <value>1</value>
8 <all_sessions>true</all_sessions>
9 </rule>
10 </rule_grouper>
11 </afl_stop_criteria>
12 </GNATfuzz>
Let’s run a fuzz-testing session using the AFL++ persist mode by executing
the following GNATfuzz commands:
$ gnatfuzz analyze -P afl_persist_reset_mem.gpr
$ gnatfuzz generate -P afl_persist_reset_mem.gpr -S ./src/test_afl_persist.ads -L 3 -o ./generated_test_harness
$ gnatfuzz fuzz -P generated_test_harness/fuzz_testing/fuzz_test.gpr --afl-mode=afl_persist --cores=2 --stop-criteria=stop_criteria.xml --no-cmin
This results in a failed attempt to launch a fuzz-testing session:
Fig. 3.12 The fuzz-testing session’s dry-run crashes when using AFL++ persistent mode without
resetting the state of the Initialize subprogram
As we can see in Fig. 3.12, AFL++’s dry-run
states that all initial test cases provided to GNATfuzz resulted in
crashes. A dry run is a phase where AFL++ executes the initial set of
test cases multiple times before starting a fuzz-testing session. During
the dry run session, stability is calculated by checking if each run of
the same test results in the same execution path. If the path differs, the
stability drops and a warning is issued in the AFL++ terminal window. For a
dry run to be successful, at least one of the initial test cases must
always run to completion without crashing or timing out. If the dry
run fails, the fuzzing session won’t start.
The Initialize subprogram under test dynamically allocates memory for
the Floor_Plan record only when not already allocated by a previous
call to Initialize. In this example, the first execution of the dry
run evaluates the condition on line 7 as True. Line 8 is then executed and
the Floor_Plan object is created. The second execution evaluates the
condition on line 7 as False (i.e. the Floor_Plan object is no longer
null). This causes the system under test to raise the exception at line 13,
which results in a failed dry run.
As previously discussed, the persistent execution mode retains any program
state while executing tests within the same process. To solve this problem, the
generated fuzz test-harness requires manual intervention to ensure the memory
associated with the Floor_Plan record is freed between each test execution.
GNATfuzz provides a mechanism you can use to reset dynamically allocated memory
before executing each test. To do so, you modify the TearDown function
within the User_Configuration package (see section
The “User_Configuration” package).
In the generated_test_harness/fuzz_testing/user_configuration/ directory of
the GNATfuzz autogenerated test-harness, there’s a file called
test_afl_persist-gnatfuzz_user_configuration.adb, with the following
procedure:
1 procedure Teardown
2 (Width : Standard.Integer;
3 Length : Standard.Integer) is
4 begin
5 -- Complete here with any specialist test deconstruction.
6 --
7 -- For instance, freeing memory.
8 --
9 -- In addition, this subprogram can be used to validate global state
10 -- data or read stored Stub values and check they are as expected.
11 --
12 -- This subprogram can optionally raise exceptions to indicate test
13 -- failure.
14
15 null;
16 end Teardown;
We can modify this as follows:
TearDown procedure used to reset dynamically allocated memory by the system under test 1 procedure Teardown
2 (Width : Standard.Integer;
3 Length : Standard.Integer) is
4
5 procedure Free is new Ada.Unchecked_Deallocation
6 (Floor_Plan_Type, Floor_Plan_Access);
7
8 begin
9
10 Free (Floor_Plan);
11
12 end Teardown;
Note that this modification will require adding the statement with
Ada.Unchecked_Deallocation; at the top of the file. This modification allows
you to free memory that was dynamically allocated for the Floor_Plan record
before GNATfuzz executes the next test. By using the custom stop criteria given
in Fig. 3.44, we can launch a new
fuzz-testing campaign using the AFL++ persistent mode:
$ gnatfuzz fuzz -P generated_test_harness/fuzz_testing/fuzz_test.gpr --cores=1 --afl-mode=afl_persist --stop-criteria=stop_criteria.xml --no-cmin
This time, the fuzz-testing session launches and completes successfully:
Fig. 3.13 The fuzz-testing session runs and completes successfully when using AFL++ persistent mode and
resetting the state of the Initialize subprogram
3.3.11. CMPLOG (aka RedQueen) support
CMPLOG is an AFL++ GCC plugin that supports the fuzzer in achieving greater code coverage by exploiting the input-to-state correspondence relationship that many programs exhibit. The technique leverages the fact that during the execution of a given program, parts of the input often end up directly (i.e., almost unmodified) in the program state (see related publication for more info). CMPLOG extracts such data that can affect the program’s state by dynamically analyzing the code and feeds them to the fuzzer in the form of new test cases that are added to the fuzzer’s queue. This can be particularly useful in the instances of magic bytes and (nested) checksum tests. The following function shows a simplified case where CMPLOG can help enter into tight branches whose conditions are based on magic numbers.
1 function CMPLOG_Test
2 (Control : in Float) return Boolean
3 is
4 begin
5 if Control = 1_234_324.45 then
6 return True;
7 else
8 return False;
9 end if;
10 end CMPLOG_Test;
In the above example, the fuzzer will need to mutate an existing test case and generate a new test
case for the parameter of the CMPLOG_Test function that will include the exact value of
“1_234_324.45” to get into the branch. The probability of such a mutation is extremely low.
CMPLOG, on the other hand, will extract this number by a dynamic analysis stage and try to
inject it into a new test case with a CMPLOG-custom mutation. Such a test case will allow the fuzzer
to enter the branch as an input-to-state correspondence exists.
Note that CMPLOG feature is enabled by default by the GNATfuzz fuzz mode. A CMPLOG fuzzing campaign
runs on a dedicate core. Therefore, in order to run GNATfuzz with CMPLOG at least two cores must be
available (can be set by the --cores GNATfuzz fuzz mode option).
Also, note that CMPLOG is incompatible with AFL modes afl_defer and afl_defer_and_persist.
Therefore, to use CMPLOG, please ensure either AFL mode afl_persist (the default)
or afl_plain are selected via the –afl-mode switch.
3.4. The “User_Configuration” package
As we’ve seen in a previous examples, the GNATfuzz auto-generated
test-harness provides a User_Configuration package (for example, see
Source Code 3.15). The actual package
namespace is either a child of GNATfuzz or, when available, a child of the
package holding the subprogram under test. In the latter case, the
User_Configuration package has access to the global and local state of
the package that contains the subprogram under test.
The User_Configuration package offers several useful subprograms that
you can modify to provide custom functionality for a fuzz-testing session:
Setupfunction: This function executes before each test is executed. You can use it to set up any specialized test initialization, such as setting Stub return values or assigning values to the global state. Also, you can use the return value of this function to indicate whether the setup of a test was successful or that the test should be skipped. In addition, the “in” parameters of the subprogram under test are passed into this function as “in out” parameters, so you can alter their values before running the test. You should be careful to only use this this feature when necessary, but auto-generated test inputs sometimes require modifications in order to make the test valid. For example, if you have a record containing a checksum, any mutation of the data will invalidate the checksum, so you should add code to this function to recalculate it.Teardownprocedure: This procedure is called after each test instance is executed. You can use it to add any specialized finalizational functionality, such as freeing memory and also use it to validate global state data or read stored Stub values and check that they are as expected. TheTeardownsubprogram can also optionally raise exceptions to indicate test failure.User_Provided_Seedsprocedure: You can use this procedure to provide custom seed values, which GNATfuzz uses to create a starting corpus and to control if a default starting corpus should also be autogenerated (see section Fuzzing an autogenerated test-harness with user-provided seeds).Execute_Subprogram_Under_Testprocedure: This is the procedure that calls the subprogram under test. You can use it to filter out any expected exceptions (see section Filtering exceptions).
3.5. GNATfuzz stopping criteria
Through the provided fuzz-test examples, we demonstrated the use of the stopping criteria to provide a predefined set of rules on when a fuzz-testing campaign should be stopped.
The stopping criteria are defined in XML format. If you don’t provide one via the --stop-criteria fuzz-mode switch, GNATfuzz uses a set of default stopping criteria . The generate mode also creates a local copy of this default stopping criteria as an XML file within the autogenerated test harness. You can inspect and modify these default criteria and point to them by specifying the switch:
--stop-criteria=<path-to-generated-test-harness>/fuzz_testing/user_configuration/stop-criteria.xml
1 <GNATfuzz>
2 <afl_stop_criteria>
3 <rule_grouper type="and">
4 <rule>
5 <state>cycles_done</state>
6 <operator>gt</operator>
7 <value>2</value>
8 <all_sessions>true</all_sessions>
9 </rule>
10 <rule>
11 <state>pending_favs</state>
12 <operator>eq</operator>
13 <value>0</value>
14 <all_sessions>true</all_sessions>
15 </rule>
16 <rule>
17 <arithmetic_state>
18 <state1>current_time</state1>
19 <state2>last_find</state2>
20 <arithmetic_operator>minus</arithmetic_operator>
21 </arithmetic_state>
22 <operator>gt</operator>
23 <value>360</value>
24 <all_sessions>true</all_sessions>
25 </rule>
26 </rule_grouper>
27 </afl_stop_criteria>
28 </GNATfuzz>
Fig. 3.48 shows the default stopping criteria generated by GNATfuzz. These default stopping criteria specify that a fuzzing session will only stop if:
All entries within the test queue are processed at least three times, and
There are no “Pending favorites” test cases left (promising test cases for finding new execution paths), and
AFL++ doesn’t find a new path for 6 minutes.
The goal of this set of default stop-criteria rules is to allow sufficient
time for obtaining a good snapshot of what GNATfuzz can achieve when
testing a subprogram. Small subprograms tend to get the maximum possible
code coverage in very little time. Note that the provided stopping criteria
offers no guarantees that GNATfuzz will reach its full potential when
testing a program. They are only intended as a good starting point for you
to modify as needed, e.g., by increasing the time when the last new path is
found to allow the fuzzer to exercise more potential corner cases. To see
all the possible stop-criteria options, you can view the XML relevant
schema file that can be found under
<install_prefix>/share/gnatfuzz/auxiliary/config.xsd.
If you want to avoid using any stopping criteria and manually stop a
fuzzing session using ctrl+c from the terminal, you must pass the
--ignore-stop-criteria fuzz-mode switch.
3.6. GNATfuzz integration in GNAT Studio
Recent versions of GNAT Studio provide an integration with GNATfuzz.
The integration is activated automatically when GNAT Studio finds the
command line executable gnatfuzz on the PATH.
3.6.2. Viewing analysis results and generating a fuzz-testing harness
The results of GNATfuzz in analyze mode, i.e., the subprograms that are identified as automatically fuzzable, are listed in the Locations view. Each entry is decorated with an icon you can click to launch GNATfuzz in generate mode on the corresponding subprogram and generate a fuzz-test harness project.
Fig. 3.14 The Locations view showing the results of gnatfuzz analyze
You can fine-tune the switches passed to the command line in the
intermediate dialog. By default, GNATfuzz will generate the harness project
in a fuzz_testing directory inside the object directory of the root
project.
If generation is successful, GNAT Studio presents a dialog allowing you to load the generated test-harness project.
3.6.3. Running a fuzz session
The first thing you might want to do before running a fuzz session is to
add your seeds to pass to the corpus generator phase. This is done by
editing the subprogram User_Provided_Seeds in
user_configuration/<>-gnatfuzz_user_configuration.adb. You might want
to edit other elements in this package to adjust the test-harness to your
code.
Fig. 3.15 The Project view showing a fuzz-test harness project
When the fuzzing session is running, you’ll see the progress in the
Messages view - the same output you’d see when running gnatfuzz
interactively in the command line.
Fig. 3.16 The Messages view showing the gnatfuzz command line report
GNAT Studio reads the coverage information obtained by gnatfuzz during
the session, displays a summary in the Coverage report, and annotates
source files in the system under test with coverage information, showing
which lines have been successfully explored by the fuzz-testing
session. This information is cumulative and shows the coverage collectively
gained by multiple individual runs of the executable under GNATfuzz.
Fig. 3.17 The source editor annotated with coverage information
3.6.4. Viewing and debugging fuzz crashes
The Fuzz Crashes view shows the crashes and hangs found during the fuzzing session.
Fig. 3.18 The Fuzz Crashes view showing the crashes found by gnatfuzz
Each row in this view corresponds to one individual test case that causes the subprogram under test to crash or hang. You can expand a row to view the parameters to the subprogram being tested that correspond to this crash or hang.
You can also launch a debugging session to place yourself at the crash point: you do this by double-clicking on a row. GNAT Studio launches the fuzzed executable in the debugger and runs it with the seed that caused the hang or crash. The debugger stops on exceptions, so you might have to manually execute past any expected exception that might occur prior to the one causing the crash.
3.7. FAQ for GNATfuzz
Question 1: If my application under test uses Ada tasks, how can I ensure any exceptions raised within task bodies are propagated up to the main test harness?
Answer: Exception propagation in Ada tasking doesn’t happen
automatically. However, the fuzzing campaign may fail to identify software
bugs within Ada tasks unless you ensure that the package
GNATfuzz.Fallback_Handler_Start_Up is included within the main
fuzz-test-harness. To help facilitate this behavior, GNATfuzz will include
the required code as an Ada comment within the generated file
fuzz_testing/generated_src/gnatfuzz-fuzz_test_harness.adb:
1 -- Uncomment the code below if the test application is multi-threaded
2 -- with GNATfuzz.Fallback_Handler_Start_Up;
To enable the task-exception-propagation behavior, uncomment the second line above before running the fuzzing campaign.
Question 2: If my Ada application under tests invokes C/C++ via Ada bindings, will the C/C++ libraries also be included in the fuzzing campaign?
Answer: Yes. However, for the C/C++ libraries to be included in a GNATfuzz campaign, you must dynamically build them via the standard gprbuild mechanism. GNATfuzz can only add the necessary fuzz-test instrumentation to the C/C++ libraries if it has visibility of the source code.
Question 3: What happens if I want to fuzz test aspects of my application that are statically built?
Answer: For your statically built libraries to be included in the control-flow path awareness feature of GNATfuzz, you have to build them using the additional pass implemented by the AFL-GCC-fast compiler plugin. To learn more about how to enable this compiler pass on a statically built library, please contact AdaCore support. Alternatively, you can change the build mechanism for the campaign so that the libraries are built dynamically.
3.8. GNATfuzz diagnostics and issue reporting
GNATfuzz reports any problems or warnings captured during the use of the GNATfuzz modes, whether specified on the command line or by the GNAT Studio IDE. You can find more information on the actions a mode is performing and any potential issues in the auto-generated log files for each mode. The location of these files are:
For the analyze mode:
<obj-dir>/gnatfuzz/analyze.log, where<obj-dir>is the location of the system-under-test project’s object directory.For the generate mode:
<obj-dir>/gnatfuzz/generate.log, where<obj-dir>is the location of the system-under-test project’s object directory.For the corpus-gen mode:
<generated_test_harness>/fuzz_testing/build/gnatfuzz/corpus_gen.log, where<generated-test-harness>is the location specified to the generate mode to create the test harness.For the fuzz mode:
<generated-test-harness>/fuzz_testing/session/fuzz.log, where<generated-test-harness>is the location specified to the generate mode to create the test harness. A separate log file afl-fuzz.<N>.log will also be generated in this directory for each instance of afl-fuzz being launched.
Note that if the GNATfuzz modes are run multiple times targeting the same
subprogram, multiple log files will be generated, with increasing numbering
as part of their naming convention indicating their order of generation,
for example, analyze.log, analyze.log_1, and analyze.log_2
3.8.1. GNATfuzz Error Codes and Exit Statuses
Below you can find a list of all error types that we expect GNATfuzz could raise, along with the corresponding status code that will have been returned when GNATfuzz exited. Further down you will find a short description of the likely causes of each error, and suggestions for steps you can take to address them.
Error Name |
Exit Status Code |
|---|---|
2 |
|
3 |
|
4 |
|
5 |
|
6 |
|
7 |
|
8 |
|
9 |
|
10 |
|
11 |
|
12 |
3.8.1.1. Error_Unknown
This is a catch all exit code for when we don’t know what caused GNATfuzz to crash. Please check the log file for whichever mode you were currently running. For more information about where to find the appropriate log file, please see section GNATfuzz diagnostics and issue reporting.
3.8.1.2. Invalid_Command_Line_Options
This exit code is returned when there was an issue with the command line arguments used. See GNATfuzz “fuzz” mode command line interface for more information on command line arguments.
3.8.1.3. Invalid_GNATfuzz_Configuration_Options
This exit code is returned if there is an issue with your one of either: your stopping criteria file, your environment variables, or your Analyze json results file. Check the log file for whichever mode you were currently running. For more information about where to find the appropriate log file, please see section GNATfuzz diagnostics and issue reporting.
3.8.1.4. Invalid_Fuzzable_Subprogram
This exit code is returned if GNATfuzz fails to compile the user’s code. Check the log file for whichever mode you were currently running. For more information about where to find the appropriate log file, please see section GNATfuzz diagnostics and issue reporting.
3.8.1.5. Error_Initializing_Fuzzing_Campaigns
This exit code is returned when GNATfuzz fails to build some component that is required for the fuzzing campaign. Check the log file for whichever mode you were currently running. For more information about where to find the appropriate log file, please see section GNATfuzz diagnostics and issue reporting.
3.8.1.6. Error_Launching_Fuzzing_Campaigns
This exit code is returned if GNATfuzz encounters an error will attempting
to start your fuzzing campaign. For more information about your issue see the
log which can be found at
<generated-test-harness>/fuzz_testing/session/fuzz.log.
3.8.1.7. Error_During_Fuzzing_Campaigns
This exit code is returned if GNATfuzz encounters an unexpected issue while
running a fuzzing campaign. For more information about your issue see the
log which can be found at
<generated-test-harness>/fuzz_testing/session/fuzz.log.
3.8.1.8. Error_Terminating_Campaigns
This exit code is returned if GNATfuzz finishes a campaign (either because
the stopping criteria were met or because it was interrupted by the user), but
then GNATfuzz encountered and issue while trying to stopping one of the running
tasks. For more information about your issue see the log which can be found at
<generated-test-harness>/fuzz_testing/session/fuzz.log.
3.8.1.9. Error_Initializing_Supporting_Tool
This exit code is returned when GNATfuzz fails to build some component that is required for a supporting tool, for example GNATcov, SymCC, CMPLOG etc. Check the log file for whichever mode you were currently running. For more information about where to find the appropriate log file, please see section If you find that the error was caused by a supporting tool like CMPLOG or SymCC, you can trying disabling the tool as a workaround. See the GNATfuzz command line interface section for more information how how to disable specific supporting tools.
3.8.1.10. Error_Executing_Supporting_Tool
This exit code is returned if GNATfuzz encounters an unexpected issue while
running a supporting tool, for example GNATcov, SymCC, CMPLOG etc. For more
information about your issue see the log which can be found at
<generated-test-harness>/fuzz_testing/session/fuzz.log. If you would
like to disable the tool that is causing the issue, see the
GNATfuzz command line interface section.
3.8.1.11. Error_Generating_Starting_Corpus
This exit code is returned if GNATfuzz encounters an unexpected issue while
generating the starting corpus. Check if the User_Provided_Seeds subprogram
needs completing within the User Configuration package . For more information
about your issue see the log which can be found at
<generated-test-harness>/fuzz_testing/session/fuzz.log. If you would
like to disable the tool that is causing the issue, see the
GNATfuzz command line interface section.
3.9. GNATfuzz Beta feature
This section provides early access to some bleeding-edge GNATfuzz features.
3.9.1. Fuzzing with an arbitrary starting corpus
One way to overcome Ada’s strict enforcement of object creation of some types
is to skip the corpus-gen phase of GNATfuzz and instead use a test
case file containing random data. For this to work, we must place the dummy
file in a standalone directory and pass the path to the directory to gnatfuzz
when executing the fuzz mode via the –corpus-path switch.
Furthermore, it is essential that the –no-cmin switch is used to ensure
the invalid data is allowed to be passed through to the fuzzing engine. GNATfuzz’s
corpus generation must also be disabled through the user’s
configuration package User_Provided_Seeds procedure (see
Source Code 3.16 lines 4 and 5).
When providing GNATfuzz with a random-data test case file as the starting corpus, various GNATfuzz test case generation methods (such as the random AFL++ mutations and SymCC concolic execution) will use this “dummy” test case as a starting point for creating new ones. These new test cases can execute part of the code that has not been reached with any previously provided test cases.
The switch:–no-cmin switch must be used when using this technique. This is because the corpus minimization requires at least one test case in the starting corpus to execute code within the subprogram under test. If the “dummy” test case contains non-valid data, executing it with the subprogram under test will not reach any of the subprogram’s executable code. Therefore, the corpus minimization will provide no test cases for the fuzzer to start. In addition, this feature is not yet supported by CMPLOG (see section CMPLOG (aka RedQueen) support), and therefore, the switch:–no-cmplog must also be used to disable CMPLOG.
Practical experience has shown that a smaller arbitrary test case helps the fuzzer create valid test cases faster. However, any arbitrary file should still work, eventually adding a valid test case to the campaign queue. Please note that this approach’s success will depend on the complexity of the unsupported types. We are further experimenting with this feature to establish its limitations. We are also working on creating better starting corpus strategies for GNATfuzz that use this feature and make it CMPLOG-compatible in future releases.
The following example package contains a type GNATfuzz cannot automatically generate a starting corpus for:
1 package Using_Arbitrary_Test_Cases is
2
3 type My_Private_Type is private;
4
5 procedure A_Subprogram_Not_Supported_For_Corpus_Gen
6 (Param_A : My_Private_Type);
7 -- A subprogram that GNATfuzz cannot automatically generate test cases for
8
9 private
10
11 type Component_A_Type is new Integer;
12
13 type My_Private_Type is record
14 Component_A : Component_A_Type;
15 end record;
16
17 end Using_Arbitrary_Test_Cases;
If we run GNATfuzz analyze on the project:
$ gnatfuzz analyze -P using_arbitrary_test_cases.gpr
The resultant analyze.json will look like this:
1 {
2 "fuzzable_subprograms": [
3 {
4 "corpus_gen_supported": false,
5 "id": 1,
6 "label": "A_Subprogram_Not_Supported_For_Corpus_Gen",
7 "reason": "Corpus gen for <ConcreteTypeDecl [\"Component_A_Type\"] using_arbitrary_test_cases.ads:11:4-11:41> is not supported",
8 "source_filename": "<install_prefix>/share/examples/gnatfuzz/using_arbitrary_test_cases/src/using_arbitrary_test_cases.ads",
9 "start_line": 5,
10 }
11 ]
12 "non_fuzzable_subprograms": [
13 ],
14 "scenario_variables": [
15 ],
16 "user_project": "<install_prefix>/share/examples/gnatfuzz/using_arbitrary_test_cases/using_arbitrary_test_cases.gpr"
17 }
Attempting to generate a corpus for this subprogram will result in an error. Instead, we can use a single arbitrary test case file to start the campaign and utilise the features of GNATfuzz to generate valid test cases.
Create a new directory to hold the arbitrary corpus and place a random test case file in the folder:
$ mkdir corpus
$ echo "Dummy Data" > corpus/dummy_file
Generate a harness for the project:
$ gnatfuzz generate -P using_arbitrary_test_cases.gpr -S ./src/using_arbitrary_test_cases.ads -L 5 -o generated_test_harness
Now launch the fuzzing campaign and pass in the arbitrary corpus whilst remembering to switch off corpus minimization:
$ gnatfuzz fuzz -P ./generated_test_harness/fuzz_testing/fuzz_test.gpr --corpus-path=./corpus --no-cmin --no-cmplog
Note that the fuzzing session starts and generates 100% coverage. By analysing the queue of the primary campaign, we can see that GNATfuzz has generated valid test cases from the original invalid test case.
3.9.2. GNATfuzz and Symbolic Execution
GNATfuzz can leverage SymCC, a symbolic execution tool, to generate test cases that increase coverage in hard-to-reach areas of the code under test.
Symbolic execution is a technique that observes how values are computed in a program,
expressing each as a formula for the inputs. This often enables it to compute back
from a desired value at an arbitrary point in the program to the input required to
produce said value. For example, if an interesting part of a subprogram under test
is only entered when the sum of two of its input parameters has a specific value,
symbolic execution can find values for the input parameters so that their sum matches
the expectation. Both the fuzzer’s mutations and CMPLOG are inherently limited in
generating test cases for such scenarios. This is because the probability of the
fuzzer’s random mutations generating a specific value required to access a tight
branch (e.g., x=1923333) is pretty low, and CMPLOG would not work in cases where
intermediate computation steps break the input-to-state correspondence (see Section
CMPLOG (aka RedQueen) support). This makes symbolic execution an excellent complementary
technology to AFL and CMPLOG, as it helps increase coverage in cases where the other
two technologies cannot help. The following function shows a simplified case where
SymCC can help the fuzzer enter a tight branch whose conditions are derived from
arithmetic algorithms and magic numbers such that there is no input-to-state
correspondence between the subprogram parameters and the c branch-control
variable.
1 package body SymCC_Test is
2
3 function Tight_Branch_Test
4 (X : Positive; Y : Positive; Z : Positive) return Boolean
5 is
6 a : constant Positive := X + Y;
7 b : constant Positive := a - Z;
8 c : constant Positive := a*b;
9 begin
10 if c = 14574000 then
11 return true;
12 else
13 return false;
14 end if;
15 end Tight_Branch_Test;
16
17 end SymCC_Test;
SymCC is an open-source symbolic execution implementation. It uses a compiler plugin to instrument the code under test and provides a run-time library to generate new test cases when the program is executed. For more information on SymCC, see the SymCC seminal publication.
3.9.2.1. Enabling SymCC
To enable SymCC with GNATfuzz, the GNATFUZZ_SYMCC environment variable needs to
be set, and any GPR project dependencies of the code under test must be specified in
the environment variable LLVM_SYMCC_GPR_PROJECT_PATH. The SymCC feature of GNATfuzz
requires configuring the LLVM_SYMCC_GPR_PROJECT_PATH environment variable in the
same way the GPR_PROJECT_PATH environment variable is being configured for the
GNAT Pro gprbuild. When GNATfuzz is executed with SymCC integration
enabled, it runs SymCC on a separate thread to produce interesting test cases. It regularly
synchronizes the sets of test cases used by the fuzzer(s) and by SymCC so that SymCC benefits
from test cases discovered by the fuzzer(s) and vice versa.
Fig. 3.10 shows SymCC’s output directory location where
all the SymCC-generated test cases are placed.
3.9.2.2. Current Limitations
SymCC currently can’t reason about floating-point values. Consequently, it won’t be able to find new test cases for conditions that depend on floating-point inputs or intermediate values. Note that this doesn’t affect its ability to reason about non-floating-point data in the code under test, even if that code also works with floating-point data.
Furthermore, the adaptation of SymCC for Ada can’t change the length of unbounded array type parameters. Note that this is particularly relevant to subprograms handling string data (which is represented in Ada as an unbounded array of Characters): if reaching an interesting portion of the subprogram under test requires an input string to have a different length, then SymCC can’t find such a test case (but the fuzzer(s) may).
Finally, SymCC requires all code in the program under test to be compilable with GNAT-LLVM, the LLVM-based Ada compiler provided by AdaCore, see the GNAT with the LLVM Back End manual. It works with most Ada constructs but currently still lacks support for a few language features.
3.9.3. GNATfuzz Adaptive - “On-target” fuzzing (experimental)
Warning
On-target fuzzing is an experimental and evolving feature of GNATfuzz. Please note that aspects of the feature set are subject to change. In addition, the GNATfuzz Adaptive engine has only been tested on CheriBSD on Morello. More extensive testing on Morello and other targets is expected for future releases.
3.9.3.1. What is it?
The term “On target fuzzing” refers to fuzzing on embedded platforms that may or may not have an operating system. This mode is similar to regular GNATfuzz, except that it involves an executable that should be executed directly on the target machine (or using an emulator).
The workflow is the same, except that GNATfuzz cannot launch the executable built for your target (as normally happens during the GNATfuzz fuzz phase for host-based fuzzing), as this process is specific to the targeted platform. It is also up to the user to push the executable to the targeted hardware or emulator. An example workflow is provided below to help understand the process.
3.9.3.2. How to use?
The analyze step is the same as with regular GNATfuzz. After invoking gnatfuzz analyze,
you should get an analyze.json file reporting all supported subprograms for a
given project.
The generate phase works like in other modes, except that some environment variables must be set to enable the GNATfuzz Adaptive on-target fuzzing mode.
export GNATFUZZ_ADAPTIVE=1
export TGEN_NO_JSON_MARSHALLING=1
export GNATFUZZ_RT_Externally_Built=false
The first variable tells GNATfuzz to generate code for Adaptive fuzzing, and the second removes JSON marshalling as it may not work on some platforms. Finally, the last variable allows rebuilding the GNATfuzz runtime for a specified target.
3.9.3.3. Setup the development environment
The GNAT Dynamic Analysis Suite and a working GNAT Pro cross-compilation toolchain from x86 64-bit Linux to the desired hardware target and operating system specification must be installed. See the GNAT cross platforms user guide for more information.
Note
For this early release of GNATfuzz Adaptive on-target fuzzing, the GNAT Pro Full Ada runtime is required as the runtime requires access to a file system to generate results files. Future releases will allow fuzzing on bareboard targets via the GNAT Pro Ada Embedded runtime.
In addition, the gnatcov runtime must be built and installed for the target. See the gnatcov user guide for more information.
3.9.3.4. Configure the fuzzer
The following environment variables can be used to configure the engine:
Variable Name |
Possible values |
Description |
Default values |
|---|---|---|---|
GNATFUZZ_ADAPTIVE |
Set to enable |
Switch the GNATfuzz generator in adaptive fuzzing mode. |
Not set |
GNATFUZZ_ADAPTIVE_ENGINE_SEED |
Integer value |
Manually set a custom value to seed the random number generator within the GNATfuzz Adaptive mutation engine. If set, the mutations are deterministic. |
Current time |
GNATFUZZ_ADAPTIVE_TIMEOUT |
Integer value |
Maximum execution time allowed for the engine to detect a hang. |
60s |
GNATFUZZ_ADAPTIVE_COVERAGE |
Set to enable |
Enable coverage-guided fuzzing. |
Not set |
GNATFUZZ_ADAPTIVE_MAX_CRASH |
Positive integer value |
Number or crashes to stop a fuzzing campaign. |
50 crashes |
3.9.3.5. Understanding the results
Each crash and hang is saved in its respective directory. Each saved test case can be decoded using the decode tests executable Decode_Test_Build_Command.
3.9.3.6. Example usage
The following example will explore fuzzing a simple Ada program on the Arm Morello development board using CheriBSD. Arm Morello offers a comprehensive set of hardware based bug finding capabilities via fine grain compartmentalization and makes for an interesting fuzzing target! For more information about Arm Morello and CHERI in general see: CHERI-Morello
Let’s pretend that we want to fuzz the following subprogram on a morello-based board using CheriBSD.
package Pkg is
function Compute_Stuff (X : Integer) return Integer;
end Pkg;
First, we need to switch GNATfuzz to Adaptive fuzzing mode and analyze the project:
$ export GNATFUZZ_ADAPTIVE=1
$ export GNATFUZZ_ADAPTIVE_COVERAGE=1
$ export TGEN_NO_JSON_MARSHALLING=1
$ gnatfuzz analyze -Pbuild.gpr
The snippet above sets gnatfuzz to use the Adaptive engine
(GNATFUZZ_ADAPTIVE), with the coverage feature (GNATFUZZ_ADAPTIVE_COVERAGE).
The last tells the test generator not to try and write new test-cases to the
GNATtest TGen JSON files, as this feature isn’t supported on some platforms
(CheriBSD being one of them).
Now we can use the generate subcommand to create a fuzz harness. Since our
project contains only one fuzzable subprogram, its id will be 1.
$ gnatfuzz generate \
-P build.gpr \
--analysis obj/gnatfuzz/analyze.json \
--subprogram-id 1
All necessary code is now generated under obj/gnatfuzz/harness/. At this
point, the generated harness needs to be compiled for CheriBSD on morello.
Before building the harness, the generated code must be instrumented using GNATcoverage to make coverage-based fuzzing work.
$ gnatcov instrument \
-P ./obj/gnatfuzz/harness/fuzz_testing/adaptive_engine.gpr \
--level=stmt \
--target=morello-freebsd,15
Note
It is possible to instrument the code for different coverage level and even combine the different coverage methods. It may improve fuzzing results on some codebases. We recommend testing different combinations of levels and picking the one that works best for a use case.
Once instrumented, the harness can be built using the following command.
Note that --implicit-with=gnatcov_rts.gpr and --src-subdirs=gnatcov-instr
options should be omitted if the coverage feature isn’t used.
$ gprbuild \
--target=morello-freebsd,15 \
-P ./obj/gnatfuzz/harness/fuzz_testing/adaptive_engine.gpr \
-j$(nproc --all)
The resulting artifacts will be located under
./obj/gnatfuzz/harness/fuzz_testing/build/gnatfuzz-test_harness.adaptive.
This executable can be sent to either a physical Arm Morello running
CheriBSD or to an emulated version via GNAT Emulator.
While running, the fuzzer will save crashes and hangs in their respective
directories. These test cases can be decoded using the gnatfuzz-decode_test
executable.
Directory |
Description |
|---|---|
|
Contains test cases that caused an unhandled exception |
|
Contains test cases that caused a hang |
|
Contains test cases that increased code coverage |
GNATfuzz test cases are binary files that match the target hardware’s in-memory representation. As such, they are susceptible to endianness and are not cross-target compatible. Therefore, binary test cases should be decoded on-target using the generated test case decoder.
Future releases of GNATfuzz Adaptive fuzzing will generate test cases that are target hardware agnostic, such that found errors and hangs can be reproduced on a host-based system. However, for this early release, the test cases can only be debugged on-target or manually reproduced on the host by inspecting the ASCII representation of the test case generated via the decoder.
The test case decoder can be built with the following command:
$ gprbuild \
--target=morello-freebsd,15 \
-P ./obj/gnatfuzz/harness/fuzz_testing/adaptive_engine.gpr \
-XGNATFUZZ_INTERNAL_MODE=DECODE_TEST \
-j$(nproc --all)
The resulting executable should be present under
./obj/gnatfuzz/harness/fuzz_testing/build/gnatfuzz-decode_test.
3.9.3.7. Known limitations
This is a list of known limitations. Some limitations may be addressed in upcoming releases.
The fuzzer can not recover from a hang. When a hang is detected, the engine saves the corresponding test case and exits with code
2.Crashes found by the fuzzer may be duplicated.
Due to differences in some target hardware in-memory representations (i.e. endianness) the binary test cases in the results folders may not be compatible with a GNATfuzz host-based test harness. The workaround is to use the generated decoder to convert the test case to an ASCII representation.
The full Ada runtime is required, future releases will allow bareboard fuzzing using the Embedded Ada runtime.
Corpus generation must be supported for the given subprogram. The analyze.json contains this information (via the corpus-gen-supported attribute).