4. Using GNATtest with GNATcoverage
Using GNATcoverage to obtain coverage information on the project being tested is the next logical step in the GNATtest workflow. There are two main ways to accomplish this: using the facilities generated by GNATtest (namely a Makefile) during the harness generation, or deriving a custom solution, while keeping a few important concepts in mind.
4.1. Using the Makefile generated by GNATtest
GNATtest will always generate a Makefile to automate as much as possible the production of a coverage report on the project being tested.
This Makefile is generated in the same directory as the harness project, and consists of two files:
The main
Makefile
which is regenerated for each invocation ofgnattest
The
coverage_settings.mk
secondary makefile, which holds most of the options that will be passed to gnatcoverage. The values present when the file is generated are extracted from the root project file that was passed to GNATtest. This file will not be re-generated by subsequent invocation ofgnattest
, so it is a suitable place to store settings for GNATcoverage if they are not all defined in the project file.
The makefile also assumes that the GNATcoverage runtime is already built and installed,
and that the path to its project file is in the environment variable
GPR_PROJECT_PATH
so it can be automaticaly located by GPRbuild.
It is also possible to specify the path to the project file in the
GNATCOV_RTS
variable in the coverage_settings.mk
file. See
Setting up the coverage runtime library for instructions on building and installing the runtime.
If all the switches that need to be passed to GNATcoverage are correctly defined in
coverage_Settings.mk
, then running:
make coverage
will build and run all the harness projects and then generate a coverage report for all the units under test.
By default, coverage data for a native project is obtained using source traces and coverage for projects targeting a cross environment is assessed using binary traces.
For cases where the default behavior isn’t appropriate for the project under test, the rules for producing a coverage report from both kind of traces are always generated.
4.1.1. Special considerations for cross environment targets
Using GNATcoverage to assess coverage for a project with a cross-environment target may require some special adaptation to be able to run the program and collect execution traces. The generated makefile tries to implement an execution and trace collection strategy that is the most likely to work for most projects, based on GNATemulator for execution.
For binary traces, this is done through the gnatcov run
command (see
gnatcov run command line), which takes care of setting up the trace
collection mechanism in GNATemulator. For this configuration to work properly,
the Board
attribute in the package Emulator
needs to be specified in
the root project file, or alternatively the GNATEMU_BOARD
variable must be
set in the coverage_settings.mk
secondary makefile.
For source traces, the instrumented program will be run using GNATemuator
directly, and assumes that the standard package Ada.GNAT_IO allows data to be
output on a serial port. The instrumentation process overrides the two switches
--dump-trigger
and --dump-channel
to the values
main-end
and base64-stdout
respectively as they are the most adapted to
the GNATtest harness project and to the GNATemulator execution environment.
The program is run under GNATemulator, and the output of the first serial port
is captured as it contains the trace in a base-64 encoding. This text trace is
then decoded and converted into a regular source trace.
There is currently no out-of-the-box support for running the GNATtest harness projects on target and collect execution traces from a hardware probe; the current makefile can still be used to compile the executables and process the traces once they are available, but the execution rules will need rewriting.
4.1.2. Usage example
In this example the project has two units, Pkg1
and Pkg2
. The code in
Pkg2
calls all the subprograms in Pkg1
, and we wish to unit test each
unit in isolation. To do so, we will use the Individual Test Drivers
feature of GNATtest so each unit under test gets its own test executable, and
we’ll be able to create each time a checkpoint containing coverage for the
unit being tested. See Handle incidental coverage effects on how checkpoints help
avoid incidental coverage.
The project is structured as follows:
<project root directory/>
|prj.gpr
|src/
|pkg1.ads
|pkg2.ads
...
|tests/
|<gnattest generated test skeletons>
|harness/
|Makefile
|coverage_settings.mk
|<GNATtest generated harness files>
and is defined with the following project file:
-- prj.gpr
project Prj is
for Source_Dirs use ("src");
for Object_Dir use "obj";
package Coverage is
Cov_Level := ("--level=stmt");
for Switches ("instrument") use ("--dump-trigger=atexit") & Cov_Level;
for Switches ("run") use Cov_Level;
for Switches ("coverage") use ("--annotate=report") & Cov_Level;
end Coverage;
package Gnattest is
for Tests_Dir use "../tests";
for Harness_Dir use "../harness";
end Gnattest;
end Prj;
Specifying the coverage preferences in the project file spares us modifying the
coverage_settings.mk
after generation.
The tests will be stored in the tests
directory, as specified by the project
attribute Tests_Dir
, and the GNATtest generated harness in the harness
directory.
After invoking GNATtest as follows:
gnattest -P prj.gpr --separate-drivers=unit
The following two files can be found in the harness directory:
The
Makefile
with (amongst other things) three rules that execute the full coverage workflow for each test driver project, and then generates a coverage report combining the results:bin-coverage: ... inst-coverage: ... coverage: inst-coverage
The first rule (
bin-coverage
) runs the binary traces workflow for GNATcoverage, whereas the second rule (inst-coverage
) runs the source-trace (or instrumentation based) workflow. The last rule (coverage
) is defined to use the workflow the most likely to work given the current target.The
coverage_settings.mk
file, which, when generated, copied all the values of the relevant root project attributes into corresponding variables:# Settings in this file were extracted from the source project # or are gnattest default values if they weren't specified in the source # project. They may need adjustments to fit your particular coverage needs. # This file won't be overwritten when regenerating the harness. # Switches for the various gnatcov commands SWITCHES_INSTRUMENT=--dump-trigger=main-end --level=stmt SWITCHES_RUN=--level=stmt SWITCHES_COVERAGE=--annotate=report --level=stmt # Path to the installed gnatcov rts project file. # No need to specify it if the project file path was added to the # GPR_PROJECT_PATH environment variable. GNATCOV_RTS=
There is an empty
GNATCOV_RTS
variable defined, which we can set to the path to the installed gnatcov runtime project file.
Once the tests are all written, generating the coverage report (on the standard output in this example) can be done by simply invoking:
make -C harness/ coverage
This outputs the sequence of commands issued to perform the coverage computation, then the results if the “report” format is selected. On our example, this would be like:
Instrumenting project Pkg1.Test_Data.Tests/test_driver.gpr:
gnatcov instrument -PPkg1.Test_Data.Tests/test_driver.gpr --dump-trigger=main-end --level=stmt --projects=Prj --units=@Pkg1.Test_Data.Tests/units.list
Building Pkg1.Test_Data.Tests/test_driver.gpr:
gprbuild -PPkg1.Test_Data.Tests/test_driver.gpr -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts
...
Running Pkg1.Test_Data.Tests/test_driver.gpr:
GNATCOV_TRACE_FILE=Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace Pkg1.Test_Data.Tests/test_driver
pkg1.ads:2:4: info: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:44)
pkg1.ads:3:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:65)
2 tests run: 0 passed; 2 failed; 0 crashed.
Creating checkpoint for Pkg1.Test_Data.Tests/test_driver.gpr:
gnatcov coverage --save-checkpoint=Pkg1.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg1.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg1.Test_Data.Tests/units.list
Instrumenting project Pkg2.Test_Data.Tests/test_driver.gpr:
gnatcov instrument -PPkg2.Test_Data.Tests/test_driver.gpr --dump-trigger=main-end --level=stmt --projects=Prj --units=@Pkg2.Test_Data.Tests/units.list
Building Pkg2.Test_Data.Tests/test_driver.gpr:
gprbuild -PPkg2.Test_Data.Tests/test_driver.gpr -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts
...
Running Pkg2.Test_Data.Tests/test_driver.gpr:
GNATCOV_TRACE_FILE=Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace Pkg2.Test_Data.Tests/test_driver
pkg2.ads:2:4: info: corresponding test PASSED
pkg2.ads:3:4: info: corresponding test PASSED
2 tests run: 2 passed; 0 failed; 0 crashed.
Creating checkpoint for Pkg2.Test_Data.Tests/test_driver.gpr:
gnatcov coverage --save-checkpoint=Pkg2.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg2.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg2.Test_Data.Tests/units.list
Creating coverage report:
gnatcov coverage -P../prj.gpr -CPkg1.Test_Data.Tests/test_driver-gnattest.ckpt -CPkg2.Test_Data.Tests/test_driver-gnattest.ckpt --annotate=report --level=stmt
** COVERAGE REPORT **
===========================
== 1. ASSESSMENT CONTEXT ==
===========================
Date and time of execution: 2021-08-23 15:35:58 +02:00
Tool version: XCOV development-tree
Command line:
gnatcov coverage -P../prj.gpr -CPkg1.Test_Data.Tests/test_driver-gnattest.ckpt -CPkg2.Test_Data.Tests/test_driver-gnattest.ckpt --annotate=report --level=stmt
Coverage level: stmt
Trace files:
Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace
kind : source
program : Pkg1.Test_Data.Tests/test_driver
date : 2021-08-23 15:35:56 +02:00
tag :
processed: gnatcov coverage --save-checkpoint=Pkg1.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg1.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg1.Test_Data.Tests/units.list @ 2021-08-23 15:35:56 +02:00
Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace
kind : source
program : Pkg2.Test_Data.Tests/test_driver
date : 2021-08-23 15:35:57 +02:00
tag :
processed: gnatcov coverage --save-checkpoint=Pkg2.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg2.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg2.Test_Data.Tests/units.list @ 2021-08-23 15:35:58 +02:00
============================
== 2. COVERAGE VIOLATIONS ==
============================
2.1. STMT COVERAGE
------------------
pkg1.adb:13:7: statement not executed
pkg1.adb:18:7: statement not executed
2 violation.
=========================
== 3. ANALYSIS SUMMARY ==
=========================
2 STMT violation.
** END OF REPORT **
The log shows all the steps necessary to obtain coverage results from the mutliple test drivers, and end with the report.
From the coverage report, we see that the only lines not covered are in
pkg1.adb
, which is expected as the tests corresponding to that unit are
not implemented. By using separate drivers, although the code in Pkg2
uses
the subprograms defined in Pkg1
, we were able to not have the coverage
results from the unit tests on Pkg1
be polluted by the tests on Pkg2
.
Regenerating the harness to use a single monolithic driver, and re-generating a
coverage report shows that without the separate drivers, Pkg1
is marked as
covered despite not having any test implemented:
gnattest -P prj.gpr && make -C harness/ coverage
which outputs:
Instrumenting project test_driver.gpr:
gnatcov instrument -Ptest_driver.gpr --dump-trigger=main-end --level=stmt --projects=Prj
Building test_driver.gpr:
gprbuild -Ptest_driver.gpr -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts
...
Running test_driver.gpr:
GNATCOV_TRACE_FILE=test_driver-gnattest_td.srctrace ./test_driver
pkg2.ads:2:4: info: corresponding test PASSED
pkg2.ads:3:4: info: corresponding test PASSED
pkg1.ads:2:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:44)
pkg1.ads:3:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:65)
4 tests run: 2 passed; 2 failed; 0 crashed.
Creating checkpoint for test_driver.gpr:
gnatcov coverage --save-checkpoint=test_driver-gnattest.ckpt -Ptest_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj test_driver-gnattest_td.srctrace
Creating coverage report:
gnatcov coverage -P../prj.gpr -Ctest_driver-gnattest.ckpt --annotate=report --level=stmt
** COVERAGE REPORT **
===========================
== 1. ASSESSMENT CONTEXT ==
===========================
Date and time of execution: 2021-08-23 15:52:36 +02:00
Tool version: XCOV development-tree
Command line:
gnatcov coverage -P../prj.gpr -Ctest_driver-gnattest.ckpt --annotate=report --level=stmt
Coverage level: stmt
Trace files:
test_driver-gnattest_td.srctrace
kind : source
program : ./test_driver
date : 2021-08-23 15:52:35 +02:00
tag :
processed: gnatcov coverage --save-checkpoint=test_driver-gnattest.ckpt -Ptest_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj test_driver-gnattest_td.srctrace @ 2021-08-23 15:52:35 +02:00
============================
== 2. COVERAGE VIOLATIONS ==
============================
2.1. STMT COVERAGE
------------------
No violation.
=========================
== 3. ANALYSIS SUMMARY ==
=========================
No STMT violation.
** END OF REPORT **
4.1.3. Instrumenting test harnesses for a SPARK project
General information about SPARK code instrumentation can be found in section Instrumentation and coverage of SPARK code. The compilation of instrumented user code needs to be controlled by a configuration pragma file. When using GNATtest, the main project is the generated test driver project, not the original user code project. As such, the configuration pragma files that need to be passed during compilation cannot be specified in the project under test.
There are two possibilities to specify the configuration pragma file to be
used when building the instrumented harness projects. The first one is to
modify the gnattest_common.gpr
project file (which is not overwritten when
the harness is regenerated), as in:
--- harness/gnattest_common.gpr
+++ harness/gnattest_common.gpr
type TD_Compilation_Type is ("contract-checks","no-contract-checks", "no-config-file");
TD_Compilation : TD_Compilation_Type := external ("TEST_DRIVER_BUILD_MODE", "no-config-file");
package Builder is
case TD_Compilation is
when "contract-checks" =>
for Global_Configuration_Pragmas use "suppress.adc";
when "no-contract-checks" =>
for Global_Configuration_Pragmas use "suppress_no_ghost.adc";
when "no-config-file" =>
- null;
+ for Global_Configuration_Pragmas use "<path to file>/instrument-spark.adc";;
end case;
end Builder;
An alternative solution is to specify the configuration pragma file on the command line when invoking the Makefile:
make BUILDERFLAGS='-cargs:Ada -gnatec=instrument-spark.adc' coverage
4.2. Developing a custom solution
If the generated makefile is not suitable to use with the execution environment, there are a few things to keep in mind in order not to have unexpected coverage results.
4.2.1. Single test driver
In the case where a monolithic test driver is generated by GNATtest, obtaining coverage results for your project is relatively simple, and the only aspect which needs attention is the specification of units of interest, particularly in the case of using source traces.
When using source traces, GNATcoverage needs to instrument the main so that execution
traces are dumped at the end of the test run. So despite none of the units in
the harness project being of interest, is is important that the root project
passed to all gnatcov commands is test_driver.gpr
.
The projects generated by GNATtest all specify that none of the units are of interest, so none of the units generated by GNATtest should appear in the reports.
4.2.2. Separate test drivers
Using separate test drivers is advisable to avoid incidental coverage of one
unit from the testing of other units (see Handle incidental coverage effects). Note
that since the smallest division of a project supported by GNATcoverage is the unit,
there is no benefit in specifying --separate-drivers=test
instead
of --separate-drivers=unit
to GNATtest, as far as incidental
coverage is concerned.
When using mutliple drivers, there will be a test_driver.gpr
generated for
each unit. For each generated driver, the project needs to be instrumented
(if source traces are used), built, run and a checkpoint
must be created from the execution trace. Then a call to gnatcov coverage merges the
coverage data from all the checkpoints and generates the desired report.
The key point in the process is to specify, when creating all the individual
checkpoints, which unit is being tested, so that the checkpoint only records
the coverage information about that unit, and discard any incidental coverage
on other units. During harness generation, a file named units.list
will be
created in the same directory as each test_driver.gpr
file. This file
contains the name of the unit tested by the driver, and can be used to specify
to gnatcov to only process the unit under test, by adding the switch
--units=@units.list
.
5. Using GNATtest with GNATfuzz (experimental)
This section presents how a pre-existing GNATtest test harness can be used as a starting corpus for a GNATfuzz fuzzing campaign, and how inputs of interest found by GNATfuzz can be imported back into a GNATtest harness. These features are still experimental; the workflow and command line interface may change in the future.
5.1. Setting up the environment
To ensure the entire workflow functions properly, it’s crucial to configure the various tools by setting specific environment variables.
The first step is to setup the value generation runtime library. For detailed instructions on how to do so, see Setting up the test generation runtime.
5.2. Using GNATStudio to perform the integration
The simplest way to sequence the invocations of the two tools is to use the GNATstudio integration plugin.
First, create a GNATtest harness project if it doesn’t already exist, using the
Analyze -> GNATtest -> Generate harness project
entries in the menu bar.
Note that in the dialog box opening there is an option to generate tests inputs
if needed.
Then, exporting tests inputs from GNATtest to GNATfuzz and running a fuzzing
campaign on a specific subprogram can be done by right-clicking on the
declaration of the target subprogram, then selecting
GNATtest -> Start/Stop fuzzing subprogram
, as illustrated bellow.
This will first instrument sources and re-generate the GNATtest harness in order to be able to intercept the inputs passed to the subprogram, then run the test harness, dumping the inputs in a binary format that can be used by GNATfuzz. GNATstudio will then setup a fuzzing session on the subprogram, for which the parameters can be controlled through the various pop-up windows that will be displayed during the process.
gnatfuzz
will periodically export newly found inputs in a human readable
JSON format under <obj>/gnattest/test/JSON_Tests
, where <obj>
designates
the object directory of the project.
The fuzzing session will stop once all the stopping criteria have been met. The
fuzzing session can also be stopped early by right clicking on the subprogram
declaration, then selecting GNATtest => Start/Stop fuzzing subprogram
.
After the fuzzing sessions has ended, a new GNATtest harness will be
regenerated, including the tests exported by the GNATfuzz session. These will
appear in
<obj>/gnattest/tests/<unit_name>-test_data-test_<subp_name>_<subp_hash>.adb
,
where <unit_name>
is the name of the unit in which the subprogram is
declared, <subp_name>
is the name of the subprogram, and <subp_hash> is a
hash based on the profile of the subprogram, in order to differentiate
overloads.