3. Using GNATtest with GNATcoverage¶
Using GNATcoverage to obtain coverage information on the project being tested is the next logical step in the GNATtest workflow. There are two main ways to accomplish this: using the facilities generated by GNATtest (namely a Makefile) during the harness generation, or deriving a custom solution, while keeping a few important concepts in mind.
3.1. Using the Makefile generated by GNATtest¶
GNATtest will always generate a Makefile to automate as much as possible the production of a coverage report on the project being tested.
This Makefile is generated in the same directory as the harness project, and consists of two files:
The main
Makefilewhich is regenerated for each invocation ofgnattestThe
coverage_settings.mksecondary makefile, which holds most of the options that will be passed to gnatcoverage. The values present when the file is generated are extracted from the root project file that was passed to GNATtest. This file will not be re-generated by subsequent invocation ofgnattest, so it is a suitable place to store settings for GNATcoverage if they are not all defined in the project file.
The makefile also assumes that the GNATcoverage runtime is already built and installed,
and that the path to its project file is in the environment variable
GPR_PROJECT_PATH so it can be automaticaly located by GPRbuild.
It is also possible to specify the path to the project file in the
GNATCOV_RTS variable in the coverage_settings.mk file. See
Setting up the coverage runtime library for instructions on building and installing the runtime.
If all the switches that need to be passed to GNATcoverage are correctly defined in
coverage_Settings.mk, then running:
make coverage
will build and run all the harness projects and then generate a coverage report for all the units under test.
By default, the trace kind (binary or source, see Selecting a trace mode; supported features and environments) used to produce the coverage report will be chosen based on the target of the root project. Coverage data for a native project will be obtained using source traces, and coverage for projects targeting a cross environment will be assessed using binary traces.
For cases where the default behavior isn’t appropriate for the project under test, the rules for producing a coverage report from both kind of traces are always generated.
3.1.1. Special considerations for cross environment targets¶
Using GNATcoverage to assess coverage for a project with a cross-environment target may require some special adaptation to be able to run the program and collect execution traces. The generated makefile tries to implement an execution and trace collection strategy that is the most likely to work for most projects, based on GNATemulator for execution.
For binary traces, this is done through the gnatcov run command (see
gnatcov run command line), which takes care of setting up the trace
collection mechanism in GNATemulator. For this configuration to work properly,
the Board attribute in the package Emulator needs to be specified in
the root project file, or alternatively the GNATEMU_BOARD variable must be
set in the coverage_settings.mk secondary makefile.
For source traces, the instrumented program will be run using GNATemuator
directly, and assumes that the standard package Ada.GNAT_IO allows data to be
output on a serial port. The instrumentation process overrides the two
switches --dump-trigger and --dump-channel to the values
main-end and base64-stdout respectively as they are the most adapted
to the GNATtest harness project and to the GNATemulator execution environment.
The program is run under GNATemulator, and the output of the first serial port
is captured as it contains the trace in a base-64 encoding. This text trace is
then decoded and converted into a regular source trace.
There is currently no out-of-the-box support for running the GNATtest harness projects on target and collect execution traces from a hardware probe; the current makefile can still be used to compile the executables and process the traces once they are available, but the execution rules will need rewriting.
3.1.2. Usage example¶
In this example the project has two units, Pkg1 and Pkg2. The code in
Pkg2 calls all the subprograms in Pkg1, and we wish to unit test each
unit in isolation. To do so, we will use the Individual Test Drivers
feature of GNATtest so each unit under test gets its own test executable, and
we’ll be able to create each time a checkpoint containing coverage for the
unit being tested. See Handle incidental coverage effects on how checkpoints help
avoid incidental coverage.
The project is structured as follows:
<project root directory/>
|prj.gpr
|src/
|pkg1.ads
|pkg2.ads
...
|tests/
|<gnattest generated test skeletons>
|harness/
|Makefile
|coverage_settings.mk
|<GNATtest generated harness files>
and is defined with the following project file:
-- prj.gpr
project Prj is
for Source_Dirs use ("src");
for Object_Dir use "obj";
package Coverage is
Cov_Level := ("--level=stmt");
for Switches ("instrument") use ("--dump-trigger=atexit") & Cov_Level;
for Switches ("run") use Cov_Level;
for Switches ("coverage") use ("--annotate=report") & Cov_Level;
end Coverage;
package Gnattest is
for Tests_Dir use "../tests";
for Harness_Dir use "../harness";
end Gnattest;
end Prj;
Specifying the coverage preferences in the project file spares us modifying the
coverage_settings.mk after generation.
The tests will be stored in the tests directory, as specified by the project
attribute Tests_Dir, and the GNATtest generated harness in the harness
directory.
After invoking GNATtest as follows:
gnattest -P prj.gpr --separate-drivers=unit
The following two files can be found in the harness directory:
The
Makefilewith (amond other things) three rules that execute the
full coverage workflow for each test driver project, and then generates a coverage report combining the results:
bin-coverage: ...
inst-coverage: ...
coverage: inst-coverage
The first rule (bin-coverage) runs the binary traces workflow for GNATcoverage,
whereas the second rule (inst-coverage) runs the source-trace (or
instrumentation based) workflow. The last rule (coverage) is defined to
use the workflow the most likely to work given the current target.
The
coverage_settings.mkfile, which, when generated, copied all the values of the relevant root project attributes into corresponding variables:
# Settings in this file were extracted from the source project
# or are gnattest default values if they weren't specified in the source project.
# They may need adjustments to fit your particular coverage needs.
# This file won't be overwritten when regenerating the harness.
# Switches for the various gnatcov commands
SWITCHES_INSTRUMENT=--dump-trigger=main-end --level=stmt
SWITCHES_RUN=--level=stmt
SWITCHES_COVERAGE=--annotate=report --level=stmt
# Path to the installed gnatcov rts project file.
# No need to specify it if the project file path was added to the GPR_PROJECT_PATH environment variable.
GNATCOV_RTS=
There is an empty GNATCOV_RTS variable defined, which we can set to the
path to the installed gnatcov runtime project file.
Once the tests are all written, generating the coverage report (on the standard output in this example) can be done by simply invoking:
make -C harness/ coverage
This outputs the sequence of commands issued to perform the coverage computation, then the results if the “report” format is selected. On our example, this would be like:
Instrumenting project Pkg1.Test_Data.Tests/test_driver.gpr:
gnatcov instrument -PPkg1.Test_Data.Tests/test_driver.gpr --dump-trigger=main-end --level=stmt --projects=Prj --units=@Pkg1.Test_Data.Tests/units.list
Building Pkg1.Test_Data.Tests/test_driver.gpr:
gprbuild -PPkg1.Test_Data.Tests/test_driver.gpr -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts_full
...
Running Pkg1.Test_Data.Tests/test_driver.gpr:
GNATCOV_TRACE_FILE=Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace Pkg1.Test_Data.Tests/test_driver
pkg1.ads:2:4: info: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:44)
pkg1.ads:3:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:65)
2 tests run: 0 passed; 2 failed; 0 crashed.
Creating checkpoint for Pkg1.Test_Data.Tests/test_driver.gpr:
gnatcov coverage --save-checkpoint=Pkg1.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg1.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg1.Test_Data.Tests/units.list
Instrumenting project Pkg2.Test_Data.Tests/test_driver.gpr:
gnatcov instrument -PPkg2.Test_Data.Tests/test_driver.gpr --dump-trigger=main-end --level=stmt --projects=Prj --units=@Pkg2.Test_Data.Tests/units.list
Building Pkg2.Test_Data.Tests/test_driver.gpr:
gprbuild -PPkg2.Test_Data.Tests/test_driver.gpr -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts_full
...
Running Pkg2.Test_Data.Tests/test_driver.gpr:
GNATCOV_TRACE_FILE=Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace Pkg2.Test_Data.Tests/test_driver
pkg2.ads:2:4: info: corresponding test PASSED
pkg2.ads:3:4: info: corresponding test PASSED
2 tests run: 2 passed; 0 failed; 0 crashed.
Creating checkpoint for Pkg2.Test_Data.Tests/test_driver.gpr:
gnatcov coverage --save-checkpoint=Pkg2.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg2.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg2.Test_Data.Tests/units.list
Creating coverage report:
gnatcov coverage -P../prj.gpr -CPkg1.Test_Data.Tests/test_driver-gnattest.ckpt -CPkg2.Test_Data.Tests/test_driver-gnattest.ckpt --annotate=report --level=stmt
** COVERAGE REPORT **
===========================
== 1. ASSESSMENT CONTEXT ==
===========================
Date and time of execution: 2021-08-23 15:35:58 +02:00
Tool version: XCOV development-tree
Command line:
gnatcov coverage -P../prj.gpr -CPkg1.Test_Data.Tests/test_driver-gnattest.ckpt -CPkg2.Test_Data.Tests/test_driver-gnattest.ckpt --annotate=report --level=stmt
Coverage level: stmt
Trace files:
Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace
kind : source
program : Pkg1.Test_Data.Tests/test_driver
date : 2021-08-23 15:35:56 +02:00
tag :
processed: gnatcov coverage --save-checkpoint=Pkg1.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg1.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg1.Test_Data.Tests/units.list @ 2021-08-23 15:35:56 +02:00
Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace
kind : source
program : Pkg2.Test_Data.Tests/test_driver
date : 2021-08-23 15:35:57 +02:00
tag :
processed: gnatcov coverage --save-checkpoint=Pkg2.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg2.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg2.Test_Data.Tests/units.list @ 2021-08-23 15:35:58 +02:00
============================
== 2. COVERAGE VIOLATIONS ==
============================
2.1. STMT COVERAGE
------------------
pkg1.adb:13:7: statement not executed
pkg1.adb:18:7: statement not executed
2 violation.
=========================
== 3. ANALYSIS SUMMARY ==
=========================
2 STMT violation.
** END OF REPORT **
The log shows all the steps necessary to obtain coverage results from the mutliple test drivers, and end with the report.
From the coverage report, we see that the only lines not covered are in
pkg1.adb, which is expected as the tests corresponding to that unit are
not implemented. By using separate drivers, although the code in Pkg2 uses
the subprograms defined in Pkg1, we were able to not have the coverage
results from the unit tests on Pkg1 be polluted by the tests on Pkg2.
Regenerating the harness to use a single monolithic driver, and re-generating a
coverage report shows that without the separate drivers, Pkg1 is marked as
covered despite not having any test implemented:
gnattest -P prj.gpr && make -C harness/ coverage
which outputs:
Instrumenting project test_driver.gpr:
gnatcov instrument -Ptest_driver.gpr --dump-trigger=main-end --level=stmt --projects=Prj
Building test_driver.gpr:
gprbuild -Ptest_driver.gpr -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts_full
...
Running test_driver.gpr:
GNATCOV_TRACE_FILE=test_driver-gnattest_td.srctrace ./test_driver
pkg2.ads:2:4: info: corresponding test PASSED
pkg2.ads:3:4: info: corresponding test PASSED
pkg1.ads:2:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:44)
pkg1.ads:3:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:65)
4 tests run: 2 passed; 2 failed; 0 crashed.
Creating checkpoint for test_driver.gpr:
gnatcov coverage --save-checkpoint=test_driver-gnattest.ckpt -Ptest_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj test_driver-gnattest_td.srctrace
Creating coverage report:
gnatcov coverage -P../prj.gpr -Ctest_driver-gnattest.ckpt --annotate=report --level=stmt
** COVERAGE REPORT **
===========================
== 1. ASSESSMENT CONTEXT ==
===========================
Date and time of execution: 2021-08-23 15:52:36 +02:00
Tool version: XCOV development-tree
Command line:
gnatcov coverage -P../prj.gpr -Ctest_driver-gnattest.ckpt --annotate=report --level=stmt
Coverage level: stmt
Trace files:
test_driver-gnattest_td.srctrace
kind : source
program : ./test_driver
date : 2021-08-23 15:52:35 +02:00
tag :
processed: gnatcov coverage --save-checkpoint=test_driver-gnattest.ckpt -Ptest_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj test_driver-gnattest_td.srctrace @ 2021-08-23 15:52:35 +02:00
============================
== 2. COVERAGE VIOLATIONS ==
============================
2.1. STMT COVERAGE
------------------
No violation.
=========================
== 3. ANALYSIS SUMMARY ==
=========================
No STMT violation.
** END OF REPORT **
3.1.3. Instrumenting test harnesses for a SPARK project¶
General information about SPARK code instrumentation can be found in section Instrumentation and coverage of SPARK code. The compilation of instrumented user code needs to be controlled by a configuration pragma file. When using GNATtest, the main project is the generated test driver project, not the original user code project. As such, the configuration pragma files that need to be passed during compilation cannot be specified in the project under test.
There are two possibilities to specify the configuration pragma file to be
used when building the instrumented harness projects. The first one is to
modify the gnattest_common.gpr project file (which is not overwritten when
the harness is regenerated), as in:
--- harness/gnattest_common.gpr
+++ harness/gnattest_common.gpr
type TD_Compilation_Type is ("contract-checks","no-contract-checks", "no-config-file");
TD_Compilation : TD_Compilation_Type := external ("TEST_DRIVER_BUILD_MODE", "no-config-file");
package Builder is
case TD_Compilation is
when "contract-checks" =>
for Global_Configuration_Pragmas use "suppress.adc";
when "no-contract-checks" =>
for Global_Configuration_Pragmas use "suppress_no_ghost.adc";
when "no-config-file" =>
- null;
+ for Global_Configuration_Pragmas use "<path to file>/instrument-spark.adc";;
end case;
end Builder;
An alternative solution is to specify the configuration pragma file on the command line when invoking the Makefile:
make BUILDERFLAGS='-cargs:Ada -gnatec=instrument-spark.adc' coverage
3.2. Developing a custom solution¶
If the generated makefile is not suitable to use with the execution environment, there are a few things to keep in mind in order not to have unexpected coverage results.
3.2.1. Single test driver¶
In the case where a monolithic test driver is generated by GNATtest, obtaining coverage results for your project is relatively simple, and the only aspect which needs attention is the specification of units of interest, particularly in the case of using source traces.
When using source traces, GNATcoverage needs to instrument the main so that execution
traces are dumped at the end of the test run. So despite none of the units in
the harness project being of interest, is is important that the root project
passed to all gnatcov commands is test_driver.gpr.
The projects generated by GNATtest all specify that none of the units are of interest, so none of the units generated by GNATtest should appear in the reports.
3.2.2. Separate test drivers¶
Using separate test drivers is advisable to avoid incidental coverage of one
unit from the testing of other units
(see Handle incidental coverage effects).
Note that since the smallest division of a project supported by GNATcoverage is the
unit, there is no benefit in specifying --separate-drivers=test
instead of --separate-drivers=unit to GNATtest, as far as incidental
coverage is concerned.
When using mutliple drivers, there will be a test_driver.gpr generated for
each unit. For each generated driver, the project needs to be instrumented
(if source traces are used), built, run and a checkpoint
must be created from the execution trace. Then a call to gnatcov coverage merges the
coverage data from all the checkpoints and generates the desired report.
The key point in the process is to specify, when creating all the individual
checkpoints, which unit is being tested, so that the checkpoint only records
the coverage information about that unit, and discard any incidental coverage
on other units. During harness generation, a file named units.list will be
created in the same directory as each test_driver.gpr file. This file
contains the name of the unit tested by the driver, and can be used to specify
to gnatcov to only process the unit under test, by adding the switch
--units=@units.list.