4. Using GNATtest with GNATcoverage

Using GNATcoverage to obtain coverage information on the project being tested is the next logical step in the GNATtest workflow. There are two main ways to accomplish this: using the facilities generated by GNATtest (namely a Makefile) during the harness generation, or deriving a custom solution, while keeping a few important concepts in mind.

4.1. Using the Makefile generated by GNATtest

GNATtest will always generate a Makefile to automate as much as possible the production of a coverage report on the project being tested.

This Makefile is generated in the same directory as the harness project, and consists of two files:

  • The main Makefile which is regenerated for each invocation of gnattest

  • The coverage_settings.mk secondary makefile, which holds most of the options that will be passed to gnatcoverage. The values present when the file is generated are extracted from the root project file that was passed to GNATtest. This file will not be re-generated by subsequent invocation of gnattest, so it is a suitable place to store settings for GNATcoverage if they are not all defined in the project file.

The makefile also assumes that the GNATcoverage runtime is already built and installed, and that the path to its project file is in the environment variable GPR_PROJECT_PATH so it can be automaticaly located by GPRbuild. It is also possible to specify the path to the project file in the GNATCOV_RTS variable in the coverage_settings.mk file. See Setting up the coverage runtime library for instructions on building and installing the runtime.

If all the switches that need to be passed to GNATcoverage are correctly defined in coverage_Settings.mk, then running:

make coverage

will build and run all the harness projects and then generate a coverage report for all the units under test.

By default, coverage data for a native project is obtained using source traces and coverage for projects targeting a cross environment is assessed using binary traces.

For cases where the default behavior isn’t appropriate for the project under test, the rules for producing a coverage report from both kind of traces are always generated.

4.1.1. Special considerations for cross environment targets

Using GNATcoverage to assess coverage for a project with a cross-environment target may require some special adaptation to be able to run the program and collect execution traces. The generated makefile tries to implement an execution and trace collection strategy that is the most likely to work for most projects, based on GNATemulator for execution.

For binary traces, this is done through the gnatcov run command (see gnatcov run command line), which takes care of setting up the trace collection mechanism in GNATemulator. For this configuration to work properly, the Board attribute in the package Emulator needs to be specified in the root project file, or alternatively the GNATEMU_BOARD variable must be set in the coverage_settings.mk secondary makefile.

For source traces, the instrumented program will be run using GNATemuator directly, and assumes that the standard package Ada.GNAT_IO allows data to be output on a serial port. The instrumentation process overrides the two switches --dump-trigger and --dump-channel to the values main-end and base64-stdout respectively as they are the most adapted to the GNATtest harness project and to the GNATemulator execution environment. The program is run under GNATemulator, and the output of the first serial port is captured as it contains the trace in a base-64 encoding. This text trace is then decoded and converted into a regular source trace.

There is currently no out-of-the-box support for running the GNATtest harness projects on target and collect execution traces from a hardware probe; the current makefile can still be used to compile the executables and process the traces once they are available, but the execution rules will need rewriting.

4.1.2. Usage example

In this example the project has two units, Pkg1 and Pkg2. The code in Pkg2 calls all the subprograms in Pkg1, and we wish to unit test each unit in isolation. To do so, we will use the Individual Test Drivers feature of GNATtest so each unit under test gets its own test executable, and we’ll be able to create each time a checkpoint containing coverage for the unit being tested. See Handle incidental coverage effects on how checkpoints help avoid incidental coverage.

The project is structured as follows:

<project root directory/>
  |prj.gpr
  |src/
    |pkg1.ads
    |pkg2.ads
    ...
  |tests/
    |<gnattest generated test skeletons>
  |harness/
    |Makefile
    |coverage_settings.mk
    |<GNATtest generated harness files>

and is defined with the following project file:

--  prj.gpr
project Prj is
   for Source_Dirs use ("src");
   for Object_Dir use "obj";

   package Coverage is
      Cov_Level := ("--level=stmt");
      for Switches ("instrument") use ("--dump-trigger=atexit") & Cov_Level;
      for Switches ("run") use Cov_Level;
      for Switches ("coverage") use ("--annotate=report") & Cov_Level;
   end Coverage;

   package Gnattest is
      for Tests_Dir use "../tests";
      for Harness_Dir use "../harness";
   end Gnattest;

end Prj;

Specifying the coverage preferences in the project file spares us modifying the coverage_settings.mk after generation.

The tests will be stored in the tests directory, as specified by the project attribute Tests_Dir, and the GNATtest generated harness in the harness directory.

After invoking GNATtest as follows:

gnattest -P prj.gpr --separate-drivers=unit

The following two files can be found in the harness directory:

  • The Makefile with (amongst other things) three rules that execute the full coverage workflow for each test driver project, and then generates a coverage report combining the results:

    bin-coverage: ...
    
    inst-coverage: ...
    
    coverage: inst-coverage
    

    The first rule (bin-coverage) runs the binary traces workflow for GNATcoverage, whereas the second rule (inst-coverage) runs the source-trace (or instrumentation based) workflow. The last rule (coverage) is defined to use the workflow the most likely to work given the current target.

  • The coverage_settings.mk file, which, when generated, copied all the values of the relevant root project attributes into corresponding variables:

    # Settings in this file were extracted from the source project
    # or are gnattest default values if they weren't specified in the source
    # project. They may need adjustments to fit your particular coverage needs.
    # This file won't be overwritten when regenerating the harness.
    
    # Switches for the various gnatcov commands
    SWITCHES_INSTRUMENT=--dump-trigger=main-end --level=stmt
    
    SWITCHES_RUN=--level=stmt
    
    SWITCHES_COVERAGE=--annotate=report --level=stmt
    
    # Path to the installed gnatcov rts project file.
    # No need to specify it if the project file path was added to the
    # GPR_PROJECT_PATH environment variable.
    GNATCOV_RTS=
    

    There is an empty GNATCOV_RTS variable defined, which we can set to the path to the installed gnatcov runtime project file.

Once the tests are all written, generating the coverage report (on the standard output in this example) can be done by simply invoking:

make -C harness/ coverage

This outputs the sequence of commands issued to perform the coverage computation, then the results if the “report” format is selected. On our example, this would be like:

Instrumenting project Pkg1.Test_Data.Tests/test_driver.gpr:
gnatcov instrument -PPkg1.Test_Data.Tests/test_driver.gpr --dump-trigger=main-end --level=stmt   --projects=Prj --units=@Pkg1.Test_Data.Tests/units.list

Building Pkg1.Test_Data.Tests/test_driver.gpr:
gprbuild  -PPkg1.Test_Data.Tests/test_driver.gpr  -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts
...

Running Pkg1.Test_Data.Tests/test_driver.gpr:
GNATCOV_TRACE_FILE=Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace Pkg1.Test_Data.Tests/test_driver
pkg1.ads:2:4: info: corresponding test FAILED:  Test not implemented. (pkg1-test_data-tests.adb:44)
pkg1.ads:3:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:65)
2 tests run: 0 passed; 2 failed; 0 crashed.

Creating checkpoint for Pkg1.Test_Data.Tests/test_driver.gpr:
gnatcov coverage --save-checkpoint=Pkg1.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg1.Test_Data.Tests/test_driver.gpr  --annotate=report --level=stmt  --cancel-annotate --projects=Prj Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg1.Test_Data.Tests/units.list

Instrumenting project Pkg2.Test_Data.Tests/test_driver.gpr:
gnatcov instrument -PPkg2.Test_Data.Tests/test_driver.gpr --dump-trigger=main-end --level=stmt   --projects=Prj --units=@Pkg2.Test_Data.Tests/units.list

Building Pkg2.Test_Data.Tests/test_driver.gpr:
gprbuild  -PPkg2.Test_Data.Tests/test_driver.gpr  -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts
...

Running Pkg2.Test_Data.Tests/test_driver.gpr:
GNATCOV_TRACE_FILE=Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace Pkg2.Test_Data.Tests/test_driver
pkg2.ads:2:4: info: corresponding test PASSED
pkg2.ads:3:4: info: corresponding test PASSED
2 tests run: 2 passed; 0 failed; 0 crashed.

Creating checkpoint for Pkg2.Test_Data.Tests/test_driver.gpr:
gnatcov coverage --save-checkpoint=Pkg2.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg2.Test_Data.Tests/test_driver.gpr  --annotate=report --level=stmt  --cancel-annotate --projects=Prj Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg2.Test_Data.Tests/units.list

Creating coverage report:
gnatcov coverage -P../prj.gpr -CPkg1.Test_Data.Tests/test_driver-gnattest.ckpt -CPkg2.Test_Data.Tests/test_driver-gnattest.ckpt --annotate=report --level=stmt
** COVERAGE REPORT **

===========================
== 1. ASSESSMENT CONTEXT ==
===========================

Date and time of execution: 2021-08-23 15:35:58 +02:00
Tool version: XCOV development-tree

Command line:
gnatcov coverage -P../prj.gpr -CPkg1.Test_Data.Tests/test_driver-gnattest.ckpt -CPkg2.Test_Data.Tests/test_driver-gnattest.ckpt --annotate=report --level=stmt

Coverage level: stmt

Trace files:

Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace
  kind     : source
  program  : Pkg1.Test_Data.Tests/test_driver
  date     : 2021-08-23 15:35:56 +02:00
  tag      :
  processed: gnatcov coverage --save-checkpoint=Pkg1.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg1.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg1.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg1.Test_Data.Tests/units.list @ 2021-08-23 15:35:56 +02:00

Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace
  kind     : source
  program  : Pkg2.Test_Data.Tests/test_driver
  date     : 2021-08-23 15:35:57 +02:00
  tag      :
  processed: gnatcov coverage --save-checkpoint=Pkg2.Test_Data.Tests/test_driver-gnattest.ckpt -PPkg2.Test_Data.Tests/test_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj Pkg2.Test_Data.Tests/test_driver-gnattest_td.srctrace --units=@Pkg2.Test_Data.Tests/units.list @ 2021-08-23 15:35:58 +02:00

============================
== 2. COVERAGE VIOLATIONS ==
============================

2.1. STMT COVERAGE
------------------

pkg1.adb:13:7: statement not executed
pkg1.adb:18:7: statement not executed

2 violation.

=========================
== 3. ANALYSIS SUMMARY ==
=========================

2 STMT violation.

** END OF REPORT **

The log shows all the steps necessary to obtain coverage results from the mutliple test drivers, and end with the report.

From the coverage report, we see that the only lines not covered are in pkg1.adb, which is expected as the tests corresponding to that unit are not implemented. By using separate drivers, although the code in Pkg2 uses the subprograms defined in Pkg1, we were able to not have the coverage results from the unit tests on Pkg1 be polluted by the tests on Pkg2.

Regenerating the harness to use a single monolithic driver, and re-generating a coverage report shows that without the separate drivers, Pkg1 is marked as covered despite not having any test implemented:

gnattest -P prj.gpr && make -C harness/ coverage

which outputs:

Instrumenting project test_driver.gpr:
gnatcov instrument -Ptest_driver.gpr --dump-trigger=main-end --level=stmt   --projects=Prj

Building test_driver.gpr:
gprbuild  -Ptest_driver.gpr  -o test_driver --src-subdirs=gnatcov-instr --implicit-with=gnatcov_rts
...
Running test_driver.gpr:
GNATCOV_TRACE_FILE=test_driver-gnattest_td.srctrace ./test_driver
pkg2.ads:2:4: info: corresponding test PASSED
pkg2.ads:3:4: info: corresponding test PASSED
pkg1.ads:2:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:44)
pkg1.ads:3:4: error: corresponding test FAILED: Test not implemented. (pkg1-test_data-tests.adb:65)
4 tests run: 2 passed; 2 failed; 0 crashed.

Creating checkpoint for test_driver.gpr:
gnatcov coverage --save-checkpoint=test_driver-gnattest.ckpt -Ptest_driver.gpr  --annotate=report --level=stmt  --cancel-annotate --projects=Prj test_driver-gnattest_td.srctrace

Creating coverage report:
gnatcov coverage -P../prj.gpr -Ctest_driver-gnattest.ckpt --annotate=report --level=stmt
** COVERAGE REPORT **

===========================
== 1. ASSESSMENT CONTEXT ==
===========================

Date and time of execution: 2021-08-23 15:52:36 +02:00
Tool version: XCOV development-tree

Command line:
gnatcov coverage -P../prj.gpr -Ctest_driver-gnattest.ckpt --annotate=report --level=stmt

Coverage level: stmt

Trace files:

test_driver-gnattest_td.srctrace
  kind     : source
  program  : ./test_driver
  date     : 2021-08-23 15:52:35 +02:00
  tag      :
  processed: gnatcov coverage --save-checkpoint=test_driver-gnattest.ckpt -Ptest_driver.gpr --annotate=report --level=stmt --cancel-annotate --projects=Prj test_driver-gnattest_td.srctrace @ 2021-08-23 15:52:35 +02:00

============================
== 2. COVERAGE VIOLATIONS ==
============================

2.1. STMT COVERAGE
------------------

No violation.

=========================
== 3. ANALYSIS SUMMARY ==
=========================

No STMT violation.

** END OF REPORT **

4.1.3. Instrumenting test harnesses for a SPARK project

General information about SPARK code instrumentation can be found in section Instrumentation and coverage of SPARK code. The compilation of instrumented user code needs to be controlled by a configuration pragma file. When using GNATtest, the main project is the generated test driver project, not the original user code project. As such, the configuration pragma files that need to be passed during compilation cannot be specified in the project under test.

There are two possibilities to specify the configuration pragma file to be used when building the instrumented harness projects. The first one is to modify the gnattest_common.gpr project file (which is not overwritten when the harness is regenerated), as in:

--- harness/gnattest_common.gpr
+++ harness/gnattest_common.gpr
  type TD_Compilation_Type is ("contract-checks","no-contract-checks", "no-config-file");
  TD_Compilation : TD_Compilation_Type := external ("TEST_DRIVER_BUILD_MODE", "no-config-file");

  package Builder is
     case TD_Compilation is
        when "contract-checks" =>
           for Global_Configuration_Pragmas use "suppress.adc";
        when "no-contract-checks" =>
           for Global_Configuration_Pragmas use "suppress_no_ghost.adc";
        when "no-config-file" =>
-          null;
+          for Global_Configuration_Pragmas use "<path to file>/instrument-spark.adc";;
     end case;
  end Builder;

An alternative solution is to specify the configuration pragma file on the command line when invoking the Makefile:

make BUILDERFLAGS='-cargs:Ada -gnatec=instrument-spark.adc' coverage

4.2. Developing a custom solution

If the generated makefile is not suitable to use with the execution environment, there are a few things to keep in mind in order not to have unexpected coverage results.

4.2.1. Single test driver

In the case where a monolithic test driver is generated by GNATtest, obtaining coverage results for your project is relatively simple, and the only aspect which needs attention is the specification of units of interest, particularly in the case of using source traces.

When using source traces, GNATcoverage needs to instrument the main so that execution traces are dumped at the end of the test run. So despite none of the units in the harness project being of interest, is is important that the root project passed to all gnatcov commands is test_driver.gpr.

The projects generated by GNATtest all specify that none of the units are of interest, so none of the units generated by GNATtest should appear in the reports.

4.2.2. Separate test drivers

Using separate test drivers is advisable to avoid incidental coverage of one unit from the testing of other units (see Handle incidental coverage effects). Note that since the smallest division of a project supported by GNATcoverage is the unit, there is no benefit in specifying --separate-drivers=test instead of --separate-drivers=unit to GNATtest, as far as incidental coverage is concerned.

When using mutliple drivers, there will be a test_driver.gpr generated for each unit. For each generated driver, the project needs to be instrumented (if source traces are used), built, run and a checkpoint must be created from the execution trace. Then a call to gnatcov coverage merges the coverage data from all the checkpoints and generates the desired report.

The key point in the process is to specify, when creating all the individual checkpoints, which unit is being tested, so that the checkpoint only records the coverage information about that unit, and discard any incidental coverage on other units. During harness generation, a file named units.list will be created in the same directory as each test_driver.gpr file. This file contains the name of the unit tested by the driver, and can be used to specify to gnatcov to only process the unit under test, by adding the switch --units=@units.list.

5. Using GNATtest with GNATfuzz (experimental)

This section presents how a pre-existing GNATtest test harness can be used as a starting corpus for a GNATfuzz fuzzing campaign, and how inputs of interest found by GNATfuzz can be imported back into a GNATtest harness. These features are still experimental; the workflow and command line interface may change in the future.

5.1. Setting up the environment

To ensure the entire workflow functions properly, it’s crucial to configure the various tools by setting specific environment variables.

The first step is to setup the value generation runtime library. For detailed instructions on how to do so, see Setting up the test generation runtime.

5.2. Using GNATStudio to perform the integration

The simplest way to sequence the invocations of the two tools is to use the GNATstudio integration plugin.

First, create a GNATtest harness project if it doesn’t already exist, using the Analyze -> GNATtest -> Generate harness project entries in the menu bar. Note that in the dialog box opening there is an option to generate tests inputs if needed.

Then, exporting tests inputs from GNATtest to GNATfuzz and running a fuzzing campaign on a specific subprogram can be done by right-clicking on the declaration of the target subprogram, then selecting GNATtest -> Start/Stop fuzzing subprogram, as illustrated bellow.

../_images/gs_menu.png

This will first instrument sources and re-generate the GNATtest harness in order to be able to intercept the inputs passed to the subprogram, then run the test harness, dumping the inputs in a binary format that can be used by GNATfuzz. GNATstudio will then setup a fuzzing session on the subprogram, for which the parameters can be controlled through the various pop-up windows that will be displayed during the process.

gnatfuzz will periodically export newly found inputs in a human readable JSON format under <obj>/gnattest/test/JSON_Tests, where <obj> designates the object directory of the project.

The fuzzing session will stop once all the stopping criteria have been met. The fuzzing session can also be stopped early by right clicking on the subprogram declaration, then selecting GNATtest => Start/Stop fuzzing subprogram.

After the fuzzing sessions has ended, a new GNATtest harness will be regenerated, including the tests exported by the GNATfuzz session. These will appear in <obj>/gnattest/tests/<unit_name>-test_data-test_<subp_name>_<subp_hash>.adb, where <unit_name> is the name of the unit in which the subprogram is declared, <subp_name> is the name of the subprogram, and <subp_hash> is a hash based on the profile of the subprogram, in order to differentiate overloads.