Object coverage analysis with gnatcov coverage

Object coverage analysis computes metrics focused on machine-level object code, concerned with machine basic instructions or conditional branches. Conditional general instructions (i.e.non-branch instructions), such as ARM conditional instructions are not supported. If such instructions are present in the subprograms to be analyzed, their reported coverage status may be incorrect.

On request, the metrics can be presented on sources, with an annotation on each line synthesizing the coverage status of all the instructions generated for this line. This mapping relies on debug information, so sources must be compiled with -g for this to work. There is no further compilation requirement for object coverage alone.

Once your application is built, the analysis proceeds in two steps: gnatcov run is used to produce execution traces, then gnatcov coverage to generate coverage reports. Object coverage is queried by passing a specific --level argument to gnatcov coverage; =insn or =branch, described in detail in the following sections. As for source coverage, there is never a requirement to recompile just because a different criterion needs to be analyzed.

The Producing binary traces with gnatcov run chapter of this document provides details on the trace production interface. The remainder of this chapter explains the use of gnatcov coverage in particular, to analyse traces once they have been produced. The general command line structure is always like:

gnatcov coverage --level=<criterion> --annotate=<format>
                 [--routines=<names>] ... <traces>

The optional --routines argument provides the set of object level subprogram names on which the analysis should focus. This set defaults to the full set of symbols defined by all the executables associated with the provided execution traces.

Later in this chapter, Focusing on subprograms of interest (--routines) explains how to construct the relevant list of names for --routines. Prior to this comes a section describing the available report formats, then more details regarding Object Instruction Coverage analysis (--level=insn), Object Branch Coverage analysis (--level=branch), and specificities regarding Inlining & Ada Generic units. Finally, Full object coverage considerations describes tools that help analyze low-level object files for issues of interest when aiming at full object coverage.

gnatcov coverage command line

Coverage analysis with GNATcoverage is performed by invoking gnatcov coverage for a set of critera queried via the --level command line option. The general interface synopsis is available from gnatcov --help:

gnatcov coverage OPTIONS TRACE_FILES

The available options are as follows:

-c, --level (mandatory):

Tell the set of coverage criteria to be assessed. The possible values for object coverage analysis are insn, branch, both explained later in this chapter.

-a, --annotate (mandatory):

Request a specific output report format. The two possible criteria support xcov[+], html, and asm, with interpretations that vary depending on the assessed criteria. See the corresponding documentation later in this chapter for more details. This option accepts comma separated values and/or can be specified multiple times on the command line, in which case there will be one report produced for each of the requested annotation formats.

--timezone :

Select the timezone to use when displaying trace creation dates in reports, which could be either local or utc. The local timezone is used by default.

--output-dir :

Request that the report files (index and annotated sources for the xcov and html output formats) be output in the provided directory. If not specified, the default is the root project’s object directory if using projects, and the current directory if not. If more than one of the above annotation formats is requested, then each report will be placed in a subdirectory named accordingly.

--report-title :

Request that generated HTML documents (index for the html output format) are assigned a customized title. For instance, passing --report-title="Project ABC" will yield titles such as: Project ABC - GNATcoverage Report. If passed multiple times, passing an empty string last will restore the default behavior. This option is ignored is the selected output format does not support titles.

-T, --trace (mandatory), possibly repeated and accepting @listfile arguments :

Provide the set of execution traces for which a report is to be produced. When multiple traces are provided, gnatcov produces a consolidated result, as described in detail in the Coverage Consolidation chapter of this manual.

--exec:

Override executable from traces. Trace files contain an indication of the executable used to generate them. This option causes the named executable to be loaded for coverage analysis, and to override the indication contained in any trace specified after it on the command line. An empty executable name may be specified to restore the default behaviour of using the indication contained in each trace file. Note that --exec may appear last on the command line, in which case it applies to no trace file, but still causes the indicated executable to be included in the coverage analysis. This ensures that any code in that executable that is not exercised by some trace file will be reported as not covered.

--routines, possibly repeated and accepting @listfile arguments:

Provide the list of object symbol names that correspond to routines for which the coverage assessment is to be performed. Each instance of this option on the command line adds to what is to be assessed eventually. See the Focusing on subprograms of interest (--routines) section for extra details and use examples.

--alis, possibly repeated and accepting @listfile arguments:

Provide set of Library Information files for units where there might be applicable exemption regions to account for, as explained in the Object coverage exemptions section of this manual.

-P:

Use the indicated project file to find default options. This can also be used, possibly combined with other project related switches, as an alternative to --alis to determine the set of units for which coverage exemptions should be honored. In this case, the target/runtime/scenario contextual options that would be passed to build the project should also be passed for proper interpretation of the project files. This is similar to the use of project facilities for the determination of SCOs for source coverage analysis, and described in the Specifying Units Of Interest chapter of this manual.

-t, --target :

State the target toolchain configuration used to build the analyzed programs, as provided to gprbuild. For cross or 32bit native configurations, this switch together with its possible --RTS companion is required for all commands using project files unless the root project provides the information with Target/Runtime attributes.

Output report formats (--annotate)

Object coverage reports may be produced in various formats, as requested with the --annotate command line argument of gnatcov coverage.

The asm format produces an annotated assembly output, with a coverage indication attached to every single instruction. This is the base information of interest to object coverage analysis, simply presented in different manners through the other possible output formats. The xcov and html formats produce a set of annotated source files, in the directory where gnatcov is launched unless overriden with a --output-dir option. Even though presented on sources, the annotations remain representative of object coverage metrics, synthesized for all the instructions associated with each source line.

Later in this chapter we name output formats by the text to add to --annotate on the command line. For example, we use “the =asm outputs” to mean “the coverage reports produced with --annotate=asm”. We also sometimes use in-source reports or outputs to designate the set of outputs in annotated source forms.

We illustrate the various formats with coverage analysis excerpts on the following example Ada support unit:

--  raise Program_Error if T is False. Do nothing otherwise.

procedure Assert (T : Boolean) is
begin
   if not T then
      raise Program_Error;
   end if;
end Assert;

As the contents suggest, this subprogram is expected never to be called with T False in nominal situations.

Machine level reports (=asm)

For object coverage analysis, --annotate=asm produces annotated assembly code for all the selected routines on standard output. The annotations are first visible as a special character on each machine code line to convey the coverage status of the corresponding instruction. The following output excerpt, for example, is part of a coverage report for our Assert subprogram compiled for the PowerPc architecture:

Coverage level: branch
_ada_assert !: 0c4-123
0c4 +:  94 21 ff e0      stwu   r1,-0x0020(r1)
...
0ec +:  2f 80 00 00      cmpiw  cr7,r0,0x0000
0f0 +:  41 9e 00 18      beq-   cr7,0x108 <_ada_assert+00000044>
...
100 -:  38 80 00 04      li     r4,0x0004
104 -:  48 00 00 a1      bl     0x1a4 <__gnat_last_chance_handler>
108 +:  60 00 00 00      ori    r0,r0,0x0000
...
120 +:  4e 80 00 20      blr

A - annotation for a line always conveys that the instruction was not executed at all. The instruction is also said to be uncovered in this case. Conversely, a + means that the instruction is fully covered with respect to the analyzed criterion, with a meaning which depends on both the criterion and the kind of instruction – whether the instruction is a conditional branch and whether we are doing mere instruction or object branch coverage analysis. Annotations conveying partial coverage might show up as well, also depending on the criterion and kind of instruction.

More details on the instruction specific annotations are provided in the sections that follow. Then, as the first line of the example suggests, the report also annotates each subprogram symbol as a whole, with the range of addresses that the subprogram spans and a synthetic coverage indication according to the following table:

Symbol Annotation

Meaning

-

All the subprogram instructions are uncovered (none executed)

+

All the subprogram instructions are fully covered

!

Some of the subprogram instructions were fully or partially covered

In our example, the code features both fully covered and uncovered instructions, and the _assert symbol as a whole is marked partially covered with a ! annotation.

Annotated sources, text (=xcov[+])

For object coverage analysis, --annotate=xcov produces annotated source files with the .xcov extension, one per original compilation unit in the selected output directory.

The annotations are visible at the beginning of every source line, as a single character which synthesizes the coverage status of all the machine instructions generated for this line. The following table provides a uniform description of this synthesis for all the object level criteria:

Source Annotation

Meaning

.

no machine code associated with this line

-

all the instructions associated with the line are - (uncovered)

+

all the instructions associated with the line are + (fully covered)

!

otherwise

The report also includes a short header, which features a global coverage count with respect to the total number of lines with associated code, as well as an indication of the assessed criterion. Below is an example of report obtained for our Assert unit:

examples/src/assert.adb:
75% of 4 lines covered

Coverage level: insn
  1 +: procedure Assert (T : Boolean) is
  2 .: begin
  3 +:    if not T then
  4 -:       raise Program_Error;
  5 .:    end if;
  6 +: end Assert;

To lines with associated object code we apply qualifiers similar to those for individual instructions: when the synthetic coverage indication for a line is -, + or !, we qualify the line as uncovered, fully covered, or partially covered, respectively. Note that even though they are rendered on source lines, the annotations are really meant to convey object code properties here, hence are of a different nature than what the DO-178B source structural coverage criteria refer to. See our Object/Source level metrics considerations section for further details on this aspect.

With --annotate=xcov+ (extra + at the end), the machine instructions and their individual coverage status are printed next to their associated source line.

Annotated sources, html (=html)

For object coverage criteria, gnatcov coverage --annotate=html produces an annotated version of each source file. Each annotated source page contains a summary of the assessment results followed by the original source lines, all numbered and marked with a coverage annotation as in the --annotate=xcov case. In addition, lines with obligations are colorized in green, orange or red for +, ! or - coverage respectively. An index.html page is also produced, which contains a description of the assessment context (assessed criteria, set of trace files involved, …) and a summary of the coverage results for all the units, with links to their annotated sources.

Similarily to the xcov format case, --annotate=html attaches to each line details about the coverage status of all the individual instructions generated for the line. These are folded within the line and expanded when a mouse click hits it.

Object Instruction Coverage analysis (--level=insn)

Object Instruction Coverage treats basic and conditional branch instructions identically, as either executed or not, hence fully covered or uncovered. The =asm instruction annotations are as follows:

Insn Annotation

Meaning

-

the instruction was not executed

+

the instruction was executed

The =asm excerpt below provides a representative example of the PowerPC instruction coverage achieved for our Assert procedure by nominal executions where the subprogram is never called with T False:

_ada_assert !: 0c4-123
0c4 +:  94 21 ff e0      stwu   r1,-0x0020(r1)
...
0ec +:  2f 80 00 00      cmpiw  cr7,r0,0x0000
0f0 +:  41 9e 00 18      beq-   cr7,0x108 <_ada_assert+00000044>
...
100 -:  38 80 00 04      li     r4,0x0004
104 -:  48 00 00 a1      bl     0x1a4 <__gnat_last_chance_handler>
108 +:  60 00 00 00      ori    r0,r0,0x0000
...
120 +:  4e 80 00 20      blr

Expectedly, the coverage annotations report all the instructions as executed except the two issuing the call to __gnat_last_chance_handler, which correspond to the raise statement in the GNAT high integrity profiles without exception propagation support. The two instructions at offsets 0ec and 0f0 are the comparison and branch conditioned on the comparison result that implement the if construct. We note here that the conditional branch is reported fully covered, as merely executed, even though always taken.

The corresponding =xcov output follows:

1 +: procedure Assert (T : Boolean) is
2 .: begin
3 +:    if not T then
4 -:       raise Program_Error;
5 .:    end if;
6 +: end Assert;

The annotations on lines 3 and 4 correspond to immediate expectations from comments we made on the =asm output. We can also observe annotations on lines 1 and 6, to which the subprogram prologue and epilogue code is attached, and executed as soon as the procedure is called.

Object Branch Coverage analysis (--level=branch)

Object Branch Coverage treats basic and conditional branch instructions differently. Basic instructions are considered fully covered as soon as executed, as in the Instruction Coverage case. Conditional branches, however, have to be executed at least twice to be claimed fully covered : once taking the branch and once executing fall-through, which we sometimes abusively refer to as taken both ways even if one case actually corresponds to the branch not being taken. The =asm instruction annotations are as follows:

Insn Annotation

Meaning

-

the instruction was never executed

+

the instruction was executed and taken both ways for a conditional branch

>

the instruction is a conditional branch, executed and always taken

v

the instruction is a conditional branch, executed and never taken

The v and > annotations are representative of situations where a conditional branch instruction is executed and taken one way only, which constitutes partial coverage of the instruction.

An example of partial coverage is observable on our Assert case, where the conditional branch at offset 0f0 is always taken, jumping over the raise statement code as expected for nominal executions:

_ada_assert !: 0c4-123
0c4 +:  94 21 ff e0      stwu   r1,-0x0020(r1)
...
0ec +:  2f 80 00 00      cmpiw  cr7,r0,0x0000
0f0 >:  41 9e 00 18      beq-   cr7,0x108 <_ada_assert+00000044>
...
100 -:  38 80 00 04      li     r4,0x0004
104 -:  48 00 00 a1      bl     0x1a4 <__gnat_last_chance_handler>
108 +:  60 00 00 00      ori    r0,r0,0x0000
...
120 +:  4e 80 00 20      blr

The corresponding =xcov output follows:

examples/src/assert.adb:
50% of 4 lines covered
Coverage level: branch
  1 +: procedure Assert (T : Boolean) is
  2 .: begin
  3 !:    if not T then
  4 -:       raise Program_Error;
  5 .:    end if;
  6 +: end Assert;

The partial branch coverage logically translates into a partial coverage annotation on the line to which the branch is attached, here the line of the if statement that the conditional branch implements. This is confirmed by the =xcov+ output, where the individual instructions are visible as well together with their own coverage indications:

examples/src/assert.adb:
Coverage level: branch
   1 +: procedure Assert (T : Boolean) is
<_ada_assert+00000000>:+
0c4 +:  94 21 ff e0  stwu   r1,-0x0020(r1)
...
0dc +:  98 1f 00 08  stb    r0,0x0008(r31)
   2 .: begin
   3 !:    if not T then
<_ada_assert+0000001c>:!
0e0 +:  88 1f 00 08  lbz    r0,0x0008(r31)
...
0ec +:  2f 80 00 00  cmpiw  cr7,r0,0x0000
0f0 >:  41 9e 00 18  beq-   cr7,0x108 <_ada_assert+00000044>
   4 -:       raise Program_Error;
<_ada_assert+00000030>:-
0f4 -:  3c 00 ff f0  lis    r0,-0x0010
...
104 -:  48 00 00 a1  bl     0x1a4 <__gnat_last_chance_handler>
   5 .:    end if;
   6 +: end Assert;
<_ada_assert+00000044>:+
...
120 +:  4e 80 00 20  blr

Focusing on subprograms of interest (--routines)

By default, in absence of a --routines argument to gnatcov coverage, object coverage results are produced for the full set of subprogram symbols defined by the executables designated by the analyzed traces.

--routines allows the specification of a set of subprogram symbols of interest so reports refer to this (sub)set only. Each occurrence of --routines on the command line expects a single argument which specifies a subset of symbols of interest. Multiple occurrences are allowed and the subsets accumulate. The argument might be either a single symbol name or a @listfile argument expected to contain a list of symbol names.

For example, focusing on three symbols sym1, sym2 and sym3 can be achieved with either one of the following set of --routines combinations:

--routines=sym1 --routines=sym2 --routines=sym3
or --routines=@symlist123
or --routines=sym3 --routines=@symlist12

… provided a symlist12 text file containing the first two symbol names and a symlist123 text file containing the three of them.

It is often convenient to compute the lists of symbols for a @listfile argument, for example as “the full set of defined subprograms except those with test_ or harness_ at the beginning of their name”. gnatcov provides the gnatcov disp-routines sub-command for this purpose.

The general synopsis of gnatcov disp-routines is as follows:

disp-routines [--exclude|--include] FILES
  Build a list of routines from object files

gnatcov disp-routines outputs the list of symbols in a set built from object files provided on the command line. Object file is to be taken in the general sense here, as conforming to a supported object file format, typically ELF, so includes executable files as well as single compilation unit objects.

The output set is built incrementally while processing the arguments left to right. --include states “from now on, until contradicted, symbols defined in object files are added to the result set”, and --exclude states “from now on, until contradicted, symbols defined in object files are removed from the result set”. An implicit --include is assumed right at the beginning, and each argument may be either the direct name of an object file or a @listfile argument containing a list of such names.

Below are a few examples of commands together with a description of the set they build:

$ gnatcov disp-routines explore
  # (symbols defined in the 'explore' executable)

$ gnatcov disp-routines explore --exclude test_stations.o
  # (symbols from the 'explore' executable)
  # - (symbols from the 'test_stations.o' object file)

$ gnatcov disp-routines --include @sl1 --exclude @sl2 --include @sl3
  # (symbols from the object files listed in text file sl1)
  # - (symbols from the object files listed in text file sl2)
  # + (symbols from the object files listed in text file sl3)

Annotated source reports, when requested, are generated for sources associated with the selected symbols’ object code via debug information, and coverage annotations are produced only on the corresponding. Inlining can have surprising effects in this context, as the following section describes in greater details.

Inlining & Ada Generic units

The generated code for an inlined subprogram call or a generic instantiation materializes two distinct source entities: the expanded source (of the inlined subprogram or of the instanciated generic body) and the expansion request (the subprogram call or the generic instanciation). While this is of no consequence for =asm outputs, which just report coverage of raw machine instructions within their object level subprograms, regardless of the object code origin, this raises a few points of note for in-source outputs.

For inlined calls, potentially surprising results might show up when a specific set of object routines is queried. Indeed, when the code for a symbol A in unit Ua embeds code inlined from unit Ub, a request for an annotated source report for routine A, intuitively expected to yield a report for Ua only, will typically produce an output file for Ub as well, for lines referenced by the machine code inlined in A.

Consider the following Ada units for example, with a functional unit in intops.ads and intops.adb, then a test driver in test_inc0.adb:

package Intops is                        -- intops.ads
   procedure Inc (X : in out Integer);
   pragma Inline (Inc);
end Intops;

package body Intops is                   -- intops.adb
   procedure Inc (X : in out Integer) is
   begin
      X := X + 1;
   end Inc;
end Intops;

procedure Test_Inc0  is                  -- test_inc0.adb
   X : Integer := 0;
begin
   Inc (X);
end Test_Inc0;

The following analysis:

gnatcov coverage --level=insn --routines=_test_inc0 --annotate=xcov+ test_inc0.trace

… requests, with --routines, to report about the Test_Inc0 procedure only, so we intuitively expect a single test_inc0.adb.xcov annotated source result. If the Inc(X) call in Test_Inc0 is inlined, however, the command actually produces an intops.adb.xcov report as well because the object code of Test_Inc0 also contains inlined code coming from the other unit.

For generic units, information for all the instances is aggregated on the generic source, so each line annotation is a super synthesis of the coverage achieved for all the instructions attached to this line through all the instances.

Let us consider the generic Ada unit below to illustrate:

generic
   type Num_T is range <>;
package Genpos is
   procedure Count (X : Num_T);
   --  Increment N_Positive is X > 0

   N_Positive : Natural := 0;
   --  Number of positive values passed to Count
end Genpos;

package body Genpos is
   procedure Count (X : Num_T) is
   begin
      if X > 0 then
         N_Positive := N_Positive + 1;
      end if;
   end Count;
end Genpos;

Then two distinct instances in their own package, producing separate object code for each instance:

package POSI is
   type T1 is new Integer;
   package Pos_T1 is new Genpos (Num_T => T1);

   type T2 is new Integer;
   package Pos_T2 is new Genpos (Num_T => T2);
end POSI;

And now a simple test driver that executes all the code for Count in the first instance (going within the if statement), and only part of the code for Count in the second instance (not going within the if statement):

procedure Test_Genpos is
begin
   Pos_T1.Count (X => 1);
   Pos_T2.Count (X => -1);
end Test_Genpos;

The precise insn coverage difference is first visible in the =asm report. The conditioned part of Count clearly shows up as uncovered in the Pos_T2 instance (- at offset 204 and on), while it is reported covered as expected in the Pos_T1 instance (+ at offset 1b4 and on):

posi__pos_t1__count +: 1ac-1e7
1ac +:  2f 80 00 00      cmpiw  cr7,r0,0x0000
1b0 +:  40 9d 00 24      ble-   cr7,0x1d4 <posi__pos_t1__count+0000003c>
1b4 +:  3c 00 00 00      lis    r0,0x0000  | cond branch not taken,
...                                        | fallthrough down to 1d4
...                                        v
1d4 +:  60 00 00 00      ori    r0,r0,0x0000
...
posi__pos_t2__count !: 1fc-237
1fc +:  2f 80 00 00      cmpiw  cr7,r0,0x0000
200 +:  40 9d 00 24      ble-   cr7,0x224 <posi__pos_t2__count+0000003c>
204 -:  3c 00 00 00      lis    r0,0x0000  | cond branch taken,
...                                        | skip everything up to 224
...                                        v
224 +:  60 00 00 00      ori    r0,r0,0x0000
...

The presence of uncovered instructions yields a partial coverage annotation for the corresponding source line in the =xcov output (! on line 10):

 6 .: package body Genpos is
 7 +:    procedure Count (X : Num_T) is
 8 .:    begin
 9 +:       if X > 0 then
10 !:          N_Positive := N_Positive + 1;
11 .:       end if;
12 +:    end Count;
13 .: end Genpos;

And the =xcov+ (or =html) output gathers everything together, with the blocks of instructions coming from different instances identifiable by the associated object symbol names:

  10 !:          N_Positive := N_Positive + 1;
<posi__pos_t1__count+0000001c>:+
1b4 +:  3c 00 00 00  lis    r0,0x0000
...
<posi__pos_t2__count+0000001c>:-
204 -:  3c 00 00 00  lis    r0,0x0000
...

Full object coverage considerations

The previous sections focus on the coverage analysis of code attached to symbols. When full object level coverage is to be reached, extra care is required regarding orphaned code regions, which are not attached to any symbol, and empty symbols, for which the reported code size is null.

Orphaned regions usually show up out of legitimate code alignment requests issued for performance or target ABI specificities. Empty symbols most often result from low level assembly programmed parts missing the assembly directives aimed at populating the symbol table. Both are typically harmless so information about them is only emitted on explicit request. gnatcov provides the scan-objects command for this purpose. The command expects the set of object files to examine on the command line, as a sequence of either object file or @listfile argument, and reports about the two kinds of situations described above.