Object coverage analysis with gnatcov coverage
Object coverage analysis computes metrics focused on machine-level object code, concerned with machine basic instructions or conditional branches. Conditional general instructions (i.e.non-branch instructions), such as ARM conditional instructions are not supported. If such instructions are present in the subprograms to be analyzed, their reported coverage status may be incorrect.
On request, the metrics can be presented on sources, with an annotation on each
line synthesizing the coverage status of all the instructions generated for
this line. This mapping relies on debug information, so sources must be
compiled with -g
for this to work. There is no further
compilation requirement for object coverage alone.
Once your application is built, the analysis proceeds in two steps: gnatcov run is
used to produce execution traces, then gnatcov coverage to generate coverage reports.
Object coverage is queried by passing a specific --level
argument to gnatcov coverage; =insn
or =branch
, described
in detail in the following sections. As for source coverage, there is never a
requirement to recompile just because a different criterion needs to be
analyzed.
The Producing binary traces with gnatcov run chapter of this document provides details on the trace production interface. The remainder of this chapter explains the use of gnatcov coverage in particular, to analyse traces once they have been produced. The general command line structure is always like:
gnatcov coverage --level=<criterion> --annotate=<format>
[--routines=<names>] ... <traces>
The optional --routines
argument provides the set of object level
subprogram names on which the analysis should focus. This set defaults to the
full set of symbols defined by all the executables associated with the provided
execution traces.
Later in this chapter, Focusing on subprograms of interest (--routines) explains how to construct the relevant
list of names for --routines
. Prior to this comes a section
describing the available report formats, then more
details regarding Object Instruction Coverage analysis (--level=insn), Object Branch Coverage analysis (--level=branch), and specificities
regarding Inlining & Ada Generic units. Finally, Full object coverage considerations describes tools that
help analyze low-level object files for issues of interest when aiming at full
object coverage.
gnatcov coverage command line
Coverage analysis with GNATcoverage is performed by invoking gnatcov coverage for a set of
critera queried via the --level
command line option. The general
interface synopsis is available from gnatcov --help
:
gnatcov coverage OPTIONS TRACE_FILES
The available options are as follows:
-c
,--level
(mandatory):Tell the set of coverage criteria to be assessed. The possible values for object coverage analysis are
insn
,branch
, both explained later in this chapter.-a
,--annotate
(mandatory):Request a specific output report format. The two possible criteria support
xcov[+]
,html
, andasm
, with interpretations that vary depending on the assessed criteria. See the corresponding documentation later in this chapter for more details. This option accepts comma separated values and/or can be specified multiple times on the command line, in which case there will be one report produced for each of the requested annotation formats.
--timezone
:Select the timezone to use when displaying trace creation dates in reports, which could be either
local
orutc
. The local timezone is used by default.--output-dir
:Request that the report files (index and annotated sources for the
xcov
andhtml
output formats) be output in the provided directory. If not specified, the default is the root project’s object directory if using projects, and the current directory if not. If more than one of the above annotation formats is requested, then each report will be placed in a subdirectory named accordingly.--report-title
:Request that generated HTML documents (index for the
html
output format) are assigned a customized title. For instance, passing--report-title="Project ABC"
will yield titles such as:Project ABC - GNATcoverage Report
. If passed multiple times, passing an empty string last will restore the default behavior. This option is ignored is the selected output format does not support titles.-T
,--trace
(mandatory), possibly repeated and accepting @listfile arguments :Provide the set of execution traces for which a report is to be produced. When multiple traces are provided, gnatcov produces a consolidated result, as described in detail in the Coverage Consolidation chapter of this manual.
--exec
:Override executable from traces. Trace files contain an indication of the executable used to generate them. This option causes the named executable to be loaded for coverage analysis, and to override the indication contained in any trace specified after it on the command line. An empty executable name may be specified to restore the default behaviour of using the indication contained in each trace file. Note that
--exec
may appear last on the command line, in which case it applies to no trace file, but still causes the indicated executable to be included in the coverage analysis. This ensures that any code in that executable that is not exercised by some trace file will be reported as not covered.
--routines
, possibly repeated and accepting @listfile arguments:Provide the list of object symbol names that correspond to routines for which the coverage assessment is to be performed. Each instance of this option on the command line adds to what is to be assessed eventually. See the Focusing on subprograms of interest (--routines) section for extra details and use examples.
--alis
, possibly repeated and accepting @listfile arguments:Provide set of Library Information files for units where there might be applicable exemption regions to account for, as explained in the Object coverage exemptions section of this manual.
-P
:Use the indicated project file to find default options. This can also be used, possibly combined with other project related switches, as an alternative to
--alis
to determine the set of units for which coverage exemptions should be honored. In this case, the target/runtime/scenario contextual options that would be passed to build the project should also be passed for proper interpretation of the project files. This is similar to the use of project facilities for the determination of SCOs for source coverage analysis, and described in the Specifying Units Of Interest chapter of this manual.-t
,--target
:State the target toolchain configuration used to build the analyzed programs, as provided to
gprbuild
. For cross or 32bit native configurations, this switch together with its possible--RTS
companion is required for all commands using project files unless the root project provides the information withTarget
/Runtime
attributes.
Output report formats (--annotate
)
Object coverage reports may be produced in various formats, as requested with
the --annotate
command line argument of gnatcov coverage.
The asm
format produces an annotated assembly output, with a
coverage indication attached to every single instruction. This is the base
information of interest to object coverage analysis, simply presented in
different manners through the other possible output formats. The
xcov
and html
formats produce a set of annotated
source files, in the directory where gnatcov is launched unless overriden with a
--output-dir
option. Even though presented on sources, the
annotations remain representative of object coverage metrics, synthesized for
all the instructions associated with each source line.
Later in this chapter we name output formats by the text to add to
--annotate
on the command line. For example, we use “the
=asm
outputs” to mean “the coverage reports produced with
--annotate=asm
”. We also sometimes use in-source reports or
outputs to designate the set of outputs in annotated source forms.
We illustrate the various formats with coverage analysis excerpts on the following example Ada support unit:
-- raise Program_Error if T is False. Do nothing otherwise.
procedure Assert (T : Boolean) is
begin
if not T then
raise Program_Error;
end if;
end Assert;
As the contents suggest, this subprogram is expected never to be called with T False in nominal situations.
Machine level reports (=asm
)
For object coverage analysis, --annotate=asm
produces annotated
assembly code for all the selected routines on standard output. The
annotations are first visible as a special character on each machine code line
to convey the coverage status of the corresponding instruction. The following
output excerpt, for example, is part of a coverage report for our Assert
subprogram compiled for the PowerPc architecture:
Coverage level: branch
_ada_assert !: 0c4-123
0c4 +: 94 21 ff e0 stwu r1,-0x0020(r1)
...
0ec +: 2f 80 00 00 cmpiw cr7,r0,0x0000
0f0 +: 41 9e 00 18 beq- cr7,0x108 <_ada_assert+00000044>
...
100 -: 38 80 00 04 li r4,0x0004
104 -: 48 00 00 a1 bl 0x1a4 <__gnat_last_chance_handler>
108 +: 60 00 00 00 ori r0,r0,0x0000
...
120 +: 4e 80 00 20 blr
A -
annotation for a line always conveys that the instruction was not
executed at all. The instruction is also said to be uncovered in this
case. Conversely, a +
means that the instruction is fully covered with
respect to the analyzed criterion, with a meaning which depends on both the
criterion and the kind of instruction – whether the instruction is a
conditional branch and whether we are doing mere instruction or object branch
coverage analysis. Annotations conveying partial coverage might show up as
well, also depending on the criterion and kind of instruction.
More details on the instruction specific annotations are provided in the sections that follow. Then, as the first line of the example suggests, the report also annotates each subprogram symbol as a whole, with the range of addresses that the subprogram spans and a synthetic coverage indication according to the following table:
Symbol Annotation |
Meaning |
---|---|
|
All the subprogram instructions are uncovered (none executed) |
|
All the subprogram instructions are fully covered |
|
Some of the subprogram instructions were fully or partially covered |
In our example, the code features both fully covered and uncovered
instructions, and the _assert
symbol as a whole is marked partially
covered with a !
annotation.
Annotated sources, text (=xcov[+]
)
For object coverage analysis, --annotate=xcov
produces annotated
source files with the .xcov
extension, one per original compilation unit in
the selected output directory.
The annotations are visible at the beginning of every source line, as a single character which synthesizes the coverage status of all the machine instructions generated for this line. The following table provides a uniform description of this synthesis for all the object level criteria:
Source Annotation |
Meaning |
---|---|
|
no machine code associated with this line |
|
all the instructions associated with the line are |
|
all the instructions associated with the line are |
|
otherwise |
The report also includes a short header, which features a global coverage count with respect to the total number of lines with associated code, as well as an indication of the assessed criterion. Below is an example of report obtained for our Assert unit:
examples/src/assert.adb:
75% of 4 lines covered
Coverage level: insn
1 +: procedure Assert (T : Boolean) is
2 .: begin
3 +: if not T then
4 -: raise Program_Error;
5 .: end if;
6 +: end Assert;
To lines with associated object code we apply qualifiers similar to those
for individual instructions: when the synthetic coverage indication for a line
is -
, +
or !
, we qualify the line as uncovered, fully covered,
or partially covered, respectively. Note that even though they are rendered
on source lines, the annotations are really meant to convey object code
properties here, hence are of a different nature than what the DO-178B source
structural coverage criteria refer to. See our Object/Source level metrics considerations section for
further details on this aspect.
With --annotate=xcov+
(extra +
at the end), the machine
instructions and their individual coverage status are printed next to their
associated source line.
Annotated sources, html (=html
)
For object coverage criteria, gnatcov coverage --annotate=html
produces an
annotated version of each source file. Each annotated source page contains a
summary of the assessment results followed by the original source lines, all
numbered and marked with a coverage annotation as in the
--annotate=xcov
case. In addition, lines with obligations are
colorized in green, orange or red for +
, !
or -
coverage
respectively. An index.html page is also produced, which contains a
description of the assessment context (assessed criteria, set of trace files
involved, …) and a summary of the coverage results for all the units, with
links to their annotated sources.
Similarily to the xcov
format case, --annotate=html
attaches to each line details about the coverage status of all the individual
instructions generated for the line. These are folded within the line and
expanded when a mouse click hits it.
Object Instruction Coverage analysis (--level=insn
)
Object Instruction Coverage treats basic and conditional branch instructions
identically, as either executed or not, hence fully covered or uncovered. The
=asm
instruction annotations are as follows:
Insn Annotation |
Meaning |
---|---|
|
the instruction was not executed |
|
the instruction was executed |
The =asm
excerpt below provides a representative example of the
PowerPC instruction coverage achieved for our Assert
procedure by nominal
executions where the subprogram is never called with T False:
_ada_assert !: 0c4-123
0c4 +: 94 21 ff e0 stwu r1,-0x0020(r1)
...
0ec +: 2f 80 00 00 cmpiw cr7,r0,0x0000
0f0 +: 41 9e 00 18 beq- cr7,0x108 <_ada_assert+00000044>
...
100 -: 38 80 00 04 li r4,0x0004
104 -: 48 00 00 a1 bl 0x1a4 <__gnat_last_chance_handler>
108 +: 60 00 00 00 ori r0,r0,0x0000
...
120 +: 4e 80 00 20 blr
Expectedly, the coverage annotations report all the instructions as executed
except the two issuing the call to __gnat_last_chance_handler
, which
correspond to the raise
statement in the GNAT high integrity profiles
without exception propagation support. The two instructions at offsets 0ec and
0f0 are the comparison and branch conditioned on the comparison result that
implement the if construct. We note here that the conditional branch is
reported fully covered, as merely executed, even though always taken.
The corresponding =xcov
output follows:
1 +: procedure Assert (T : Boolean) is
2 .: begin
3 +: if not T then
4 -: raise Program_Error;
5 .: end if;
6 +: end Assert;
The annotations on lines 3 and 4 correspond to immediate expectations from
comments we made on the =asm
output. We can also observe
annotations on lines 1 and 6, to which the subprogram prologue and epilogue
code is attached, and executed as soon as the procedure is called.
Object Branch Coverage analysis (--level=branch
)
Object Branch Coverage treats basic and conditional branch instructions
differently. Basic instructions are considered fully covered as soon as
executed, as in the Instruction Coverage case. Conditional branches, however,
have to be executed at least twice to be claimed fully covered : once taking
the branch and once executing fall-through, which we sometimes abusively refer
to as taken both ways even if one case actually corresponds to the
branch not being taken. The =asm
instruction annotations are as
follows:
Insn Annotation |
Meaning |
---|---|
|
the instruction was never executed |
|
the instruction was executed and taken both ways for a conditional branch |
|
the instruction is a conditional branch, executed and always taken |
|
the instruction is a conditional branch, executed and never taken |
The v
and >
annotations are representative of situations where a
conditional branch instruction is executed and taken one way only, which
constitutes partial coverage of the instruction.
An example of partial coverage is observable on our Assert case, where the conditional branch at offset 0f0 is always taken, jumping over the raise statement code as expected for nominal executions:
_ada_assert !: 0c4-123
0c4 +: 94 21 ff e0 stwu r1,-0x0020(r1)
...
0ec +: 2f 80 00 00 cmpiw cr7,r0,0x0000
0f0 >: 41 9e 00 18 beq- cr7,0x108 <_ada_assert+00000044>
...
100 -: 38 80 00 04 li r4,0x0004
104 -: 48 00 00 a1 bl 0x1a4 <__gnat_last_chance_handler>
108 +: 60 00 00 00 ori r0,r0,0x0000
...
120 +: 4e 80 00 20 blr
The corresponding =xcov
output follows:
examples/src/assert.adb:
50% of 4 lines covered
Coverage level: branch
1 +: procedure Assert (T : Boolean) is
2 .: begin
3 !: if not T then
4 -: raise Program_Error;
5 .: end if;
6 +: end Assert;
The partial branch coverage logically translates into a partial coverage
annotation on the line to which the branch is attached, here the line of the
if statement that the conditional branch implements. This is confirmed by the
=xcov+
output, where the individual instructions are visible as
well together with their own coverage indications:
examples/src/assert.adb:
Coverage level: branch
1 +: procedure Assert (T : Boolean) is
<_ada_assert+00000000>:+
0c4 +: 94 21 ff e0 stwu r1,-0x0020(r1)
...
0dc +: 98 1f 00 08 stb r0,0x0008(r31)
2 .: begin
3 !: if not T then
<_ada_assert+0000001c>:!
0e0 +: 88 1f 00 08 lbz r0,0x0008(r31)
...
0ec +: 2f 80 00 00 cmpiw cr7,r0,0x0000
0f0 >: 41 9e 00 18 beq- cr7,0x108 <_ada_assert+00000044>
4 -: raise Program_Error;
<_ada_assert+00000030>:-
0f4 -: 3c 00 ff f0 lis r0,-0x0010
...
104 -: 48 00 00 a1 bl 0x1a4 <__gnat_last_chance_handler>
5 .: end if;
6 +: end Assert;
<_ada_assert+00000044>:+
...
120 +: 4e 80 00 20 blr
Focusing on subprograms of interest (--routines
)
By default, in absence of a --routines
argument to gnatcov coverage,
object coverage results are produced for the full set of subprogram symbols
defined by the executables designated by the analyzed traces.
--routines
allows the specification of a set of subprogram
symbols of interest so reports refer to this (sub)set only. Each occurrence of
--routines
on the command line expects a single argument which
specifies a subset of symbols of interest. Multiple occurrences are allowed and
the subsets accumulate. The argument might be either a single symbol name or a
@listfile argument expected to contain a list of symbol names.
For example, focusing on three symbols sym1
, sym2
and sym3
can be
achieved with either one of the following set of --routines
combinations:
--routines=sym1 --routines=sym2 --routines=sym3
or --routines=@symlist123
or --routines=sym3 --routines=@symlist12
… provided a symlist12
text file containing the first two symbol names
and a symlist123
text file containing the three of them.
It is often convenient to compute the lists of symbols for a @listfile
argument, for example as “the full set of defined subprograms except those
with test_
or harness_
at the beginning of their name”. gnatcov provides
the gnatcov disp-routines sub-command for this purpose.
The general synopsis of gnatcov disp-routines is as follows:
disp-routines [--exclude|--include] FILES
Build a list of routines from object files
gnatcov disp-routines outputs the list of symbols in a set built from object files provided on the command line. Object file is to be taken in the general sense here, as conforming to a supported object file format, typically ELF, so includes executable files as well as single compilation unit objects.
The output set is built incrementally while processing the arguments left to
right. --include
states “from now on, until contradicted, symbols
defined in object files are added to the result set”, and
--exclude
states “from now on, until contradicted, symbols
defined in object files are removed from the result set”. An implicit
--include
is assumed right at the beginning, and each argument
may be either the direct name of an object file or a @listfile argument
containing a list of such names.
Below are a few examples of commands together with a description of the set they build:
$ gnatcov disp-routines explore
# (symbols defined in the 'explore' executable)
$ gnatcov disp-routines explore --exclude test_stations.o
# (symbols from the 'explore' executable)
# - (symbols from the 'test_stations.o' object file)
$ gnatcov disp-routines --include @sl1 --exclude @sl2 --include @sl3
# (symbols from the object files listed in text file sl1)
# - (symbols from the object files listed in text file sl2)
# + (symbols from the object files listed in text file sl3)
Annotated source reports, when requested, are generated for sources associated with the selected symbols’ object code via debug information, and coverage annotations are produced only on the corresponding. Inlining can have surprising effects in this context, as the following section describes in greater details.
Inlining & Ada Generic units
The generated code for an inlined subprogram call or a generic instantiation
materializes two distinct source entities: the expanded source (of the inlined
subprogram or of the instanciated generic body) and the expansion request (the
subprogram call or the generic instanciation). While this is of no consequence
for =asm
outputs, which just report coverage of raw machine
instructions within their object level subprograms, regardless of the object
code origin, this raises a few points of note for in-source outputs.
For inlined calls, potentially surprising results might show up when a specific set of object routines is queried. Indeed, when the code for a symbol A in unit Ua embeds code inlined from unit Ub, a request for an annotated source report for routine A, intuitively expected to yield a report for Ua only, will typically produce an output file for Ub as well, for lines referenced by the machine code inlined in A.
Consider the following Ada units for example, with a functional unit in
intops.ads
and intops.adb
, then a test driver in test_inc0.adb
:
package Intops is -- intops.ads
procedure Inc (X : in out Integer);
pragma Inline (Inc);
end Intops;
package body Intops is -- intops.adb
procedure Inc (X : in out Integer) is
begin
X := X + 1;
end Inc;
end Intops;
procedure Test_Inc0 is -- test_inc0.adb
X : Integer := 0;
begin
Inc (X);
end Test_Inc0;
The following analysis:
gnatcov coverage --level=insn --routines=_test_inc0 --annotate=xcov+ test_inc0.trace
… requests, with --routines
, to report about the Test_Inc0
procedure only, so we intuitively expect a single test_inc0.adb.xcov
annotated source result. If the Inc(X)
call in Test_Inc0 is inlined,
however, the command actually produces an intops.adb.xcov
report as well
because the object code of Test_Inc0 also contains inlined code coming from the
other unit.
For generic units, information for all the instances is aggregated on the generic source, so each line annotation is a super synthesis of the coverage achieved for all the instructions attached to this line through all the instances.
Let us consider the generic Ada unit below to illustrate:
generic
type Num_T is range <>;
package Genpos is
procedure Count (X : Num_T);
-- Increment N_Positive is X > 0
N_Positive : Natural := 0;
-- Number of positive values passed to Count
end Genpos;
package body Genpos is
procedure Count (X : Num_T) is
begin
if X > 0 then
N_Positive := N_Positive + 1;
end if;
end Count;
end Genpos;
Then two distinct instances in their own package, producing separate object code for each instance:
package POSI is
type T1 is new Integer;
package Pos_T1 is new Genpos (Num_T => T1);
type T2 is new Integer;
package Pos_T2 is new Genpos (Num_T => T2);
end POSI;
And now a simple test driver that executes all the code for Count
in the
first instance (going within the if statement), and only part of the code
for Count
in the second instance (not going within the if statement):
procedure Test_Genpos is
begin
Pos_T1.Count (X => 1);
Pos_T2.Count (X => -1);
end Test_Genpos;
The precise insn
coverage difference is first visible in the
=asm
report. The conditioned part of Count
clearly shows up
as uncovered in the Pos_T2
instance (-
at offset 204 and on), while it
is reported covered as expected in the Pos_T1
instance (+
at offset 1b4
and on):
posi__pos_t1__count +: 1ac-1e7
1ac +: 2f 80 00 00 cmpiw cr7,r0,0x0000
1b0 +: 40 9d 00 24 ble- cr7,0x1d4 <posi__pos_t1__count+0000003c>
1b4 +: 3c 00 00 00 lis r0,0x0000 | cond branch not taken,
... | fallthrough down to 1d4
... v
1d4 +: 60 00 00 00 ori r0,r0,0x0000
...
posi__pos_t2__count !: 1fc-237
1fc +: 2f 80 00 00 cmpiw cr7,r0,0x0000
200 +: 40 9d 00 24 ble- cr7,0x224 <posi__pos_t2__count+0000003c>
204 -: 3c 00 00 00 lis r0,0x0000 | cond branch taken,
... | skip everything up to 224
... v
224 +: 60 00 00 00 ori r0,r0,0x0000
...
The presence of uncovered instructions yields a partial coverage annotation for
the corresponding source line in the =xcov
output (!
on line
10):
6 .: package body Genpos is
7 +: procedure Count (X : Num_T) is
8 .: begin
9 +: if X > 0 then
10 !: N_Positive := N_Positive + 1;
11 .: end if;
12 +: end Count;
13 .: end Genpos;
And the =xcov+
(or =html
) output gathers
everything together, with the blocks of instructions coming from different
instances identifiable by the associated object symbol names:
10 !: N_Positive := N_Positive + 1;
<posi__pos_t1__count+0000001c>:+
1b4 +: 3c 00 00 00 lis r0,0x0000
...
<posi__pos_t2__count+0000001c>:-
204 -: 3c 00 00 00 lis r0,0x0000
...
Full object coverage considerations
The previous sections focus on the coverage analysis of code attached to symbols. When full object level coverage is to be reached, extra care is required regarding orphaned code regions, which are not attached to any symbol, and empty symbols, for which the reported code size is null.
Orphaned regions usually show up out of legitimate code alignment requests
issued for performance or target ABI specificities. Empty symbols most often
result from low level assembly programmed parts missing the assembly directives
aimed at populating the symbol table. Both are typically harmless so
information about them is only emitted on explicit request. gnatcov provides the
scan-objects
command for this purpose. The command expects the
set of object files to examine on the command line, as a sequence of either
object file or @listfile argument, and reports about the two kinds of
situations described above.