Coverage Consolidation¶

Coverage consolidation is the facility allowing the computation of the overall coverage achieved by a set of executions. Consolidation is queried by passing the corresponding set of execution traces to gnatcov coverage, which produces a single coverage report as a result. The focus of the analysis must be specified, via –scos or project files for source coverage, or via –routines for object coverage.

A typical case where consolidation is useful is when some part of an application depends on external inputs and several executions are required to exercise different scenarios in the application program. The execution traces to consolidate are obtained from the same executable in this case.

Another common situation is when execution of different executables is needed to achieve the required coverage for a software, either because distinct software modules are tested independently (e.g. the different units of a library), or because different aspects of the behavior of modules are tested separately (e.g. the different subprograms of a library unit or different scenarios of a given subprogram).

A simple example is provided for each of these cases in the following sections.

Example 1: consolidation over a single program¶

Consider the example C program below, offering a simple command line interface to perform very basic math operations. This is splitted in two main source files: process.c doing the computation and displaying the result, and main.c for the main entry point and basic usage control:

#include <stdio.h>        /* main.c */
#include <assert.h>
#include "process.h"

void usage ()
{
  printf ("calc <int1> <int2> <op>, print result of <int1> <op> <int2>\n");
}

int main (int argc, const char * argv[])
{
  if (argc != 4)
    {
      usage ();
      exit (1);
    }

  process (argv);
  return 0;
}

#include <stdio.h>        /* process.c */
#include <assert.h>
#include "process.h"

void process (const char * argv[])
{
  int x = atoi (argv[1]), y = atoi (argv[2]);
  char opcode = argv[3][0];

  int result;

  switch (opcode)
    {
    case '*':
      result = x * y;
      break;
    case '+':
      result = x + y;
      break;
    default:
      printf ("unsupported opcode %c\n", opcode);
      return;
    }

  printf ("%d %c %d = %d\n", x, opcode, y, result);
}

#ifndef __PROCESS_H__     /* process.h */
#define __PROCESS_H__
extern void process (const char * argv[]);
#endif

Here is a sequence of compilation/executions for various use cases, on a native system where command line arguments for the program are supported by gnatcov run. Each execution is requested to produce a specific trace file:

gcc -o calc main.c process.c -g -fpreserve-control-flow -fdump-scos
gnatcov run --output=mult.trace -eargs ./calc 6 5 '*'
gnatcov run --output=plus.trace -eargs ./calc 2 3 '+'
gnatcov run --output=div.trace -eargs ./calc 2 3 '/'
gnatcov run --output=misuse.trace -eargs ./calc

Now we can use gnatcov coverage to assess the coverage achieved by arbitrary combinations of the executions, just by passing the corresponding traces. For example, combining the two executions exercising the * and + computations for statement coverage can be achieved with:

gnatcov coverage --scos=main.c.gli --scos=process.c.gli \
   --annotate=xcov --level=stmt mult.trace plus.trace

And this yields reports in main.c.xcov and process.c.xcov like:

 ...
.: void usage ()
.: {
-:   printf ("calc <i1> <i2> <op>, print result of <i1> <op> <i2>\n");
.: }
.:
.: int main (int argc, const char * argv[])
.: {
+:   if (argc != 4)
.:     {
-:       usage ();
-:       exit (1);
.:     }
.:
+:   process (argv);
+:   return 0;
.: }

 ...
.: void process (const char * argv[])
.: {
+:   int x = atoi (argv[1]), y = atoi (argv[2]);
+:   char opcode = argv[3][0];
.:
+:   int result;
.:
+:   switch (opcode)
.:     {
.:     case '*':
+:       result = x * y;
+:       break;
.:     case '+':
+:       result = x + y;
+:       break;
.:     default:
-:       printf ("unsupported opcode %c\n", opcode);
-:       return;
.:     }
.:
+:   printf ("%d %c %d = %d\n", x, opcode, y, result);
.: }

We observe a reported absence of coverage for statements corresponding to the treatment of two kinds of usage error: wrong number of command line arguments, visible on lines 7, 14, and 15 of main.c, and attempt to compute an unsupported operation, visible on lines 21 and 22 of process.c. These two scenarios, exercised through div.trace and misuse.trace were indeed not included in the consolidation scope.

Example 2: consolidation over a single unit by different programs¶

We will consider achieving statement coverage of the following example Ada unit, which implements part of a robot controller able to send actuator commands depending on what a front sensor perceives is ahead of the robot:

package Commands is
   type Command is (Step, Hold);
   type Perceived is (Room, Rock, Pit);

   function Safe (Cmd : Command; Front : Perceived) return Boolean;
   --  Whether executing CMD is safe with FRONT perceived ahead

   N_Safe, N_Unsafe : Integer := 0;
   --  Count the number of safe/unsafe cases we have evaluated
end Commands;

package body Commands is

   procedure Stat (Safe : Boolean) is
   begin
      if Safe then
         N_Safe := N_Safe + 1;
      else
         N_Unsafe := N_Unsafe + 1;
      end if;
   end Stat;

   function Safe (Cmd : Command; Front : Perceived) return Boolean is

      --  Standing straight is always safe, and any other action is
      --  safe as soon as there is room ahead:
      Result : constant Boolean := Cmd = Hold or else Front = Room;
   begin
      Stat (Result);
      return Result;
   end Safe;
end Commands;

We test the Commands body by combining two sorts of drivers. The first one exercises safe commands only:

procedure Test_Cmd_Safe is
begin
   --  Remaining still is always safe, as is stepping with room ahead:
   Assert (Safe (Cmd => Hold, Front => Rock));
   Assert (Safe (Cmd => Hold, Front => Pit));
   Assert (Safe (Cmd => Step, Front => Room));
end Test_Cmd_Safe;

Running this first program and analysing the achieved coverage would be something as follows:

gnatcov run test_cmd_safe   # produces test_cmd_safe.trace
gnatcov coverage --level=stmt --scos=commands.ali --annotate=xcov test_cmd_safe.trace

Producing a commands.adb.xcov report with:

.:    procedure Stat (Safe : Boolean) is
.:    begin
+:       if Safe then
+:          N_Safe := N_Safe + 1;
.:       else
-:          N_Unsafe := N_Unsafe + 1;
.:       end if;
.:    end Stat;

In accordance with the testcase strategy, aimed at exercising safe situations only, everything is statement covered except the code specific to unsafe situations, here the counter update on line 11. Now comes the other driver, exercising cases where the Safe function is expected to return False:

procedure Test_Cmd_Unsafe is
begin
   --  Stepping forward without room ahead is always unsafe
   Assert (not Safe (Cmd => Step, Front => Rock));
   Assert (not Safe (Cmd => Step, Front => Pit));
end Test_Cmd_Unsafe;

This one alone produces the symmetric commands.adb.xcov report, with:

.:    procedure Stat (Safe : Boolean) is
.:    begin
+:       if Safe then
-:          N_Safe := N_Safe + 1;
.:       else
+:          N_Unsafe := N_Unsafe + 1;
.:       end if;
.:    end Stat;

There again, the coverage results are in accordance with the intent, testing everything except the parts specific to safe situations. The combination of the two drivers was intended to achieve a pretty complete testing of the provided functionality, and the corresponding coverage can be computed thanks to the GNATcoverage consolidation facility, by simply providing the two execution traces to gnatcov coverage, which indeed yields full statement coverage of the Commands package body:

gnatcov coverage [...] test_cmd_safe.trace test_cmd_unsafe.trace
...

.:    procedure Stat (Safe : Boolean) is
.:    begin
+:       if Safe then
+:          N_Safe := N_Safe + 1;
.:       else
+:          N_Unsafe := N_Unsafe + 1;
.:       end if;
.:    end Stat;

In this example, consolidation involved different programs with only partial object code overlap, as depicted on the following representation:

Overlapping executables¶

Consolidation actually doesn’t require overlapping: users might well, for example, consolidate results from different programs testing entirely disjoint sets of units. A typical situation where this would happen is when testing independent units of a library, as illustrated by the following example.

Example 3: consolidation over a library by different programs¶

This example is a nice opportunity to illustrate a possible use of project files to denote the units of interest, so we’ll provide more details on that aspect. Let us consider an example library composed of the following two Ada procedures, implemented in separate source files inc.adb and mult.adb:

procedure Inc (X : in out Integer; Amount : Integer) is   -- inc.adb
begin
   X := X + Amount;
end;

procedure Mult (X : in out Integer; Amount : Integer) is  -- mult.adb
begin
   X := X * Amount;
end;

We first build an archive library from these, using the gprbuild tool (part of the GNAT toolchain). We place the two sources in a libops subdirectory and use the libops.gpr example project file below at the toplevel:

library project Libops is

   for Library_Dir use "lib";     -- Request creation of lib/libops.a
   for Library_Kind use "static";
   for Library_Name use "ops";

   for Languages use ("Ada");     -- Sources are Ada, in libops/ subdir
   for Source_Dirs use ("libops");
   for Object_Dir use "obj";

   package Compiler is
      for default_switches ("Ada") use
         ("-g", "-fdump-scos", "-fpreserve-control-flow");
   end compiler;

end Libops;

gprbuild -Plibops builds the library with the proper compilation options, then we move on to unit tests. We write two different programs for this purpose:

with Inc, Assert;     -- test_inc.adb
procedure Test_Inc is
   X : Integer := 0;
begin
   Inc (X, 1);
   Assert (X = 1);
end;

with Mult, Assert;    -- test_mult.adb
procedure Test_Mult is
   X : Integer := 2;
begin
   Mult (X, 2);
   Assert (X = 4);
end;

We build the corresponding executables using gprbuild again, with the test.gpr project file below:

with "libops";  -- test.gpr

project Test is
  for Languages use ("Ada");
  for Object_Dir use "obj";

  package Compiler is
    for Default_Switches("Ada") use ("-fno-inline");
  end Compiler;
end Test;

gprbuild -Ptest.gpr test_inc.adb test_mult.adb

We’re not interested in the coverage of the test procedures themselves so we don’t need the coverage related compilation options. -fno-inline is enforced to make sure that the library object code really gets exercised and not an inlined version of it within the test harness.

Now we can run the tests and perform coverage analysis for various combinations. To begin with:

gnatcov run obj/test_inc   -- produces test_inc.trace
gnatcov run obj/test_mult  -- produces test_mult.trace

Then assessing the library statement coverage achieved by test_inc alone, as a violations report, would go as:

gnatcov coverage --level=stmt --annotate=report -Plibops test_inc.trace

Note the use of -Plibops to state that the library units are those of interest for our analysis. There is no reference to the mult unit at all in the test and all the associated statements are marked uncovered in this case:

2.1. STMT COVERAGE
------------------

mult.adb:3:4: statement not executed

1 violation.

Proper coverage of the library units is achieved by the two unit tests, which we can see by requesting the consolidated coverage achieved by the two executions:

gnatcov coverage --level=stmt --annotate=report -Plibops test_*.trace
...
2.1. STMT COVERAGE
------------------

No violation.

Processing of object code overlap during consolidation¶

For object or source level criteria, gnatcov computes the coverage achieved for the full set of routines or source units declared to be of interest amongst those exposed by the union of the exercised executables, as designated by the set of consolidated traces;

On symbols found to overlap across executables, gnatcov computes the combined coverage achieved by all the executions.

For the purpose of computing combined coverage achievements, two symbols are considered overlapping when all the following conditions are met:

Both symbols have identical names at the object level,
Both symbols have DWARF debug information attached to them,
According to this debug information, both symbols originate from the same compilation unit, denoted by the full path of the corresponding source file.

By this construction, a symbol missing debug information is never considered overlapping with any other symbol. Whatever coverage is achieved on such a symbol never gets combined with anything else and the only kind of report where the symbol coverage is exposed is the =asm assembly output for object level criteria.

Moreover, for object level coverage criteria, gnatcov coverage will issue a consolidation error when two symbols are found to overlap but have structurally different machine code, which happens for example when the same unit is compiled with different different optimization levels for different executables.

The set of traces involved in a computation is visible in various places:

In the Assessment Context section of =report outputs, where the command line is quoted and detailed information about each trace is provided (trace file name, timestamp, tag, ...)
In the =html index page, where the list of trace names and tags used to produce the report is provided.

Coverage Consolidation¶

Example 1: consolidation over a single program¶

Example 2: consolidation over a single unit by different programs¶

Example 3: consolidation over a library by different programs¶

Processing of object code overlap during consolidation¶

Table Of Contents

Previous topic

Next topic

This Page