Coverage Exemptions¶
In some circumstances, there are good and well understood reasons why proper coverage of some source construct is not achievable. The GNATcoverage exemptions facility was designed to allow abstracting these coverage violations away from the genuine defects of a testing campaign.
Defining Exemption Regions¶
Exemption regions are lexical sections of sources in which coverage violations are expected and can be justified. For Ada with the GNAT compilers, regions are defined by the insertion of dedicated pragmas in the sources:
pragma Annotate (Xcov, Exempt_On, "justification text");
starts a region, providing some justification text that will be recalled in coverage reports.pragma Annotate (Xcov, Exempt_Off);
closes the current exemption region.
There may be no overlap between exemption regions.
To illustrate, let us consider a common assertion control procedure in Ada:
procedure Eassert (T : Boolean) is
begin
pragma Annotate (Xcov, Exempt_On, "assert condition is never False");
if not T then
raise Program_Error;
end if;
pragma Annotate (Xcov, Exempt_Off);
end Eassert;
We expect never to reach here with T
False, so we declare an exemption
region to state that coverage violations across this region are expected and
not a testing campaign deficiency.
Reporting about coverage exemptions¶
Exempted regions are reported as blocks in both the annotated source and the
synthetic text reports, for both source and object coverage metrics.
In annotated source reports, a #
or *
caracter annotates all the
exempted lines, depending on whether 0 or at least 1 violation was exempted
over the whole section, respectively.
For our Eassert
example above, a typical =xcov
output
for stmt+decision
coverage for would be:
6 .: procedure Eassert (T : Boolean) is
7 .: begin
8 *: pragma Annotate (Xcov, Exempt_On, "assert condition never to be False");
9 *: if not T then
10 *: raise Program_Error;
11 *: end if;
12 *: pragma Annotate (Xcov, Exempt_Off);
13 .: end Eassert;
The whole block is marked with *
annotations to indicate that some
violations were actually exempted; 2 in this case: the statement coverage
violation for the raise
and the decision coverage violation for the if
control.
In synthetic text reports, a single indication is emitted for each exempted region as a whole, and the indications for all the regions are grouped in a separate Exempted Regions report section, only present if there are exemption regions in the analysis scope. This section lists the exempted regions, displaying for each the source location span, the number of actually exempted violations in the region, and the exemption justification text. It also includes a total count of the number of exempted regions at the end.
The corresponding =report
excerpt below illustrates
this for the Eassert
example:
...
=========================
== 3. EXEMPTED REGIONS ==
=========================
eassert.adb:8:4-12:4: 2 exempted violations, justification:
assert condition never to be False
1 exempted region.
=========================
== 4. ANALYSIS SUMMARY ==
=========================
No non-exempted STMT violation.
No non-exempted DECISION violation.
1 exempted region.
The Coverage Violations section is renamed to convey that it contains “NON-EXEMPTED” violations only, and the Analysis Summary counters are adjusted in a similar manner. The number of exempted regions is added to the list of counters in this section.
If the executed tests actually trigger an assertion failure, there is no coverage violation to be exempted any more and this translates as visible differences in the reports:
In annotated sources, the region is annotated with #
signs instead of
*
, as in:
6 .: procedure Eassert (T : Boolean) is
7 .: begin
8 #: pragma Annotate (Xcov, Exempt_On, "assert condition never to be False");
9 #: if not T then
10 #: raise Program_Error;
11 #: end if;
12 #: pragma Annotate (Xcov, Exempt_Off);
13 .: end Eassert;
In synthetic reports, the count of exempted violations is 0, like:
=========================
== 3. EXEMPTED REGIONS ==
=========================
eassert.adb:8:4-12:4: 0 exempted violation, justification:
assert condition never to be False
1 exempted region.
Object coverage exemptions¶
Exemption regions specified via annotations in source files actually apply to both source and object level criteria analyzed over the annotated regions.
In the previous example, we would have used similar exemption annotations to
deal with expected object instruction and branch coverage failures in Eassert,
as the conditional branch used to implement the if
statement is expected
to remain partially covered, as well as the sequence of machine instructions
triggerring the Ada exception raise.
As for Source Coverage Obligations for source level criteria, information about the declared exemption regions is located in the Library Information files produced by the compiler for every compilation unit. The mechanisms described in chapter Specifying Units Of Interest can then also be used to designate units for which exemptions regions should be accounted for.