2. GNATtest User’s Guide

gnattest tool is a utility that creates unit-test skeletons as well as a test driver infrastructure (harness). gnattest creates a skeleton for each visible subprogram in the packages under consideration when they do not exist already.

gnattest is a project-aware tool. A project file is mandatory for test driver generation. The project file package that can specify gnattest switches is named gnattest.

The user can choose to generate a single test driver that will run all individual tests, or separate test drivers for each test. The second option allows much greater flexibility in test execution environment, allows to benefit from parallel tests execution to increase performance, and provides stubbing support.

gnattest also has a mode of operation where it acts as the test aggregator when multiple test executables must be run, in particular when the separate test drivers were generated. In this mode it handles individual tests execution and upon completion reports the summary results of the test run.

In order to process source files from a project, gnattest has to semantically analyze the sources. Therefore, test skeletons can only be generated for legal Ada units. If a unit is dependent on other units, those units should be among the source files of the project or of other projects imported by this one. Note that it is no longer necessary to specify the Ada language version; gnattest can process Ada source code written in any version from Ada 83 onward without specifying any language version switch.

Generated skeletons and harnesses are based on the AUnit testing framework. AUnit is an Ada adaptation of the xxxUnit testing frameworks, similar to JUnit for Java or CppUnit for C++. While it is advised that gnattest users read the AUnit manual, deep knowledge of AUnit is not necessary for using gnattest.

2.1. Setting up the runtimes required by gnattest

gnattest depends on one or more runtime libraries in order to execute the test drivers.

As mentioned earlier, the test harness generated by gnattest is based on Aunit, as such AUnit should be installed and aunit.gpr must be on the project path. If no test input generation is used, and the version of GNATDAS is kept identical to that of GNAT Pro, then the Aunit library distributed with GNAT Pro will work out of the box and no further steps are needed.

Automatic test input generation and testcase serialization for integration with GNATfuzz also depend on a test generation runtime library, which should be compiled and made available on the GPR_PROJECT_PATH environment variable.

In order to build and install both Aunit and the test generation runtime library, the following command should be run once per toolchain installation.

$ gnattest setup [options] [-gargs gprbuild-options]

where the options are:

• --prefix=dir

  specifies the directory in which the libraries should be installed. It defaults to the install prefix of gnattest itself.

• --compiler-prefix

  instructs gnattest to install the runtime libraries in the toolchain installation directory. This removes the need for adding the installation directory to the GPR_PROJECT_PATH environment variable. It overrides --prefix.

• --target=target

  Specifies the target for which the runtime libraries should be compiled

• --RTS=runtime

  Specifies the Ada runtime library name or profile to use for compiling the test runtime libraries.

• --rts-profile=profile

  Specifies the AUnit runtime profile to build against. profile must be one of auto, full, zfp, zfp-cross, ravenscar, ravenscar-cert or cert. It defaults to auto, in which case the profile is inferred from --RTS and --target.

• --config=file

  Passes file to gprbuild as its configuration project.

• --tgen, --no-tgen

  Force or skip building the test generation runtime. See below for the default.

• -q, -v

  Quiet mode, and verbose mode, which echoes the commands being run.

• -gargs gprbuild-options

  Passes all the remaining arguments to gprbuild.

The AUnit library is always built and installed. The test generation runtime is additionally built when the Ada runtime profile in use is full, embedded or ravenscar and the compiler supports Ada 2022, which gnattest setup probes for; otherwise it is skipped. Pass --tgen to build it without probing the compiler, or --no-tgen to skip it altogether. Since test input generation needs that runtime, skipping it means the automatic test case generation features will not be usable.

If --compiler-prefix is not passed to the gnattest setup invocation, it is then necessary to add <installation_dir>/share/gpr to the GPR_PROJECT_PATH environment variable to make sure the generated test harness can be compiled.

This step needs only needs to be executed once for a given configuration. If any of the target,, the runtime profile, the version of GNAT Pro or the version of GNAT DAS change, then it is necessary to re-run gnattest setup to make sure the installed runtime libraries are up to date.

2.2. Running gnattest

There are two ways of running gnattest.

2.2.1. Framework Generation Mode

In this mode gnattest has the following command-line interface:

$ gnattest -Pprojname [ switches ] [ filename ]

where

• -Pprojname

  specifies the project defining the location of source files. This switch is required. See Selecting the sources to process for the rules that determine which of the project’s sources are processed when no file name is given on the command line.

  For the semantics of aggregate project processing by gnattest, see the Aggregate project handling section.

• filename

  is the name of the source file containing the library unit package declaration (the package “spec”) for which a test package will be created. The file name may be given with a path.

• switches

  is an optional sequence of switches as described below.

gnattest results can be found in two different places.

• automatic harness:

  This is the harness code, which is located by default in “gnattest/harness” directory created in the object directory of the main project file. All of this code is generated completely automatically and can be destroyed and regenerated at will, with the exception of the file gnattest_common.gpr, which is created if absent, but never overwritten. It is not recommended to modify other files manually, since these modifications will be lost if gnattest is re-run. The entry point in the harness code is the project file named test_driver.gpr. Tests can be compiled and run using a command such as:

  $ gprbuild -P<harness-dir>/test_driver
  

  Note that if you need to adjust any options used to compile the harness, you can do so by editing the file gnattest_common.gpr.

• actual unit test skeletons:

  A test skeleton for each visible subprogram is created in a separate file, if it doesn’t exist already. By default, those separate test files are located in a “gnattest/tests” directory that is created in the object directory of corresponding project file. For example, if a source file my_unit.ads contains a visible subprogram Proc, then the corresponding unit test will be found in file <object-dir>/gnattest/tests/my_unit-test_data-tests.adb and will be called Test_Proc_<code>. <code> is a signature encoding used to differentiate test names in case of overloading. The --tests-dir, --subdirs and --tests-root switches select other locations.

  Note that if the project already has both my_unit.ads and my_unit-test_data.ads, this will cause a name conflict with the generated test package.

2.2.2. Test Execution Mode

In this mode gnattest has a the following command-line interface:

$ gnattest test_drivers.list [ switches ]

where

• test_drivers.list

  is the name of the text file containing the list of executables to treat as test drivers. This file is automatically generated by gnattest, but can be hand-edited to add or remove tests. This switch is required.

• switches

  is an optional sequence of switches as described below.

2.3. Switches for gnattest in framework generation mode

--strict

Return a failure exit status if gnattest failed to process one of the argument sources, either because the source has diagnostics when it is analyzed, or because an error occurred while generating its test package, its stub or its instrumented version. Without this switch, gnattest reports such failures and skips the offending sources.

--strict also makes gnattest print the exception name and a symbolic traceback for each of those failures, which is useful when reporting a bug.

-q, --quiet

Quiet mode: suppresses noncritical output messages.

-v, --verbose

Verbose mode: produces additional output about the execution of the tool, starting with the tool version banner. Note that this does not stop gnattest from running; use --version to print the version and exit.

--version

Print the tool version and exit.

--help

Print a usage summary and exit.

-r, -U

Process all the sources visible from the root project, that is to say its own sources plus those of the projects it imports. This overrides the default selection based on the project’s mains; see Selecting the sources to process.

-U source_file

Process only those source files for units in the closure of the Ada source contained in source_file. Note that this option expects the source file name but not the Ada unit name as its parameter.

--no-subprojects

Process only source files from the root project.

-Xname=val

Indicates that the external variable name in the project has the value val.

-files=filename

Take as arguments the files listed in text file file. Text file file may contain empty lines that are ignored. Each nonempty line should contain the name of an existing file. Several such switches may be specified simultaneously.

--ignore=filename

Do not process the sources listed in a specified file.

--RTS=rts-name

Specifies the name of the runtime library the harness should use. For restricted profiles, gnattest takes into account the run-time limitations when generating the harness.

--additional-tests=projname

Sources described in projname are considered potential additional manual tests to be added to the test suite.

--harness-only

When this option is given, gnattest creates a harness for all sources, treating them as test packages. This option is not compatible with closure computation done by -U main.

--separate-drivers[=val]

Generates a separate test driver for each test or unit under test, rather than a single executable incorporating all tests. val can be “unit” or “test”, or may be omitted, which defaults to “unit”.

--separate-drivers=test is not supported on light runtime profiles.

--stub

Generates the testing framework that uses subsystem stubbing to isolate the code under test.

--recursive-stub

Used along –stub, indicates gnattest to generate stubs for all the packages that are withed by the stubbed units, recursively.

--harness-dir=dirname

Specifies the directory that will hold the harness packages and project file for the test driver. If the dirname is a relative path, it is considered relative to the object directory of the project file.

--tests-dir=dirname

All test packages are placed in the dirname directory. If the dirname is a relative path, it is considered relative to the object directory of the project file. When all sources from all projects are taken recursively from all projects, dirname directories are created for each project in their object directories and test packages are placed accordingly.

--subdirs=dirname

Test packages are placed in a subdirectory of the corresponding source directory, with the name dirname. Thus, each set of unit tests is located in a subdirectory of the code under test. If the sources are in separate directories, each source directory has a test subdirectory named dirname.

--tests-root=dirname

The hierarchy of source directories, if any, is recreated in the dirname directory, with test packages placed in directories corresponding to those of the sources. If the dirname is a relative path, it is considered relative to the object directory of the project file. When projects are considered recursively, directory hierarchies of tested sources are recreated for each project in their object directories and test packages are placed accordingly.

--stubs-dir=dirname

The hierarchy of directories containing stubbed units is recreated in the dirname directory, with stubs placed in directories corresponding to projects they are derived from. If the dirname is a relative path, it is considered relative to the object directory of the project file. When projects are considered recursively, directory hierarchies of stubs are recreated for each project in their object directories and test packages are placed accordingly.

--exclude-from-stubbing=filename

Disables stubbing of units listed in filename. The file should contain corresponding spec files, one per line.

--exclude-from-stubbing=spec:filename

Same as above, but corresponding units will not be stubbed only when testing unit whose specification is declared in specified spec file.

--include-for-stubbing=filename

Enables stubbing of units listed in filename. The file should contain corresponding spec files, one per line.

--include-for-stubbing=spec:filename

Same as above, but corresponding units will be stubbed only when testing unit whose specification is declared in specified spec file.

Note: in case of using both include-for-stubbing and exclude-from-stubbing, local configuration will override the global configuration, e.g. if one unit is excluded by default from stubbing, and then included for a specific unit by using the {spec}:{filename} variant, then it shall be included for this specific unit.

gnattest rejects cases of using both options with the same granularity level.

--validate-type-extensions

Enables substitution check: run all tests from all parents in order to check substitutability in accordance with the Liskov substitution principle (LSP).

--inheritance-check

Enables inheritance check: run inherited tests against descendants.

--no-inheritance-check

Disables inheritance check.

--test-case-only

Generates test skeletons only for subprograms that have at least one associated pragma or aspect Test_Case.

--skeleton-default=val

Specifies the default behavior of generated skeletons. val can be either “fail” or “pass”, “fail” being the default.

--passed-tests=val

Specifies whether or not passed tests should be shown. val can be either “show” or “hide”, “show” being the default.

--exit-status=val

Specifies whether or not generated test driver should return failure exit status if at least one test fails or crashes. val can be either “on” or “off”, “off” being the default. If --exit-status=on is used to generate the test harness, it should also be used if running the test drivers via the gnattest test_drivers.list command.

--omit-sloc

Suppresses comment line containing file name and line number of corresponding subprograms in test skeletons.

--no-command-line

Don’t add command line support to test driver. Note that regardless of this switch, gnattest will automatically refrain from adding command line support if it detects that the selected run-time doesn’t provide this capability.

--test-duration

Adds time measurements for each test in generated test driver.

--reporter=val

Use the specified reporter to output the test results. val must be one of gnattest (the default), text, xml or junit; any other value is rejected. The deprecated value xml_deprecated selects the legacy XML output and emits a warning; use xml instead.

The xml and junit reporters need the name of the tested subprogram in their output, so they imply --include-subp-name.

This switch has no effect when combined with --stub or --separate-drivers, and gnattest emits a warning saying so. Since the test execution mode only applies to the individual test drivers produced by those two switches, the test drivers it runs always use the default reporter.

--tests-root, --subdirs and --tests-dir switches are mutually exclusive.

2.4. Selecting the sources to process

The set of sources gnattest processes is determined as follows.

If one or more file names are given on the command line, or with one or more -files=filename switches, only those sources are processed.

Otherwise, the sources are taken from the project, and the selection depends on whether the root project defines mains:

• if the root project has at least one Main and all of its mains are Ada sources, only the units in the closure of those mains are processed;

• otherwise, all the sources visible from the root project, that is to say its own sources plus those of the projects it imports, are processed. Sources belonging to externally built projects are never processed.

Note in particular that, for a project that defines a Main, the default is not to process every source of the project: units that the main does not depend on are left out. Use -U to process them as well.

The following switches change this selection:

• -U processes all the sources visible from the root project, regardless of any Main the project may define.

• -U source_file processes the closure of source_file. Note that this switch expects a source file name, not an Ada unit name.

• --no-subprojects processes only the sources of the root project, leaving out those of the imported projects.

• --ignore=filename removes from the selection the sources listed in filename, whichever way the selection was made.

2.5. Switches for gnattest in test execution mode

--passed-tests=val

Specifies whether or not passed tests should be shown. val can be either “show” or “hide”, “show” being the default.

--exit-status=val

Specifies whether or not generated test driver should return failure exit status if at least one test fails or crashes. val can be either “on” or “off”, “off” being the default. The switch --exit-status=on should be used both when generating the test harness and when running the test drivers via the gnattest test_drivers.list command.

--queues=n, -jn

Runs n tests in parallel (default is 1).

--copy-environment=dir

Contents of dir directory will be copied to temporary directories created by gnattest in which individual test drivers are spawned.

--subdirs=dirname

Test driver executables from test_drivers.list are searched in dirname subdirectories of specified locations.

2.6. Project Attributes for gnattest

Most of the command-line options can also be passed to the tool by adding special attributes to the project file. Those attributes should be put in package Gnattest. Here is the list of attributes:

• Tests_Root

  is used to select the same output mode as with the --tests-root option. This attribute cannot be used together with Subdir or Tests_Dir.

• Subdir

  is used to select the same output mode as with the --subdirs option. This attribute cannot be used together with Tests_Root or Tests_Dir.

• Tests_Dir

  is used to select the same output mode as with the --tests-dir option. This attribute cannot be used together with Subdir or Tests_Root.

• Stubs_Dir

  is used to select the same output mode as with the --stubs-dir option.

• Harness_Dir

  is used to specify the directory in which to place harness packages and project file for the test driver, otherwise specified by --harness-dir.

• Additional_Tests

  is used to specify the project file, otherwise given by --additional-tests switch.

• Skeletons_Default

  is used to specify the default behaviour of test skeletons, otherwise specified by --skeleton-default option. The value of this attribute should be either pass or fail.

• Default_Stub_Exclusion_List

  is used to specify the file with list of units whose bodies should not be stubbed, otherwise specified by --exclude-from-stubbing=filename.

• Stub_Exclusion_List ("spec")

  is used to specify the file with list of units whose bodies should not be stubbed when testing “spec”, otherwise specified by --exclude-from-stubbing=spec:filename.

Each of those attributes can be overridden from the command line if needed.

Other gnattest switches can be passed via the project file using the two following attributes, both of which take a list of switches:

• Default_Switches

  switches to pass to every gnattest invocation on this project.

• Switches ("source_file")

  switches to pass to gnattest when it is invoked on source_file. This attribute is only taken into account when exactly one file name is given on the command line, and, when it applies, it replaces Default_Switches rather than adding to it.

For instance:

project My_Project is
   package Gnattest is
      for Default_Switches use ("--passed-tests=hide", "--exit-status=on");
      for Switches ("tricky_unit.ads") use ("--exit-status=on", "--omit-sloc");
   end Gnattest;
end My_Project;

2.7. Simple Example

Let’s take a very simple example using the first gnattest example located in:

<install_prefix>/share/examples/gnattest/simple

This project contains a simple package containing one subprogram. By running gnattest:

$ gnattest --harness-dir=driver -Psimple.gpr

a test driver is created in directory driver. It can be compiled and run:

$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner

One failed test with the diagnosis “test not implemented” is reported. Since no special output option was specified, the test package Simple.Tests is located in:

<install_prefix>/share/examples/gnattest/simple/obj/gnattest/tests

For each package containing visible subprograms, a child test package is generated. It contains one test routine per tested subprogram. Each declaration of a test subprogram has a comment specifying which tested subprogram it corresponds to. Bodies of test routines are placed in test package bodies and are surrounded by special comment sections. Those comment sections should not be removed or modified in order for gnattest to be able to regenerate test packages and keep already written tests in place. The test routine Test_Inc_4f8b9f located at simple-test_data-tests.adb contains a single statement: a call to procedure Assert. It has two arguments: the Boolean expression we want to check and the diagnosis message to display if the condition is false.

That is where actual testing code should be written after a proper setup. An actual check can be performed by replacing the Assert call with:

Assert (Inc (1) = 2, "wrong incrementation");

After recompiling and running the test driver, one successfully passed test is reported.

2.8. Setting Up and Tearing Down the Testing Environment

Besides test routines themselves, each test package has a parent package Test_Data that has two procedures: Set_Up and Tear_Down. This package is never overwritten by the tool. Set_Up is called before each test routine of the package, and Tear_Down is called after each test routine. Those two procedures can be used to perform necessary initialization and finalization, memory allocation, etc. Test type declared in Test_Data package is parent type for the test type of test package and can have user-defined components whose values can be set by Set_Up routine and used in test routines afterwards.

2.9. Regenerating Tests

Bodies of test routines and Test_Data packages are never overridden after they have been created once. As long as the name of the subprogram, full expanded Ada names and order of its parameters are the same, and comment sections are intact, the old test routine will fit in its place and no test skeleton will be generated for the subprogram.

This can be demonstrated with the previous example. By uncommenting declaration and body of function Dec in simple.ads and simple.adb, running gnattest on the project, and then running the test driver:

$ gnattest --harness-dir=driver -Psimple.gpr
$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner

The old test is not replaced with a stub, nor is it lost, but a new test skeleton is created for function Dec.

The only way of regenerating tests skeletons is to remove the previously created tests together with corresponding comment sections.

2.10. Default Test Behavior

The generated test driver can treat unimplemented tests in two ways: either count them all as failed (this is useful to see which tests are still left to implement) or as passed (to sort out unimplemented ones from those actually failing).

The test driver accepts a switch to specify this behavior: --skeleton-default=val, where val is either pass or fail (exactly as for gnattest).

The default behavior of the test driver is set with the same switch as passed to gnattest when generating the test driver.

Passing it to the driver generated on the first example:

$ test_runner --skeleton-default=pass

makes both tests pass, even the unimplemented one.

2.11. Selecting Tests to Run

By default the generated test driver runs every test it contains. It also accepts a --routines switch to run only the tests associated with selected subprograms, which is convenient to re-run a single test during development (this switch is, for instance, what the GNAT Studio and VS Code extensions use to run an individual test).

A subprogram is designated by the source location of its declaration, given as the simple name of its spec file and the line number, separated by a colon:

$ test_runner --routines=my_unit.ads:7

This runs all the tests associated with the subprogram declared at line 7 of my_unit.ads. The selection follows the test type hierarchy, so any inherited, overridden, or per-instance tests derived from that subprogram are run as well.

The switch may be repeated to select several subprograms:

$ test_runner --routines=my_unit.ads:7 --routines=other.ads:10

The list of source locations may also be read from a file, one file:line entry per line, by prefixing its name with @:

$ test_runner --routines=@selected.txt

If no subprogram corresponds to a given source location, the driver reports an error such as no subprogram corresponds to sloc my_unit.ads:7 and aborts.

2.12. Testing Primitive Operations of Tagged Types

Creation of test skeletons for primitive operations of tagged types entails a number of features. Test routines for all primitives of a given tagged type are placed in a separate child package named according to the tagged type. For example, if you have tagged type T in package P, all tests for primitives of T will be in P.T_Test_Data.T_Tests.

Consider running gnattest on the second example (note: actual tests for this example already exist, so there’s no need to worry if the tool reports that no new stubs were generated):

$ cd <install_prefix>/share/examples/gnattest/tagged_rec
$ gnattest --harness-dir=driver -Ptagged_rec.gpr

Taking a closer look at the test type declared in the test package Speed1.Controller_Test_Data is necessary. It is declared in:

<install_prefix>/share/examples/gnattest/tagged_rec/obj/gnattest/tests

Test types are direct or indirect descendants of AUnit.Test_Fixtures.Test_Fixture type. In the case of non-primitive tested subprograms, the user doesn’t need to be concerned with them. However, when generating test packages for primitive operations, there are some things the user needs to know.

Type Test_Controller has components that allow assignment of various derivations of type Controller. And if you look at the specification of package Speed2.Auto_Controller, you will see that Test_Auto_Controller actually derives from Test_Controller rather than AUnit type Test_Fixture. Thus, test types mirror the hierarchy of tested types.

The Set_Up procedure of Test_Data package corresponding to a test package of primitive operations of type T assigns to Fixture a reference to an object of that exact type T. Note, however, that if the tagged type has discriminants, the Set_Up only has a commented template for setting up the fixture, since filling the discriminant with actual value is up to the user.

The knowledge of the structure of test types allows additional testing without additional effort. Those possibilities are described below.

2.13. Testing Inheritance

Since the test type hierarchy mimics the hierarchy of tested types, the inheritance of tests takes place. An example of such inheritance can be seen by running the test driver generated for the second example. As previously mentioned, actual tests are already written for this example.

$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner

There are 6 passed tests while there are only 5 testable subprograms. The test routine for function Speed has been inherited and run against objects of the derived type.

2.14. Tagged Type Substitutability Testing

Tagged Type Substitutability Testing is a way of verifying the global type consistency by testing. Global type consistency is a principle stating that if S is a subtype of T (in Ada, S is a derived type of tagged type T), then objects of type T may be replaced with objects of type S (that is, objects of type S may be substituted for objects of type T), without altering any of the desirable properties of the program. When the properties of the program are expressed in the form of subprogram preconditions and postconditions (let’s call them pre and post), the principle is formulated as relations between the pre and post of primitive operations and the pre and post of their derived operations. The pre of a derived operation should not be stronger than the original pre, and the post of the derived operation should not be weaker than the original post. Those relations ensure that verifying if a dispatching call is safe can be done just by using the pre and post of the root operation.

Verifying global type consistency by testing consists of running all the unit tests associated with the primitives of a given tagged type with objects of its derived types.

In the example used in the previous section, there was clearly a violation of type consistency. The overriding primitive Adjust_Speed in package Speed2 removes the functionality of the overridden primitive and thus doesn’t respect the consistency principle. gnattest has a special option to run overridden parent tests against objects of the type which have overriding primitives:

$ gnattest --harness-dir=driver --validate-type-extensions -Ptagged_rec.gpr
$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner

While all the tests pass by themselves, the parent test for Adjust_Speed fails against objects of the derived type.

Non-overridden tests are already inherited for derived test types, so the --validate-type-extensions enables the application of overridden tests to objects of derived types.

2.15. Testing with Contracts

gnattest supports pragmas Pre, Post, and Test_Case, as well as the corresponding Ada 2012 aspects. Test routines are generated, one per each Test_Case associated with a tested subprogram. Those test routines have special wrappers for tested functions that have composition of pre- and postcondition of the subprogram with “requires” and “ensures” of the Test_Case (depending on the mode, pre and post either count for Nominal mode or do not count for Robustness mode).

The third example demonstrates how this works:

$ cd <install_prefix>/share/examples/gnattest/contracts
$ gnattest --harness-dir=driver -Pcontracts.gpr

Putting actual checks within the range of the contract does not cause any error reports. For example, for the test routine which corresponds to test case 1:

Assert (Sqrt (9.0) = 3.0, "wrong sqrt");

and for the test routine corresponding to test case 2:

Assert (Sqrt (-5.0) = -1.0, "wrong error indication");

are acceptable:

$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner

However, by changing 9.0 to 25.0 and 3.0 to 5.0, for example, you can get a precondition violation for test case one. Also, by using any otherwise correct but positive pair of numbers in the second test routine, you can also get a precondition violation. Postconditions are checked and reported the same way.

2.16. Additional Tests

gnattest can add user-written tests to the main suite of the test driver. gnattest traverses the given packages and searches for test routines. All procedures with a single in out parameter of a type which is derived from AUnit.Test_Fixtures.Test_Fixture and that are declared in package specifications are added to the suites and are then executed by the test driver. (Set_Up and Tear_Down are filtered out.)

An example illustrates two ways of creating test harnesses for user-written tests. Directory additional_tests contains an AUnit-based test driver written by hand.

<install_prefix>/share/examples/gnattest/additional_tests/

To create a test driver for already-written tests, use the --harness-only option:

gnattest -Padditional/harness/harness.gpr --harness-dir=harness_only \\
  --harness-only
gprbuild -Pharness_only/test_driver.gpr
harness_only/test_runner

Additional tests can also be executed together with generated tests:

gnattest -Psimple.gpr --additional-tests=additional/harness/harness.gpr \\
  --harness-dir=mixing
gprbuild -Pmixing/test_driver.gpr
mixing/test_runner

2.17. Individual Test Drivers

By default, gnattest generates a monolithic test driver that aggregates the individual tests into a single executable. It is also possible to generate separate executables for each test or each unit under test, by passing the switch --separate-drivers[={val}]. The parameter val selects the granularity: unit (the default, also used when val is omitted) generates one test driver per unit under test, while test generates one test driver per individual test. This approach scales better for large testing campaigns, especially involving target architectures with limited resources typical for embedded development. It can also provide a major performance benefit on multi-core systems by allowing simultaneous execution of multiple tests.

gnattest can take charge of executing the individual tests; for this, instead of passing a project file, a text file containing the list of executables can be passed. Such a file is automatically generated by gnattest under the name test_drivers.list, but it can be hand-edited to add or remove tests, or replaced. The individual tests can also be executed standalone, or from any user-defined scripted framework.

2.18. Stubbing

Depending on the testing campaign, it is sometimes necessary to isolate the part of the algorithm under test from its dependencies. This is accomplished via stubbing, i.e. replacing the subprograms that are called from the subprogram under test by stand-in subprograms that match the profiles of the original ones, but simply return predetermined values required by the test scenario.

This mode of test harness generation is activated by the switch --stub.

The implementation approach chosen by gnattest is as follows. For each package under consideration all the packages it is directly depending on are stubbed, excluding the generic packages and package instantiations. The stubs are shared for each package under test. The specs of packages to stub remain intact, while their bodies are replaced, and hide the original bodies by means of extending projects. Also, for each stubbed package, a child package with setter routines for each subprogram declaration is created. These setters are meant to be used to set the behavior of stubbed subprograms from within test cases.

Note that subprograms belonging to the same package as the subprogram under test are not stubbed. This guarantees that the sources being tested are exactly the sources used for production, which is an important property for establishing the traceability between the testing campaign and production code.

Due to the nature of stubbing process, this mode implies the switch --separate-drivers, i.e. an individual test driver (with the corresponding hierarchy of extending projects) is generated for each unit under test.

Note

Developing a stubs-based testing campaign requires good understanding of the infrastructure created by gnattest for this purpose. We recommend following the two stubbing tutorials simple_stubbing and advanced_stubbing provided under <install_prefix>/share/examples/gnattest before attempting to use this powerful feature.

2.19. Integration with GNATcoverage

In addition to the harness, gnattest generates a Makefile. This Makefile provides targets for building the test drivers and also the targets for computing the coverage information using GNATcoverage framework when this coverage analysis tool is available.

The target coverage fully automates the process using source traces: it instruments the test driver projects with gnatcov instrument, builds them against the instrumented sources, runs them, turns each resulting source trace into a checkpoint and finally consolidates the checkpoints into a report:

make coverage

The Makefile also provides the following targets:

• all

  builds all the test drivers, without any coverage instrumentation.

• instrument-all

  runs gnatcov instrument on all the test driver projects, without building them.

• instr-build-all

  same, and builds the instrumented drivers.

• clean

  runs gprclean on every test driver project and removes the trace files.

The switches passed to the various gnatcov commands are held in coverage_settings.mk, which is generated once and then owned by the user; see Structure of the Generated Harness.

For more details about using GNATtest with GNATcoverage see Using GNATtest with GNATcoverage.

2.20. Structure of the Generated Harness

gnattest generates a number of artifacts. Understanding what each one is for, and in particular which ones are regenerated on every run versus created once and then owned by the user, is essential both for customizing the harness and for deciding what to put under version control.

The artifacts fall into two broad families: the harness (the test driver infrastructure) and the test code (the skeletons you fill in). With the default object-directory locations and a single test driver, the layout looks like this, where <u> stands for the name of a unit under test, <U> for its Ada unit name and <prj> for the name of the project:

<object-dir>/gnattest/
  harness/                             <- test driver infrastructure
    test_driver.gpr                    regenerated   project to build/run the driver
    test_<prj>.gpr                     regenerated   project compiling the test skeletons
    gnattest_common.gpr                created once  shared build options (user-owned)
    test_runner.adb                    regenerated   driver main
    gnattest_main_suite.ad[bs]         regenerated   top-level AUnit suite
    <u>-test_data-tests-suite.ad[bs]   regenerated   per-unit AUnit suite

    common/
      gnattest_generated.ads           regenerated   support unit shared by the drivers
      gnattest_generated-persistent.ad[bs]
                                       created once  setup/teardown shared by the drivers (user owned)
    gnattest.xml                       regenerated   test/source mapping file
    suppress.adc,                      created once  global configuration pragmas (user owned)
      suppress_no_ghost.adc
    preprocessor.def                   regenerated   preprocessor symbol definitions
    .gnattest-config.json              regenerated   internal harness configuration
    Makefile                           regenerated   GNATcoverage integration driver
    coverage_settings.mk               created once  gnatcov switches (user-owned)
  tests/                               <- test code
    <u>-test_data.ad[bs]               owned         Set_Up / Tear_Down, fixture type
    <u>-test_data-tests.ads            regenerated   test package spec
    <u>-test_data-tests.adb            owned         your test routine bodies

With --separate-drivers (and hence with --stub), the per-driver files move into one subdirectory per unit, or per test, under test, and two more files appear at the top of the harness directory:

<object-dir>/gnattest/
  harness/
    <U>.Test_Data.Tests/             <- one such directory per driver
      test_driver.gpr                regenerated   project to build/run this driver
      <u>-test_data-tests-suite-test_runner.adb
                                     regenerated   this driver's main
      <u>-test_data-tests-suite.ad[bs]
                                     regenerated   this driver's AUnit suite
      units.list                     regenerated   unit under test, for gnatcov
    test_drivers.gpr                 regenerated   aggregate project building them all
    test_drivers.list                regenerated   list of driver executables
    ...                              the files listed above, except test_driver.gpr,
                                     test_runner.adb and the suite units

Only the files described in the next section have a stable interest for the user; the others are internal to the harness, regenerated on every run, and should not be edited.

2.20.1. Responsibilities of the harness components

• test_driver.gpr

  The entry point of the harness: the project file used to build and run the test driver (gprbuild -P<harness-dir>/test_driver). It is regenerated on every run. With --separate-drivers, one such project is generated per unit (or per test) under test, each in its own subdirectory.

• gnattest_common.gpr

  A project file, imported by every generated test driver project, that holds build options shared across the harness (compiler and linker switches, extra source directories, global configuration pragmas, etc.). It is created once if absent and never overwritten, so it is the intended place for the user to customize how the harness is compiled. See for instance Instrumenting test harnesses for a SPARK project for using it to pass a configuration pragma file for SPARK code.

• test_runner and the suite units

  The generated main (test_runner.adb, or <u>-test_data-tests-suite-test_runner.adb with separate drivers) and the AUnit suite-aggregation packages (gnattest_main_suite.ad[bs] and <u>-test_data-tests-suite.ad[bs]). They are fully automatic, regenerated on every run, and should not be edited.

• test_drivers.gpr

  An aggregate project, generated with --separate-drivers, that builds all the individual test driver projects at once. Regenerated on every run.

• gnattest.xml

  The mapping between the sources under test and the generated test artifacts, used by IDEs to navigate between a subprogram and its tests. Regenerated on every run.

• Makefile

  Automates the production of a coverage report with GNATcoverage (see Integration with GNATcoverage). It is regenerated on every invocation of gnattest and should not be edited.

• coverage_settings.mk

  A secondary makefile, included by Makefile, holding the switches passed to the various gnatcov commands. Its values are extracted from the root project when first generated, but unlike Makefile it is created once and not regenerated, so it is the intended place to adjust GNATcoverage settings that are not expressed in the project file. See Using the Makefile generated by GNATtest for details.

• units.list

  Generated alongside each test_driver.gpr, and therefore only with separate drivers; lists the unit under test so that gnatcov can be told which unit is of interest and avoid incidental coverage. Regenerated on every run.

• test_drivers.list

  The list of test driver executables consumed by the test execution mode. Like units.list it is only generated with separate drivers, since that mode is what the test execution mode runs. It is generated automatically but may be hand-edited to add or remove tests; it is also safe to let gnattest regenerate it.

2.20.2. Files the user is expected to modify

Most of the harness is disposable, but two harness files are deliberately created once and never regenerated so that the user can adapt the harness to the project’s needs:

• gnattest_common.gpr — to customize how the harness is built.

• coverage_settings.mk — to customize the gnatcov switches used by the generated Makefile (only relevant when using the GNATcoverage integration).

Because they are preserved across regenerations and hold project-specific settings, both of these files are good candidates for version control (see Putting Tests under Version Control).

The following table summarizes, for each generated artifact, whether it is regenerated, whether it is meant to be edited, and whether it should be put under version control:

File	Regenerated?	User-editable?	Version control?
test_driver.gpr	Yes	No	No
gnattest_common.gpr	No (created once)	Yes	Yes
gnattest_generated-persistent.ad[bs]	No (created once)	Yes	Yes
test_runner, suite units	Yes	No	No
test_drivers.gpr (separate drivers)	Yes	No	No
gnattest.xml	Yes	No	No
Makefile	Yes	No	No
coverage_settings.mk	No (created once)	Yes	Yes (with GNATcov integration)
units.list (separate drivers)	Yes	No	No
test_drivers.list (separate drivers)	Yes	Yes (optional)	Optional
*-test_data.adb	No (preserved)	Yes	Yes
*-test_data.ads	No (preserved)	Yes	Yes
*-test_data-tests.adb	No (preserved)	Yes	Yes
*-test_data-tests.ads	Yes	No	No
stub / stub-data bodies (--stub)	No (preserved)	Yes	Yes

2.21. Putting Tests under Version Control

As has been stated earlier, gnattest generates two different types of code, test skeletons and harness. With the exception of the two user-owned files noted below, the harness is generated completely automatically each time, does not require manual changes and therefore should not be put under version control. It makes sense to put under version control the test data packages, both their specs and their bodies, and the bodies of the test packages. Note that the test package specs (*-test_data-tests.ads) are the only part of the test code that is regenerated on every run, and should not be put under version control.

Test data package specs (*-test_data.ads) are created once and never overwritten afterwards, except for the sections surrounded by read only markers. They are meant to be edited, since this is where the components of the test fixture type are declared (see Setting Up and Tearing Down the Testing Environment), so they belong under version control together with their bodies.

Additionally, if stubbing is enabled with --stub, it also makes sense to put the stubbed bodies, as well as the stub-data bodies under source control, as gnattest will preserve modifications made to these files.

Two files located in the harness directory are an exception to the rule above: they are created once and never overwritten, are meant to be adapted by the user, and should therefore be put under version control:

• gnattest_common.gpr, which holds the build options shared by the harness projects;

• coverage_settings.mk, which holds the gnatcov switches used by the generated Makefile, when the GNATcoverage integration is used.

See Structure of the Generated Harness for a description of all the generated artifacts and a summary of which ones should be put under version control.

Option --omit-sloc may be useful when putting test packages under version control.

2.22. Aggregate project handling

If the project passed to gnattest with the -P switch is an aggregate project, the aggregated projects will be processed sequentially and independently. This will result in one harness directory being generated by default, in the object directories of each of the aggregated projects.

gnattest will not generate any project file or makefile to automate the build of the harnesses of each of the aggregated project.

2.22.1. Artifact directories and aggregate projects

By default, all artifacts generated by gnattest are located in subdirectories of the object directory of each of the aggregated projects. This in particular means that tests or stubs for a common dependency of two aggregated projects will be duplicated. In order to avoid this, options such as --stubs-dir., --tests-dir or --subdirs can be used, with relative paths so that the artifacts for the common dependencies are generated in the same location, and re-used across each test harness.

On the contrary, using --harness-dir or --tests-dir with an absolute path will result in the harness and/or files of a first aggregated project being overwritten by the generation of the test harness for subsequent aggregated projects, and should thus be avoided.

2.23. Current Limitations

The tool currently has the following limitations:

• generic tests for nested generic packages and their instantiations are not supported;

• tests for protected subprograms and entries are not supported;

• pragma No_Run_Time is not supported;

• pragma No_Secondary_Stack is not supported;

• if pragmas for interfacing with foreign languages are used, manual adjustments might be necessary to make the test harness compilable;

• use of some constructs, such as elaboration-control pragmas, Type_Invariant aspects, and complex variable initializations that use Subprogram’Access, may result in elaboration circularities in the generated harness;

• heavy usage of preprocessor that affects constructs like subprogram profiles or tagged type hierarchies may result in improper test driver generation.

2.24. Automatically generating test cases (experimental)

Please note that all the features described bellow are experimental, and the interface is subject to change.

GNATtest has the capability to generate test inputs for subprograms under test. This test generation feature is also usable in conjunction with GNATfuzz, in order to use GNATtest harnesses (generated or manually written) as a starting corpus for a fuzzing session, and to integrate inputs of interest found by GNATfuzz back into the test harness. For more details, see section Using GNATtest with GNATfuzz (experimental).

Test input generation and execution is supported on native platforms, as well as cross bareboard targets provided that the Ada runtime is of an embedded profile. Test input generation for target with only light or light tasking profiles is not supported.

2.24.1. Setting up the test generation runtime

Generation of values for Ada cannot be fully done statically, as the bounds of some types may only be defined at runtime. As such, the test generation feature requires the compilation and installation of a runtime project. This can be done using the gnattest setup command, see Setting up the runtimes required by gnattest for more information.

2.24.2. Generating test inputs

Note

Test generation is only supported for projects built as static libraries or as standalone executables. It is not supported when the project under test is a shared library, i.e. when its Library_Kind attribute is set to relocatable. The test generation runtime and the support library generated for the project are only ever provided as static libraries, so a relocatable library cannot be linked against them, and building the harness will fail at link time. To use test generation on such a project, build it as a static library (Library_Kind set to static, or static-pic if position-independent code is required).

gnattest provides a --gen-test-vectors switch that can be used to automatically generate test cases for all of the supported subprogram profiles. The number of generated test cases can be configured through the --gen-test-num switch.

As mentioned in section Setting up the test generation runtime, test input generation requires executing code to determine some of the characteristics of the types at hand. This means that for both native and cross targets, a GNAT Pro toolchain for the corresponding target must be available in the environment.

For cross targets, the test input generation harness will be executed through GNATemulator. GNATtest may thus compile it against a different runtime than the one specified through the --RTS switch, to use one corresponding to a board supported by GNATemulator for the given target. This should have no impact on the definition of the various types, as only the target CPU helps define the size of predefined types. The mapping between targets and the runtime used for test input generation is defined under <gnatdas_install_dir>/share/tgen/tgen_target_runtimes.json, users may copy and modify it, and feed the modified mapping to GNATtest using the --tgen-target-config=FILE command line option.

2.24.2.1. Supported types

gnattest can natively generate test cases for a subprogram when it can generate a value for every one of its parameters, and for the global inputs listed in its Global aspect, if it has one. Note that this applies to parameters of every mode, out parameters included: gnattest needs a representation of the type in order to marshall the value back, so an unsupported type disables generation for the subprogram whichever mode it appears in.

A type is not natively supported when it is, or has a subcomponent that is, one of the following:

• an access type, including a subprogram access type;

• a class-wide type, or a type derived from an abstract type;

• an anonymous array or anonymous access type;

• a private type declared in a nested package;

• a type derived from a private type that is unconstrained;

• a generic formal type;

• a type declared in a generic package instantiation that is a library item;

• a task type or a protected type;

• System.Address;

• an array type whose number of elements exceeds the configured array size limit, which defaults to 1000 and can be changed with the TGEN_ARRAY_LIMIT environment variable.

Note that concrete tagged types and limited types are supported.

gnattest reports the reason why it cannot generate values for a given parameter, so the diagnostics it prints are the authoritative answer for a given subprogram. For instance:

pkg.print_acc.A: pkg.int_acc is not supported (Access types are not supported)

For types not respecting the above restrictions, it is possible to generate test inputs using a proxy generation function.

2.24.2.2. Test input generation strategies

Input value generation currently follows a simple strategy for each input parameter of the subprogram under test. Parameters of scalar types, and scalar components of composite types have their values uniformly generated. For unconstrained array types, a total number of elements is randomly chosen between 0 and 10 for each dimension, with thus an upper limit of 10 ** n elements, where n is the number of dimensions of the array (so between 0 and 10 elements for the common case of a one-dimensional array), then, for each dimension, the low bound is randomly chosen and the high bound computed accordingly to those two first points.

Independently of this, the marshallers refuse to read back an array with more than 1000 elements per dimension, to avoid allocating overly large arrays on the stack. This limit can be changed through the TGEN_ARRAY_LIMIT environment variable; array types whose number of elements is statically known to exceed this value (all dimensions combined) are reported as unsupported.

For record discriminants, different strategies are chosen depending on the use of the discriminant within the record: If the discriminant constraints a array component, then the array strategy described above is used. If the discriminant is used in a variant part, generation will be biased in order to generated all possible shapes of the record (i.e. explore all variants). Otherwise, these are generated as any other scalar component.

The generated test cases are then stored in a ad-hoc (and yet to be specified) JSON format, in files under the <obj_dir>/gnattest/tests/JSON_Tests directory. The generated JSON files are preserved through a gnattest rerun. The user is thus free to modify them, to e.g. fill in expected return values, though backward compatibility of the format is not guaranteed at this stage.

gnattest also generates Ada files to actually execute the test cases. Each test vector has its own AUnit test case, and all test cases for a specific subprogram are all stored in a dedicated file, namely <unit_name>-test_data-test_<subp_name>_<subp_hash>.ad[bs], where <unit_name> is the name of the unit in which the subprogram is declared, <subp_name> is the name of the subprogram, and <subp_hash> is a hash based on the profile of the subprogram, in order to differentiate overloads.

The content of these files are re-generated each time gnattest is invoked, independently of the presence of the --gen-test-vectors switch on the command line. It is thus not necessary to re-invoke gnattest with that switch more than once, unless the goal is to generate additional test inputs.

2.24.3. Test input generation through a proxy subprogram

It is possible to assign to each type, even those not natively supported, a proxy function, which GNATtest will use every time it need to generate a value for the return type of the proxy function. For a given type T, a subprogram is eligible to be the proxy of T if it meets the following conditions:

• The proxy must be a function whose return type is T;

• The proxy must have no out parameters;

• The proxy must have at least one in or in out parameters, and all the input parameters should either be natively supported or have a proxy subprogram themselves;

• The proxy subprogram must be visible from the package in which T is declared.

A proxy subprogram can be designated explicitly, by using the TGen_Proxy => <Proxy Name> aspect on the type definition, or gnattest can automatically identify one for types that are not natively supported.

The --detect-tgen-proxies={policy} switch controls how far GNATtest looks when automatically identifying a proxy for a type that is not natively supported. policy must be one of:

• none

  no automatic search is performed at all. Only the proxies designated explicitly with the TGen_Proxy aspect are used.

• unit (the default)

  only the unit in which the target type is declared is searched.

• all_refs

  the unit in which the target type is declared is searched first; if that yields nothing, GNATtest then inspects the references to the target type across the units of the project to find a suitable proxy function.

Note that this switch has no effect on proxies designated explicitly with the TGen_Proxy aspect, which are always honored.

A proxy cannot be used for every unsupported type. Types for which no type-compatible helper subprogram can be declared, namely anonymous array and access types, generic formal types, private types declared in a nested package, and types declared in a library-level generic package instantiation, are rejected regardless of any proxy.

Once a proxy is defined for a given type, GNATtest will generate test inputs for the proxy. These test inputs are written to the serialized test files. When executing the tests or loading the test inputs in GNATfuzz, the input values for the proxy are de-serialized, then the proxy subprogram called to obtain a value for its return type. In particular, this means that it is not possible to serialize Ada values for a type with a proxy, and thus test input dumping from pre-existing tests is not supported for proxy annotated types.

For example, given the following Ada package:

package Pkg is

   type Int_Acc is access all Integer with
     TGen_Proxy => Make_Access;

   function Make_Access (Val : Integer) return Int_Acc;

   procedure Print_Int_Acc (Acc : Int_Acc);

end Pkg;

When generating test inputs for Print_Int_Acc, GNATtest will first generate integer values, and invoke Make_Access with those values as parameters when executing the test.