6.3. Preprocessing

6.3.1. Libadalang.Preprocessing

This package implements a preprocessor for Ada sources that is compatible with GNATprep. It also provides a file reader implementing such a preprocessor, to be used in an analysis context. Please refer to GNATprep’s documentation for a description of the main concepts, of preprocessor data files and of preprocessing features. The API provided here closely follows these concepts.

6.3.1.1. Preprocessor data basics

The action of preprocessing an Ada source file is done according to parameters (definition of preprocessor symbols, how to format directives and disabled lines in the output, …). The File_Config type is used to represent such parameters, and the Preprocess procedure taking a File_Config argument can be used to preprocess a given source buffer.

--  Create a file configuration using symbol definitions from GNATprep's
--  "foo.txt" definition file, replacing directives and disabled lines
--  with blank lines.

Cfg : constant File_Config :=
  (Enabled => True,
   Definitions => Parse_Definition_File ("foo.txt"),
   Line_Mode   => Blank_Lines,
   others      => <>);

 Input_Buffer  : String := "...";
 Output_Buffer : Preprocessed_Source;
 Diagnostics   : Langkit_Support.Diagnostics.Diagnostics_Vectors.Vector;

--  Preprocess the "Input_Buffer" source, writing the result to
--  ``Output_Buffer``.

Preprocess (Cfg, Input_Buffer, Output_Buffer, Diagnostics);

if not Diagnostics.Is_Empty then
   --  Raise some error

else
   declare
      Buffer : String renames
        Output_Buffer.Buffer (1 .. Output_Buffer.Last)
   begin
      --  Use the preprocessed source in "Buffer"
   end;
end if;

Preprocessing for a whole Ada project is determined by a set of file configurations: optionally several File_Config values for sources with specific file names (see the File_Config_Maps.Map type), plus an additional File_Config value to use for files not described in this map (the “default” file config).

To create these data structures, it is possible to manipulate the Ada types directly and call the Create_Preprocessor_Data function to get a Preprocessor_Data value. It is also possible to parse GNATprep’s “preprocessor data file” (i.e. the argument for the -gnatep compiler switch) with the Parse_Preprocessor_Data_File function.

Path : constant Any_Path :=
  Create_Path_From_Environ ("ADA_INCLUDE_PATH");
--  This path allows to find the preprocessor data file and the
--  definition files it references in the current directory or in any of
--  the directories pointed by the "ADA_INCLUDE_PATH" environment
--  variable.

Prep : constant Preprocessor_Data :=
  Parse_Preprocessor_Data_File ("prep-data.txt", Path);
--  Parse the "prep-data.txt" preprocessor data file and create a full
--  preprocessor configuration from it.

6.3.1.2. Preprocessor data from GPR files

When the preprocessor for GNAT is set up with compiler switches (-gnatep and -gnateD) registered in project files, it is convenient to use the Extract_Preprocessor_Data_From_Project functions to extract the preprocessor configuration from these compiler switches.

The following GPR attributes are searched for compiler switches:

Builder.Global_Compilation_Switches
Builder.Default_Switches
Builder.Switches
Compiler.Default_Switches
Compiler.Switches

Note that these functions produce an approximation, assuming that all compiler switches affect all files. This approximation should be a good fit for most projects, since when GNAT processes a compilation unit, it uses the same preprocessor configuration for all the files it analyzes (the source that is being compiled as well as all its closure). This is because it is not possible in the general case to produce a single preprocessor data that will match exactly what happens when compiling all the units in a project.

For instance, imagine the following sources:

-- foo.ads
package Foo is
end Foo;

-- bar.ads
with Foo;
package Bar is
end Bar;

-- no matter what there is in foo.adb and bar.adb

And the GPR files:

for Switches ("foo.adb") use ("-gnateDX=1");
for Switches ("bar.adb") use ("-gnateDX=2");

Then the preprocessor configuration used when analyzing foo.ads will depend on whether it’s done during the compilation of foo.adb or of bar.adb, which cannot be represented with the preprocessor data model coming from GNATprep and GNAT itself.

For this simple example above, it is not a real problem because foo.ads does not actually use the preprocessor, but this illustrates how untractable it is to recover the preprocessor data from compiler switches exactly as the compiler would see it.

6.3.1.3. Running the preprocessor

Once the preprocessor configuration is ready, it is possible to call the Preprocess procedure, which takes a Preprocessor_Data argument, plus the file name for the source file to preprocess (used to look up the corresponding file configuration).

--  Preprocess the "Input_Buffer" source as being the content of a
--  "foo.adb" Ada source file, writing the result to "Output_Buffer".

Preprocess (Prep, "foo.adb", Input_Buffer, Output_Buffer, Diagnostics);

if not Diagnostics.Is_Empty then
   --  Raise some error

else
   declare
      Buffer : String renames
        Output_Buffer.Buffer (1 .. Output_Buffer.Last)
   begin
      --  Use the preprocessed source in "Buffer"
   end;
end if;

Finally, in order to instruct a Libadalang analysis context to automatically preprocess source files when loading files through the Get_From_File function, one needs to use the file reader mechanism (see Langkit_Support.File_Readers): first create a File_Reader_Reference value that implements preprocessing (see the Create_Preprocessor and Create_Preprocessor_From_File functions defined in this package) and then pass it to the Create_Context context constructor.

FR  : constant File_Reader_Reference :=
  Create_Preprocessor_From_File ("prep-data.txt", Path);
Ctx : constant Analysis_Context := Create_Context (File_Reader => FR);

--  Analyze the "foo.adb" source file after preprocessing it according
--  to configuration for "foo.adb" files in "prep-data.txt". The
--  analysis of any other source file that this implies will also
--  trigger preprocessing for these files.

U : constant Analysis_Unit := Ctx.Get_From_File ("foo.adb");

6.3.1.4. Package contents

type Value_Kind

type Value_Type

Discriminants:

Kind (Value_Kind) –

Components:

String_Value (Ada.Strings.Unbounded.Unbounded_String) –
Symbol_Value (Ada.Strings.Unbounded.Unbounded_String) –

function As_String (Value : Value_Type) → Ada.Strings.Unbounded.Unbounded_String: Return the string that Value must substitute to in the preprocessed sources.

procedure Parse_Definition_Option (Option : Standard.String; Name : Ada.Strings.Unbounded.Unbounded_String; Value : Value_Type; Empty_Valid : Standard.Boolean)

If Option matches <name>=<value>, where <name> is a valid preprocessor symbol name, set Name and Value to the corresponding values.

If it matches <name> or <name>= and Empty_Valid is true, set Name to it and Value to the empty value.

Otherwise, raise a Syntax_Error exception.

package Definition_Maps is new Ada.Containers.Hashed_Maps

package Definition_Maps is new Ada.Containers.Hashed_Maps
     (Key_Type        => US.Unbounded_String,
      Element_Type    => Value_Type,
      Hash            => US.Hash,
      Equivalent_Keys => US."=");

Instantiated generic package:: Definition_Maps

function Parse_Definition_File (Filename : Standard.String) → Libadalang.Preprocessing.Definition_Maps.Map

Parse the symbol file at Filename and return the corresponding definitions.

See GNATprep’s documentation for a description of this file.

type Any_Line_Mode

Determine how the preprocessor treats directives and disabled lines in the output.

Delete_Lines

Just delete these lines: this breaks line number correspondance between the original source and the preprocessed one. This corresponds to GNATprep’s default mode.

Blank_Lines

Replace these lines with empty lines. This corresponds to GNATprep’s -b option.

Comment_Lines

Preserve these lines and emit a --! comment marker in front of them. This corresponds to GNATprep’s -c option.

Empty_Comment_Lines

These lines are replaced with exactly --!. This corresponds to GNATprep’s -e option.

function Create_Preprocessor_From_File (Filename : Standard.String; Path : GNATCOLL.File_Paths.Any_Path; Line_Mode : Any_Line_Mode) → File_Reader_Reference 

Convenience Create_Preprocessor_From_File overload, to force a given line mode for all source files on which the preprocessor is enabled.

Forcing the line mode is often needed as the default is to remove lines that contain preprocessor directives and disabled code, which breaks the line number correspondance between original source code and preprocessed one. Forcing to Blank_Lines or Comment_Lines preserves this correspondance.

function Create_Preprocessor_From_File (Filename : Standard.String; Path : GNATCOLL.File_Paths.Any_Path; Line_Mode : Any_Line_Mode) → File_Refiner_Reference 

type File_Config

Discriminants:

Enabled (Standard.Boolean) –

Components:

Definitions (Libadalang.Preprocessing.Definition_Maps.Map) – Symbol/value associations for this file. Note that, in order for the preprocessing to work correctly, symbols must be lower case.
Line_Mode (Any_Line_Mode) – Determine how the preprocessor treats directives and disabled lines in the output.
Print_Symbols (Standard.Boolean) – Whether to print a sorted list of symbol and values on the standard output. Actually unused in this module.
Undefined_Is_False (Standard.Boolean) – Whether to treat undefined symbols as False in the context of a preprocessor test (see GNATprep’s -u option).

Whether to treat undefined symbols as False in the context of a preprocessor test (see GNATprep’s -u option).

Disabled_File_Config : constant File_Config

Object type:: File_Config
Default value:: (Enabled => False)

By default, the preprocessor is disabled on all Ada sources

Base_Enabled_File_Config : constant File_Config

Object type:: File_Config
Default value:: (Enabled => True, Line_Mode => Blank_Lines, others => <>)

Default file configuration when enabling preprocessing for a source file for the preprocessor integrated into GNAT.

procedure Move (Target, Source : File_Config): Move data from Source to Target. When this procedure returns Source is Disabled_File_Config.

procedure Parse_Preprocessor_Data_File (Filename : Standard.String; Path : GNATCOLL.File_Paths.Any_Path; Default_Config : File_Config; File_Configs : Libadalang.Preprocessing.File_Config_Maps.Map)

Like the Parse_Preprocessor_Data_File function, but instead fill out the Default_Config and File_Configs arguments. This procedure is useful in order to modify the parsed configuration before creating the Preprocessor_Data object.

See GNATprep’s documentation for a description of the preprocessor data file format.

procedure Extract_Preprocessor_Data_From_Project (Tree : GNATCOLL.Projects.Project_Tree'Class; Project : GNATCOLL.Projects.Project_Type; Default_Config : File_Config; File_Configs : Libadalang.Preprocessing.File_Config_Maps.Map): Like the Extract_Preprocessor_Data_From_Project function, but instead fill out the Default_Config and File_Configs arguments. This procedure is useful in order to modify the parsed configuration before creating the Preprocessor_Data object.

procedure Extract_Preprocessor_Data_From_Project (Tree : GPR2.Project.Tree.Object; Project : GPR2.Project.View.Object; Default_Config : File_Config; File_Configs : Libadalang.Preprocessing.File_Config_Maps.Map): Likewise, but with GPR2 projects

function Create_Preprocessor_Data (Default_Config : File_Config; File_Configs : Libadalang.Preprocessing.File_Config_Maps.Map) → Preprocessor_Data 

Create preprocessor data using the given file-specific configurations and the given default configuration (for other files).

Note that this “consumes” both arguments, which are left respectively to File_Config_Maps.Empty and Disabled_File_Config upon return.

procedure Iterate (Default_Config : File_Config; File_Configs : Libadalang.Preprocessing.File_Config_Maps.Map; Process : access procedure (Config : in out File_Config)): Call Process on all the file configurations passed as arguments. This procedure helps forcing some configuration, for instance, forcing the line mode for all configurations.

procedure Preprocess (Config : File_Config; Input : Standard.String; Contents : Preprocessed_Source; Diagnostics : Langkit_Support.Diagnostics.Diagnostics_Vectors.Vector)

Preprocess the Input source buffer according to the given Config preprocessor file configuration.

On success, leave Diagnostics empty and return in Contents a newly allocated string containing the preprocessed source.

On failure, leave Contents uninitialized and put error messages in Diagnostics.

function Create_Preprocessor (Default_Config : File_Config; File_Configs : Libadalang.Preprocessing.File_Config_Maps.Map) → File_Reader_Reference : Like Create_Preprocessor_Data, but return a file reader implementing the preprocessing instead.

function Create_Preprocessor (Default_Config : File_Config; File_Configs : Libadalang.Preprocessing.File_Config_Maps.Map) → File_Refiner_Reference 

procedure Dump (Config : File_Config; Prefix : Standard.String): Dump the content of Config on the standard output, each line prefixed with Prefix.

package File_Config_Maps is new Ada.Containers.Hashed_Maps

package File_Config_Maps is new Ada.Containers.Hashed_Maps
     (Key_Type        => US.Unbounded_String,
      Element_Type    => File_Config,
      Hash            => US.Hash,
      Equivalent_Keys => US."=");

Instantiated generic package:: File_Config_Maps

For each source file (identifier by basename only), preprocessor configuration to use.

type Preprocessor_Data

File-specific Symbol/value associations and options to run the preprocessor.

This type is a reference to constant preprocessing configuration: copying this object is cheap.

No_Preprocessor_Data : constant Preprocessor_Data

Object type:: Preprocessor_Data

No reference to preprocessor data

function "=" (Left, Right : Preprocessor_Data) → Standard.Boolean

function Parse_Preprocessor_Data_File (Filename : Standard.String; Path : GNATCOLL.File_Paths.Any_Path) → Preprocessor_Data 

Parse the preprocessor data file at Filename and return the corresponding data.

Path is used to look for the preprocessor data file itself and for definition files that the preprocessor data file may refer to.

See GNATprep’s documentation for a description of the preprocessor data file format.

function Extract_Preprocessor_Data_From_Project (Tree : GNATCOLL.Projects.Project_Tree'Class; Project : GNATCOLL.Projects.Project_Type) → Preprocessor_Data 

Create preprocessor data from compiler arguments found in the given GPR project (-gnatep and -gnateD arguments).

If a non-null Project is given, look for compiler arguments in it and the other projects in its closure. If Project is left to No_Project, try to use the whole project tree.

Note that this function collects all arguments and returns an approximation from them: it does not replicates exactly gprbuild’s behavior.

function Extract_Preprocessor_Data_From_Project (Tree : GPR2.Project.Tree.Object; Project : GPR2.Project.View.Object) → Preprocessor_Data : Likewise, but with GPR2 projects

function Default_Config (Data : Preprocessor_Data) → File_Config : Return the default file configuration in Data

function File_Configs (Data : Preprocessor_Data) → Libadalang.Preprocessing.File_Config_Maps.Map: Return all file configurations in Data

function Needs_Preprocessing (Data : Preprocessor_Data; Filename : Standard.String) → Standard.Boolean: Return whether Filename must be preprocessed according to Data

procedure Write_Preprocessor_Data_File (Data : Preprocessor_Data; Filename : Standard.String; Definition_Files_Directory : Standard.String)

Write a preprocessor data file that encodes the information in Data to Filename.

This operation may need to create definition files: they are created in Definition_Files_Directory, which must be a writeable directory. Note that all created definition files will be named defs-*.txt, so it is safe to write the preprocessor data file in the same directory with a different name.

Exceptions from Ada.Text_IO primitives are propagated when there is trouble writing files.

procedure Preprocess (Data : Preprocessor_Data; Filename, Input : Standard.String; Contents : Preprocessed_Source; Diagnostics : Langkit_Support.Diagnostics.Diagnostics_Vectors.Vector)

Preprocess the Input source buffer according to the corresponding source filename Filename and the given preprocessor data.

Note that Filename is used here only to look in Data for the File_Config value to use in order to preprocess the Input source buffer.

On success, leave Diagnostics empty and return in Contents a newly allocated string containing the preprocessed source.

On failure, leave Contents uninitialized and put error messages in Diagnostics.

procedure Dump (Data : Preprocessor_Data; Prefix : Standard.String): Dump the content of Data on the standard output, each line prefixed with Prefix.

type Preprocessed_Source

Components:

Buffer (System.Strings.String_Access) –
Last (Standard.Natural) –

procedure Free (Self : Preprocessed_Source): Deallocate the given source buffer

function Create_Preprocessor_From_File (Filename : Standard.String; Path : GNATCOLL.File_Paths.Any_Path) → File_Reader_Reference : Like Parse_Preprocessor_Data_File, but return a file reader implementing the preprocessing instead.

function Create_Preprocessor_From_File (Filename : Standard.String; Path : GNATCOLL.File_Paths.Any_Path) → File_Refiner_Reference 

procedure Dump (Definitions : Libadalang.Preprocessing.Definition_Maps.Map; Prefix : Standard.String): Dump the content of Definitions on the standard output, each line prefixed with Prefix.