# The GAP build system

This file is meant to give an overview of how the GAP build system works. It
is targeted at people who need to work on the build system itself (to extend
it, fixes bugs in it, etc.). It should not be necessary to read this if all
you want to do is compile GAP and work on the GAP library and kernel.

Note that this really is just an overview; for details, please refer to the
comments inside the various parts of the build system.


## Prerequisites

In order to work on the build system, you need at least the following:

* GNU autoconf (we recommend 2.69 or later)
* GNU make

Note that we extensively use features provided by GNU make, so in general
another version of make, such as BSD make, is not suitable.


## Quick start: building GAP with no frills

If you are working with a fresh clone of the GAP repository, you need to
run the `autogen.sh` script first, which will generate the `configure`
script. Afterwards, or if you are using a release version of GAP, you
can follow the standard procedure:

```
./configure
make
```


== Overview of the files constituting the GAP build system

* `autogen.sh`: sets up the build system; typically the first thing to run in
  a fresh clone of the GAP repository. It runs `autoreconf` which in turn runs
  several tools include `aclocal`, `autoheader`, `autoconf`, `libtoolize`

* `configure`: generated by `autogen.sh` from `configure.ac`.

* `configure.ac`: the GNU autoconf source of our configure script.

* `GNUmakefile`, `GNUmakefile.in`: The former file is generated from the
  latter by `configure`. It is the primary Makefile (GNU make prefers it
  over `Makefile`). It only contains variables and vpath settings, and
  includes `Makefile.rules` for the actual build rules.

* `Makefile`: This is a placeholder file, and serves two purposes:
   1. If the user runs `make` before `configure`, it prints a warning.
   2. If `configure` did run, but `make` is not GNU make, it produces
      a corresponding error message.

* `Makefile.rules`: This is the core of the build system. If you want
  to add or remove a kernel C source file, you need to add or remove
  its name here and only here.

* `bin/`: This directory is created for compatibility mode (see below).

* `cnf/`: All files in this directory are part of the build system.

* `extern/`: External libraries we bundle with GAP (such as GMP) are
  put in here.

* `gen/`: Generated code (such as `config.h` and `gap_version.c`) is put
  into this directory.

* `obj/`: All `*.o` files are placed into this directory.

- `.deps/` directories contain `*.d` files generated by the build system,
  and which are used to track dependencies, e.g. of C source files on header
  files.

- `.libs/` directories are created by libtool. Please refer to the libtool
  documentation for details.


## Out-of-tree builds

The old GAP build system had a concept of "configs" and "CONFIGNAME", which
allowed you to build GAP in different configurations from a single set of
sources. This is gone in the current build system. However, a similar goal can
be achieved by using so-called "out-of-tree builds".

In the following and also in the files that make up the build system, "srcdir"
refers to the directory containing the GAP files, i.e. it contains this
README, the src and lib directories and more.

To create a new out-of-tree build, create a new directory anywhere
in your filesystem. A typical setup places the out-of-tree dirs into
subdirectories of a "build" directory inside the srcdir. So you might
have directories

```
   srcdir/build/default
   srcdir/build/default32
   srcdir/build/hpcgap
   srcdir/build/hpcgap32
   ...
```

We will refer to this directory from now on as the "builddir".

To initialize the out-of-tree build, change into the builddir and
execute the configure script from the srcdir, like this:

```
cd $builddir
$srcdir/configure
```

You can pass any additional options you like to configure, e.g. `ABI=32`
or `--enable-hpcgap`.

Once the configure script has completed, you can run `make` as usual,
and all the object files and the gap executable will be placed inside
builddir. Your srcdir will remain untouched.


## Compatibility mode

Compatibility mode emulates the build environment of the old GAP build system
in order to allow packages with kernel extensions to be used with the new
build system unmodified. As such, it mainly exists to ease the transition
between new and old build system, and the plan is to remove it once all
packages have been adapted to the new build system. However, that is still
far off.

Compatibility mode does the following things:

* create `sysinfo.gap` file in the build dir
* create a symlink `sysinfo.gap-default$ABI` pointing at `sysinfo.gap`
* create a `bin/$GAPARCH/config.h` symlink pointing at `gen/config.h`
* create a `bin/$GAPARCH/gac` symlink pointing at `gac`
* create a `bin/$GAPARCH/src` symlink pointing at `$srcdir/src`
* for out-of-tree builds, it creates a `${builddir}/src/compiled.h` file
* ...

For now, using compatibility mode is required if one wants to build the
kernel extension for most packages which have one. In the future, we will
provide an alternative way to do this, and also will extend `gac` to
cleanly supported building kernel extensions.


## Dependency tracking

The build system tracks depdencies between files, such as between C source and
header files, via `*.d` files in `.deps` directories below `obj/` and `gen/`.
These files are mostly generated by the compiler; for this, the compiler needs
to support the relevant flags (gcc, clang, icc all do so).

For a detailed explanation of a very similar scheme, see here:
<https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/>


## HPC-GAP integration

One of the main features of the new build system is that it optionally allows
to build HPC-GAP instead of plain GAP. HPC-GAP is an experimental fork of GAP
which implements concurrent programming, multi-threading, etc..

The HPC-GAP kernel and library were forked from the GAP kernel and library and
developed semi-independently for several years, with occasional merges between
the two. In order to recombine the two, we merged the HPC-GAP fork into a
subdirectory `hpcgap` of the GAP repository.  Then, all files inside `hpcgap`
which were identical to their counterparts in the GAP repository were deleted
(e.g. `hpcgap/src/ariths.c` was deleted as it was identical to `src/ariths.c`).
At this point, `hpcgap/src` has been fully merged, but there are still files
in `hpcgap/lib/` which differ from their counterparts in `lib/`

The new build system can optionally be instructed to build HPC-GAP, by
passing the `--enable-hpcgap` flag to the `configure` script. For the
resulting HPC-GAP binary to work, a trick is used:  HPC-GAP mode uses multiple
GAP root paths. Specifically, the GAP kernel function `SySetGapRootPath` was
modified so that for every root directory `FOO` that gets added, we first add
`FOO/hpcgap` to the list of root directories. This way, `GAPROOT/hpcgap/lib`
is searched first for files, and only if no matching file is found there does
GAP also search in `GAPROOT/lib`.


## Open tasks

There are many things that still need to be done in the new build system. For
an overview, see
<https://github.com/gap-system/gap/issues?q=is%3Aopen+is%3Aissue+label%3A%22topic%3A+build+system%22>

The main open task is to add support for `make install`. There are some
pitfalls to that, esp. when it comes to handling packages with kernel
extensions.


## FAQ

### Q: Why don't you just use `automake`? Then you get `make install` for free!

A: This is simply wrong. Using `automake` does not give us `make install`
support for free. While it does indeed provide various parts of the puzzle,
from our perspective those are mostly the easy ones (they may be tedious, but
are not conceptually difficult). The real obstacles for `make install` support
are unfortunately not resolved by using `automake`. Among these are:
- dealing with installing the generated `config.h`. This file should ideally
  not be installed at all and to this end not be included from any other
  headers. But note that in GAP, currently *all* headers need to include
  config.h directly or indirectly... Alternatively one can attempt to use
  something like `AX_PREFIX_CONFIG_H`, but this, too, has its pitfalls
- removing dependencies on compiler specific features in the headers and API,
  mainly as introduced by `config.h`
- making `gac` installable; in particular installing a version of GNU
  libtool needed by `gac`
- The probably hardest problem of them all is coming up with a sane way for
  dealing with the installation of GAP packages with kernel modules in such a
  way that existing workflows are not unduly broken. This has the added
  difficulty that it requires convincing the authors of ~20 GAP packages to
  apply changes needed to support this, in a way that the packages can still
  be used with the at that time latest stable release of GAP; then getting
  them to make releases of their packages with those changes.
- There are probably more bits and pieces that also need to be resolved, which
  I just can't remember right now.

Of course none of this is rocket science, but it is a lot of tedious and
finicky work, and care must be taken not to break stuff. Help with this is
welcome, but it is strongly suggested that you first talk to us, to avoid
disappointments (e.g. due to rejected pull requests) later on.

In any case: GNU `automake` does not help with this at all.

# HPC-GAP

GAP includes experimental code to support multithreaded programming in GAP,
dubbed HPC-GAP (where HPC stands for "high performance computing"). GAP and
HPC-GAP codebases diverged during the project, and we are currently working
on unifying the codebases and incorporating the HPC-GAP code back into the
mainstream GAP versions.

This is work in progress, and HPC-GAP as it is included with GAP right now
still suffers from various limitations and problems, which we are actively
working on to resolve. However, including it with GAP (disabled by default)
considerably simplifies development of HPC-GAP. It also means that you can
very easily get a (rough!) sneak peak of HPC-GAP. It comes together with the
new manual book called "HPC-GAP Reference Manual" and located in the `doc/hpc`
directory.

Users interested in experimenting with shared memory parallel programming in
GAP can build HPC-GAP by following the instructions from
<https://github.com/gap-system/gap/wiki/Building-HPC-GAP>. While it is possible
to build HPC-GAP from a release version of GAP you downloaded from the GAP
website, due to the ongoing development of HPC-GAP, we recommend that you
instead build HPC-GAP from the latest development version available in the
GAP repository at GitHub, i.e., <https://github.com/gap-system/gap>.


## HPC-GAP code in the GAP kernel

Some hints on how and where to find HPC-GAP specific code in the GAP kernel:

* The `hpcgap/src/` directory contains two files, `c_oper1.c` and `c_type1.c`,
  which are counterparts to their namesakes in `src/`; all these `c_*.c` files
  are generated by `make docomp`, which invokes the GAP compiler `gac` to
  compile `lib/oper1.g` respectively `lib/type1.g` into C code. The `c_*.c`
  files should never be manually edited.

* Code inside `src/hpc/` is mostly HPC-GAP specific. E.g. the code for dealing
  with threads, locks, guards, regions, atomic lists and records, etc., all can
  be found in here.

  Note that `src/hpc/serialize.{c,h}` and `src/hpc/traverse.{c,h}` might one
  day be used in regular GAP as well, but then they would be moved from
  `src/hpc/` to `src/`.

* Most of the remaining HPC-GAP code is inside of `#ifdef HPCGAP / #endif`
  blocks and similar constructs, and thus can be easily found by searching
  for `HPCGAP`.

* An exception to the previous rule is that `HashLock` and `HashUnlock` may be
  called in regular GAP, too, where they do nothing and are optimized away. This
  is for convenience, as typically code using these needs to call `HashUnlock`
  in multiple places, i.e., at every exit of a function, making it cumbersome
  to wrap each call into `#ifdef HPCGAP / #endif`.

* The interface to the Boehm garbage collector in `src/boehm_gc.c` is in theory
  not limited to HPC-GAP, but in practice it is, at least for now.


## HPC-GAP code in the GAP library

Normally, the GAP library is contained in the `lib` directory inside the
GAPROOT directory. But when GAP is compiled and run in HPC-GAP code mode, then
whenever a library file is to be loaded, it first looks for that file inside
`hpcgap/lib/`, and only if that fails, then it falls back to searching in
`lib`. This allows us to take a file from `lib`, duplicate it into `hpcgap/lib`,
and then apply HPC-GAP specific changes.

However, we try to keep such duplicated files to a minimum, as it is cumbersome
to maintain them: if `lib/foo.gi` is modified, then one must always remember to
also apply the same changes to its clone twin `hpcpgap/lib/foo.gi`.

Instead, it is preferable to place HPC-GAP specific code inside a conditional
using `IsHPCGAP`, like this:

```
  if IsHPCGAP then
     # perform HPC-GAP specific actions
  else
    # perform altrnative actions for regular GAP
  fi;
```

Note that `IsHPCGAP` is a global constant, set to either `true` or `false` by
the kernel before loading the library. Since GAP optimizes conditionals using
global constants away, the above code actually has zero performance overhead
when compared to the version not using it. I.e., `if IsHPCGAP` in GAP code
behaves similarly to `#ifdef HPCGAP` in C code.

As a caveat to this, code which needs to use `atomic` statements usually is
difficult to write using `if IsHPCGAP` without duplicating a lot of code. For
this reason, we allow using `atomic` statements in regular GAP, too, where
they have no effect and also cause zero overhead. Exploiting this, we allow
ourselves to use `atomic` outside of `if IsHPCGAP then ... fi` in some limited
cases inside the `lib/` directory. However, it is recommended to keep this to
a minimum, as the code becomes harder to understand and maintain for people
who are not familiar with HPC-GAP. Instead, it is preferable to find other
solutions, such as high-level abstractions which can be used uniformly by
regular GAP and HPC-GAP code, which encapsulate and hide the HPC specifics. An
example for that is `MemoizePosIntFunction`. See also
<https://github.com/gap-system/gap/issues/1889>.

As of the time this is written, only the following library files in `lib`
make use of `atomic` statements:

  1. `lib/coll.gd`
  1. `lib/coll.gi`
  1. `lib/filter.g`
  1. `lib/grppc.gi`
  1. `lib/helpdef.gi`
  1. `lib/info.gi`
  1. `lib/methwhy.g`
  1. `lib/oper.g`
  1. `lib/package.gi`
  1. `lib/profile.g`
  1. `lib/type.g`

Some further places with HPC-GAP specific code include the following:

* `hpcgap/lib/hpc/` and `hpcgap/lib/distributed/` contain all truly HPC-GAP
  specific code.

* `lib/hpc/` contains placeholders for some of the code in `hpcgap/lib/hpc/`,
   make it writing code which works in both regular GAP and HPC-GAP more
   convenient.

[![Travis build Status](https://travis-ci.org/gap-system/gap.svg?branch=master)](https://travis-ci.org/gap-system/gap)
[![AppVeyor build Status](https://ci.appveyor.com/api/projects/status/github/gap-system/gap?branch=master&svg=true)](https://ci.appveyor.com/project/gap-system/gap)
[![Code Coverage](https://codecov.io/github/gap-system/gap/coverage.svg?branch=master&token=)](https://codecov.io/gh/gap-system/gap)
[![Coverage Status](https://coveralls.io/repos/github/gap-system/gap/badge.svg)](https://coveralls.io/github/gap-system/gap)

# What is GAP?

GAP is a system for computational discrete algebra, with particular emphasis
on computational group theory. GAP provides a programming language, a library
of thousands of functions implementing algebraic algorithms written in the GAP
language as well as large data libraries of algebraic objects. For a more
detailed overview, see
  <https://www.gap-system.org/Overview/overview.html>.
For a description of the mathematical capabilities, see
  <https://www.gap-system.org/Overview/Capabilities/capabilities.html>.

GAP is used in research and teaching for studying groups and their
representations, rings, vector spaces, algebras, combinatorial structures, and
more. The system, including source, is distributed freely. You can study and
easily modify or extend it for your special use.


# How to obtain GAP?

## Download a stable release version

The latest stable release of the GAP system, including all currently
redistributed GAP packages, can be obtained from
  <https://www.gap-system.org/Releases/index.html>.
Afterwards, follow the instructions in the file `INSTALL.md` in the GAP root
directory.


# Using a GAP development version

Alternatively, you can compile the latest development version of GAP. However,
most users should instead use the latest official release instead as described
in the previous section.

If you really want to use a development version of GAP, start by cloning the
GAP source repository using git:

    git clone https://github.com/gap-system/gap


## Installing required dependencies

In this case, you need to have some more software dependencies installed than
with a stable release in order to compile GAP. In particular, you need at
least these:

* a C compiler, e.g. GCC or Clang
* a C++ compiler
* GNU Make
* GNU Autoconf
* GNU Libtool

In addition, we recommend that you install at least the following optional
dependencies:
* Development headers for GMP, the GNU Multiple Precision Arithmetic Library
* Development headers for zlib
* Development headers for GNU Readline

On Ubuntu or Debian, you can install these with the following command:

    sudo apt-get install build-essential autoconf libtool libgmp-dev libreadline-dev zlib1g-dev

On macOS, you can install the dependencies in several ways:

 * using Homebrew: `brew install autoconf libtool gmp readline`
 * using Fink: `fink install autoconf2.6 libtool2 gmp5 readline7`
 * using MacPorts: `port install autoconf libtool gmp readline`

On other operating systems, you will need to figure out equivalent commands
to install the required dependencies.


## Building GAP

Then to build GAP, first run this command to generate the `configure` script:

    ./autogen.sh

Afterwards you can proceed similar to what is described in `INSTALL.md`, in
particular enter the following commands to compile GAP itself (for macOS users,
see below for a few additional hints):

    ./configure
    make

For macOS users you may ned to tell GAP where it can find these dependencies.

For Homebrew, use these commands:

    ./configure --with-readline=/usr/local/opt/readline
    make

For Fink, use these commands:

    ./configure CPPFLAGS=-I/sw/include LDFLAGS=-L/sw/lib
    make

For MacPorts, use these commands:

    ./configure CPPFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib
    make


## Obtaining the GAP package distribution

In contrast to the GAP stable releases, the development version does not come
bundled with all the GAP packages. Therefore, if you do not have a GAP package
archive yet, we recommend that you bootstrap the stable versions of packages
by executing one of the following commands. Whether you choose to
`bootstrap-pkg-minimal` or `bootstrap-pkg-full` depends on your needs for
development.

    make bootstrap-pkg-minimal

or

    make bootstrap-pkg-full

In the latter case please note that `make bootstrap-pkg-full` only unpacks packages
but does not build those of them that require compilation. You can change to the
`pkg` directory and then call `../bin/BuildPackages.sh` from there to build as many
packages as possible.

If everything goes well, you should be able to start GAP by executing

    sh bin/gap.sh

You can also find development versions of some of the GAP packages on
<https://github.com/gap-packages> resp. on <https://gap-packages.github.io>.


# We welcome contributions

The GAP Project welcomes contributions from everyone, in the shape of code,
documentation, blog posts, or other. For contributions to this repository,
please read the [guidelines](CONTRIBUTING.md).

To keep up to date on GAP news (discussion of problems, release announcements,
bug fixes), you can subscribe to the
[GAP forum](https://www.gap-system.org/Contacts/Forum/forum.html) and
[GAP development](https://mail.gap-system.org/mailman/listinfo/gap) mailing lists,
notifications on GitHub, and follow us on [Twitter](https://twitter.com/gap_system).

If you have any questions about working with GAP, you can ask them on
[GAP forum](https://www.gap-system.org/Contacts/Forum/forum.html) (requires subscription)
or [GAP Support](https://www.gap-system.org/Contacts/People/supportgroup.html) mailing lists.

Please tell us about your use of GAP in research or teaching. We maintain a
[bibliography of publications citing GAP](https://www.gap-system.org/Doc/Bib/bib.html).
Please [help us](https://www.gap-system.org/Contacts/publicationfeedback.html)
keeping it up to date.


# License

GAP is free software; you can redistribute it and/or modify it under the terms
of the GNU General Public License as published by the Free Software
Foundation; either version 2 of the License, or (at your option) any later
version. For details, please refer to the GAP reference manual, as well as the
file `LICENSE` in the root directory of the GAP distribution or see
<https://www.gnu.org/licenses/gpl.html>.