1title: CMake Build Recipes
2
3# DBCSR CMake Build Recipes
4
5Following are recipes for different combinations of compilers, platforms and libraries.
6Unless otherwise noted, the examples assume that after fetching/unpacking DBCSR you created
7a directory `build/` inside the DBCSR directory and switched into it using `cd build/`.
8
9The listed examples can usually be combined with other build options like *libxsmm* or *CUDA*
10even if the examples are not explicitly given.
11
12The instructions used for building in the Continuous Integration can be found in
13the `.ci/` folder or in the `.travis.yml`.
14
15## GNU
16
17### GNU compiler, system MPI and system-provided OpenBLAS
18
19Most Linux systems provide the GNU compiler, a system MPI (OpenMPI or MPICH) using the
20GNU compiler as a backend and OpenBLAS for BLAS/LAPACK:
21
22```bash
23    cmake ..
24```
25
26### GNU compiler, system MPI and Intel MKL
27
28To use the Intel MKL together with the GNU compiler and possibly a system-MPI,
29assuming that MKL is installed in `/sw/intel/mkl`.
30
31Verified with MKL provided as part of the Intel Parallel Studio XE 2019.5 installed in `/sw/intel`
32with an OS-provided GCC 7.4.1 on Linux openSUSE Leap 15.1, using CMake 3.12.0.
33
341. Make sure the MKL environment is properly loaded:
35
36```bash
37       source /sw/intel/mkl/bin/mklvars.sh intel64
38```
39
402. Make sure CMake picks the Intel MKL over any system-provided BLAS library:
41
42```bash
43       cmake -DBLA_VENDOR=Intel10_64lp_seq ..
44```
45
46## Intel
47
48Instructions for using Intel compiler or libraries for different parts on non-Cray systems.
49For Cray systems, please check further below.
50
51*Note*: in Intel Parallel Studio 2019 there is a potential issue that `mpirun` fails with
52the error `OFI addrinfo() failed` on local (non-cluster) installations.
53This can be worked around by setting `export I_MPI_FABRICS=shm`.
54
55### Intel MPI, GNU Compiler and system-provided OpenBLAS
56
57Verified with Intel Parallel Studio XE 2019.5 installed in `/sw/intel`
58with an OS-provided GCC 7.4.1 on Linux openSUSE Leap 15.1, using CMake 3.12.0.
59
601. Make sure that the Intel environment is properly loaded:
61
62```bash
63       source /sw/intel/bin/compilervars.sh intel64
64```
65
662. Use the Intel-provided MPI compiler wrappers for the GNU toolchain,
67   to override CMake's auto-detection which may pick up the system MPI:
68
69```bash
70       CC=mpicc FC=mpifc CXX=mpicxx cmake ..
71```
72
73### Intel MPI, GNU Compiler and Intel MKL
74
75Verified with Intel Parallel Studio XE 2019.5 installed in `/sw/intel`
76with an OS-provided GCC 7.4.1 on Linux openSUSE Leap 15.1, using CMake 3.12.0.
77
781. Make sure that the Intel environment is properly loaded:
79
80```bash
81       source /sw/intel/bin/compilervars.sh intel64
82```
83
842. Use the Intel-provided MPI compiler wrappers for the GNU toolchain:
85
86```bash
87       CC=mpicc FC=mpifc CXX=mpicxx cmake -DBLA_VENDOR=Intel10_64lp_seq ..
88```
89
90### Intel MPI, Intel Compiler and Intel MKL
91
92Verified with Intel Parallel Studio XE 2019.5 installed in `/sw/intel`
93on Linux openSUSE Leap 15.1, using CMake 3.12.0.
94
951. Make sure that the Intel environment is properly loaded:
96
97```bash
98       source /sw/intel/bin/compilervars.sh intel64
99```
100
1012. Use the Intel-provided MPI compiler wrappers:
102
103```bash
104       CC=mpiicc FC=mpiifort CXX=mpiicxx cmake -DBLA_VENDOR=Intel10_64lp_seq ..
105```
106
107## MacOS
108
109Follow what is described in the previous sections.
110For GNU, if you have installed Command Line Tools by Apple and GCC with Homebrew that can lead to a
111conflict in which compiler CMake will use. Therefore, we suggest specifying GCC, for example
112
113```bash
114    CC=gcc-9 CXX=g++-9 cmake ..
115```
116
117where `-9` can be adapted to your version.
118
119### PGI
120
121Please note that you need at least PGI >= 19.11.
122
123Assuming that your `$PATH` is set correctly such that `pgcc`, `pgc++` and `pgfortran` can be found,
124run the following to get a DBCSR version without MPI:
125
126```bash
127    CC=pgcc CXX=pgc++ FC=pgfortran cmake -DUSE_MPI=OFF ..
128```
129
130the `-DUSE_MPI=OFF` is needed here to avoid that CMake picks up any MPI installation, for example from Homebrew.
131
132To build with MPI you need an MPI implementation built for/with the PGI compiler, for example the MPICH
133usually bundled with the PGI installation.
134
135Make sure that `$PATH` is correctly set to include `mpicc` and `mpifort` from the PGI MPICH installation, then run:
136
137```bash
138    CC=mpicc CXX=mpicxx FC=mpifort MPICH_CC=pgcc cmake ..
139```
140
141## Cray
142
143Some machines require additional environments to be loaded to either provide
144the modules specified below or to be able to properly build with the loaded modules.
145
146Please contact your cluster/datacenter administrator for more information.
147
148Example for the CSCS' Piz Daint:
149
150```bash
151    module load daint-mc  # to build for the non-GPU partition
152    module load daint-gpu  # to build for the GPU partition
153```
154
155*Note*: the `libsci-cray` has different variants for MPI or OpenMP.
156When disabling either MPI or OpenMP support in DBCSR you might want to adjust the
157selected BLAS/LAPACK library accordingly (e.g. drop the `_mpi`, or `_mp`).
158
159### CCE and libsci-cray
160
161Verified on CSCS' Piz Daint with CCE 9.0.2 and cray-libsci 19.06.1,
162using CMake 3.14.5.
163
1641. Make sure that the `PrgEnv-cray` module is loaded:
165
166```bash
167       module load PrgEnv-cray
168```
169
1702. While the MPI wrapper/compiler will be detected automatically,
171   must the BLAS/LAPACK libraries be specified manually:
172
173```bash
174       cmake \
175         -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment \
176         -DBLAS_LIBRARIES="-lsci_cray_mpi_mp -lhugetlbfs" \
177         -DLAPACK_LIBRARIES="-lsci_cray_mpi_mp" \
178         ..
179```
180
181### Intel Compiler and libsci-cray
182
183Verified on CSCS' Piz Daint with Intel 19.01 and cray-libsci 19.06.1,
184using CMake 3.14.5.
185
1861. Make sure that the `PrgEnv-intel` module is loaded:
187
188```bash
189       module load PrgEnv-intel
190```
191
1922. While the MPI wrapper/compiler will be detected automatically,
193   must the BLAS/LAPACK libraries be specified manually:
194
195```bash
196       cmake \
197         -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment \
198         -DBLAS_LIBRARIES="-lsci_intel_mpi_mp -lhugetlbfs" \
199         -DLAPACK_LIBRARIES="-lsci_intel_mpi_mp" \
200         ..
201```
202
203### GNU Compiler and libsci-cray
204
205Verified on CSCS' Piz Daint with GNU 8.3.0 and cray-libsci 19.06.1,
206using CMake 3.14.5.
207
2081. Make sure that the `PrgEnv-gnu` module is loaded:
209
210```bash
211       module load PrgEnv-gnu
212```
213
2142. While the MPI wrapper/compiler will be detected automatically,
215   must the BLAS/LAPACK libraries be specified manually:
216
217```bash
218       cmake \
219         -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment \
220         -DBLAS_LIBRARIES="-lsci_gnu_mpi_mp -lhugetlbfs" \
221         -DLAPACK_LIBRARIES="-lsci_gnu_mpi_mp" \
222         ..
223```
224
225## Any compiler
226
227### Custom compiler flags
228
229In the DBCSR build system we preload the default compiler flags (especially the ones for Fortran) with flags
230required to build the code with a specific compiler, while additional optimization flags are added based on the
231CMake build type.
232
233This allows the user to override optimization flags by setting a custom build type and providing optimization flags
234for that build type as follows:
235
236```bash
237       cmake \
238         -DCMAKE_BUILD_TYPE=custom \
239         -DCMAKE_C_FLAGS_CUSTOM="-O3 -march=native" \
240         -DCMAKE_CXX_FLAGS_CUSTOM="-O3 -march=native" \
241         -DCMAKE_Fortran_FLAGS_CUSTOM="-O3 -march=native" \
242         ..
243```
244