1title: CMake Build Recipes 2 3# DBCSR CMake Build Recipes 4 5Following are recipes for different combinations of compilers, platforms and libraries. 6Unless otherwise noted, the examples assume that after fetching/unpacking DBCSR you created 7a directory `build/` inside the DBCSR directory and switched into it using `cd build/`. 8 9The listed examples can usually be combined with other build options like *libxsmm* or *CUDA* 10even if the examples are not explicitly given. 11 12The instructions used for building in the Continuous Integration can be found in 13the `.ci/` folder or in the `.travis.yml`. 14 15## GNU 16 17### GNU compiler, system MPI and system-provided OpenBLAS 18 19Most Linux systems provide the GNU compiler, a system MPI (OpenMPI or MPICH) using the 20GNU compiler as a backend and OpenBLAS for BLAS/LAPACK: 21 22```bash 23 cmake .. 24``` 25 26### GNU compiler, system MPI and Intel MKL 27 28To use the Intel MKL together with the GNU compiler and possibly a system-MPI, 29assuming that MKL is installed in `/sw/intel/mkl`. 30 31Verified with MKL provided as part of the Intel Parallel Studio XE 2019.5 installed in `/sw/intel` 32with an OS-provided GCC 7.4.1 on Linux openSUSE Leap 15.1, using CMake 3.12.0. 33 341. Make sure the MKL environment is properly loaded: 35 36```bash 37 source /sw/intel/mkl/bin/mklvars.sh intel64 38``` 39 402. Make sure CMake picks the Intel MKL over any system-provided BLAS library: 41 42```bash 43 cmake -DBLA_VENDOR=Intel10_64lp_seq .. 44``` 45 46## Intel 47 48Instructions for using Intel compiler or libraries for different parts on non-Cray systems. 49For Cray systems, please check further below. 50 51*Note*: in Intel Parallel Studio 2019 there is a potential issue that `mpirun` fails with 52the error `OFI addrinfo() failed` on local (non-cluster) installations. 53This can be worked around by setting `export I_MPI_FABRICS=shm`. 54 55### Intel MPI, GNU Compiler and system-provided OpenBLAS 56 57Verified with Intel Parallel Studio XE 2019.5 installed in `/sw/intel` 58with an OS-provided GCC 7.4.1 on Linux openSUSE Leap 15.1, using CMake 3.12.0. 59 601. Make sure that the Intel environment is properly loaded: 61 62```bash 63 source /sw/intel/bin/compilervars.sh intel64 64``` 65 662. Use the Intel-provided MPI compiler wrappers for the GNU toolchain, 67 to override CMake's auto-detection which may pick up the system MPI: 68 69```bash 70 CC=mpicc FC=mpifc CXX=mpicxx cmake .. 71``` 72 73### Intel MPI, GNU Compiler and Intel MKL 74 75Verified with Intel Parallel Studio XE 2019.5 installed in `/sw/intel` 76with an OS-provided GCC 7.4.1 on Linux openSUSE Leap 15.1, using CMake 3.12.0. 77 781. Make sure that the Intel environment is properly loaded: 79 80```bash 81 source /sw/intel/bin/compilervars.sh intel64 82``` 83 842. Use the Intel-provided MPI compiler wrappers for the GNU toolchain: 85 86```bash 87 CC=mpicc FC=mpifc CXX=mpicxx cmake -DBLA_VENDOR=Intel10_64lp_seq .. 88``` 89 90### Intel MPI, Intel Compiler and Intel MKL 91 92Verified with Intel Parallel Studio XE 2019.5 installed in `/sw/intel` 93on Linux openSUSE Leap 15.1, using CMake 3.12.0. 94 951. Make sure that the Intel environment is properly loaded: 96 97```bash 98 source /sw/intel/bin/compilervars.sh intel64 99``` 100 1012. Use the Intel-provided MPI compiler wrappers: 102 103```bash 104 CC=mpiicc FC=mpiifort CXX=mpiicxx cmake -DBLA_VENDOR=Intel10_64lp_seq .. 105``` 106 107## MacOS 108 109Follow what is described in the previous sections. 110For GNU, if you have installed Command Line Tools by Apple and GCC with Homebrew that can lead to a 111conflict in which compiler CMake will use. Therefore, we suggest specifying GCC, for example 112 113```bash 114 CC=gcc-9 CXX=g++-9 cmake .. 115``` 116 117where `-9` can be adapted to your version. 118 119### PGI 120 121Please note that you need at least PGI >= 19.11. 122 123Assuming that your `$PATH` is set correctly such that `pgcc`, `pgc++` and `pgfortran` can be found, 124run the following to get a DBCSR version without MPI: 125 126```bash 127 CC=pgcc CXX=pgc++ FC=pgfortran cmake -DUSE_MPI=OFF .. 128``` 129 130the `-DUSE_MPI=OFF` is needed here to avoid that CMake picks up any MPI installation, for example from Homebrew. 131 132To build with MPI you need an MPI implementation built for/with the PGI compiler, for example the MPICH 133usually bundled with the PGI installation. 134 135Make sure that `$PATH` is correctly set to include `mpicc` and `mpifort` from the PGI MPICH installation, then run: 136 137```bash 138 CC=mpicc CXX=mpicxx FC=mpifort MPICH_CC=pgcc cmake .. 139``` 140 141## Cray 142 143Some machines require additional environments to be loaded to either provide 144the modules specified below or to be able to properly build with the loaded modules. 145 146Please contact your cluster/datacenter administrator for more information. 147 148Example for the CSCS' Piz Daint: 149 150```bash 151 module load daint-mc # to build for the non-GPU partition 152 module load daint-gpu # to build for the GPU partition 153``` 154 155*Note*: the `libsci-cray` has different variants for MPI or OpenMP. 156When disabling either MPI or OpenMP support in DBCSR you might want to adjust the 157selected BLAS/LAPACK library accordingly (e.g. drop the `_mpi`, or `_mp`). 158 159### CCE and libsci-cray 160 161Verified on CSCS' Piz Daint with CCE 9.0.2 and cray-libsci 19.06.1, 162using CMake 3.14.5. 163 1641. Make sure that the `PrgEnv-cray` module is loaded: 165 166```bash 167 module load PrgEnv-cray 168``` 169 1702. While the MPI wrapper/compiler will be detected automatically, 171 must the BLAS/LAPACK libraries be specified manually: 172 173```bash 174 cmake \ 175 -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment \ 176 -DBLAS_LIBRARIES="-lsci_cray_mpi_mp -lhugetlbfs" \ 177 -DLAPACK_LIBRARIES="-lsci_cray_mpi_mp" \ 178 .. 179``` 180 181### Intel Compiler and libsci-cray 182 183Verified on CSCS' Piz Daint with Intel 19.01 and cray-libsci 19.06.1, 184using CMake 3.14.5. 185 1861. Make sure that the `PrgEnv-intel` module is loaded: 187 188```bash 189 module load PrgEnv-intel 190``` 191 1922. While the MPI wrapper/compiler will be detected automatically, 193 must the BLAS/LAPACK libraries be specified manually: 194 195```bash 196 cmake \ 197 -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment \ 198 -DBLAS_LIBRARIES="-lsci_intel_mpi_mp -lhugetlbfs" \ 199 -DLAPACK_LIBRARIES="-lsci_intel_mpi_mp" \ 200 .. 201``` 202 203### GNU Compiler and libsci-cray 204 205Verified on CSCS' Piz Daint with GNU 8.3.0 and cray-libsci 19.06.1, 206using CMake 3.14.5. 207 2081. Make sure that the `PrgEnv-gnu` module is loaded: 209 210```bash 211 module load PrgEnv-gnu 212``` 213 2142. While the MPI wrapper/compiler will be detected automatically, 215 must the BLAS/LAPACK libraries be specified manually: 216 217```bash 218 cmake \ 219 -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment \ 220 -DBLAS_LIBRARIES="-lsci_gnu_mpi_mp -lhugetlbfs" \ 221 -DLAPACK_LIBRARIES="-lsci_gnu_mpi_mp" \ 222 .. 223``` 224 225## Any compiler 226 227### Custom compiler flags 228 229In the DBCSR build system we preload the default compiler flags (especially the ones for Fortran) with flags 230required to build the code with a specific compiler, while additional optimization flags are added based on the 231CMake build type. 232 233This allows the user to override optimization flags by setting a custom build type and providing optimization flags 234for that build type as follows: 235 236```bash 237 cmake \ 238 -DCMAKE_BUILD_TYPE=custom \ 239 -DCMAKE_C_FLAGS_CUSTOM="-O3 -march=native" \ 240 -DCMAKE_CXX_FLAGS_CUSTOM="-O3 -march=native" \ 241 -DCMAKE_Fortran_FLAGS_CUSTOM="-O3 -march=native" \ 242 .. 243``` 244