1Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
2 University Research and Technology
3 Corporation. All rights reserved.
4Copyright (c) 2004-2007 The University of Tennessee and The University
5 of Tennessee Research Foundation. All rights
6 reserved.
7Copyright (c) 2004-2008 High Performance Computing Center Stuttgart,
8 University of Stuttgart. All rights reserved.
9Copyright (c) 2004-2007 The Regents of the University of California.
10 All rights reserved.
11Copyright (c) 2006-2018 Cisco Systems, Inc. All rights reserved.
12Copyright (c) 2006-2011 Mellanox Technologies. All rights reserved.
13Copyright (c) 2006-2012 Oracle and/or its affiliates. All rights reserved.
14Copyright (c) 2007 Myricom, Inc. All rights reserved.
15Copyright (c) 2008-2017 IBM Corporation. All rights reserved.
16Copyright (c) 2010 Oak Ridge National Labs. All rights reserved.
17Copyright (c) 2011 University of Houston. All rights reserved.
18Copyright (c) 2013-2017 Intel, Inc. All rights reserved.
19Copyright (c) 2015 NVIDIA Corporation. All rights reserved.
20Copyright (c) 2017-2018 Los Alamos National Security, LLC. All rights
21 reserved.
22Copyright (c) 2017 Research Organization for Information Science
23 and Technology (RIST). All rights reserved.
24
25$COPYRIGHT$
26
27Additional copyrights may follow
28
29$HEADER$
30
31===========================================================================
32
33When submitting questions and problems, be sure to include as much
34extra information as possible. This web page details all the
35information that we request in order to provide assistance:
36
37 http://www.open-mpi.org/community/help/
38
39The best way to report bugs, send comments, or ask questions is to
40sign up on the user's and/or developer's mailing list (for user-level
41and developer-level questions; when in doubt, send to the user's
42list):
43
44 users@lists.open-mpi.org
45 devel@lists.open-mpi.org
46
47Because of spam, only subscribers are allowed to post to these lists
48(ensure that you subscribe with and post from exactly the same e-mail
49address -- joe@example.com is considered different than
50joe@mycomputer.example.com!). Visit these pages to subscribe to the
51lists:
52
53 http://lists.open-mpi.org/mailman/listinfo/users
54 http://lists.open-mpi.org/mailman/listinfo/devel
55
56Thanks for your time.
57
58===========================================================================
59
60Much, much more information is also available in the Open MPI FAQ:
61
62 https://www.open-mpi.org/faq/
63
64===========================================================================
65
66The following abbreviated list of release notes applies to this code
67base as of this writing (January 2020):
68
69General notes
70-------------
71
72- Open MPI now includes two public software layers: MPI and OpenSHMEM.
73 Throughout this document, references to Open MPI implicitly include
74 both of these layers. When distinction between these two layers is
75 necessary, we will reference them as the "MPI" and "OpenSHMEM"
76 layers respectively.
77
78- OpenSHMEM is a collaborative effort between academia, industry, and
79 the U.S. Government to create a specification for a standardized API
80 for parallel programming in the Partitioned Global Address Space
81 (PGAS). For more information about the OpenSHMEM project, including
82 access to the current OpenSHMEM specification, please visit:
83
84 http://openshmem.org/
85
86 This OpenSHMEM implementation will only work in Linux environments
87 with a restricted set of supported networks.
88
89- Open MPI includes support for a wide variety of supplemental
90 hardware and software package. When configuring Open MPI, you may
91 need to supply additional flags to the "configure" script in order
92 to tell Open MPI where the header files, libraries, and any other
93 required files are located. As such, running "configure" by itself
94 may not include support for all the devices (etc.) that you expect,
95 especially if their support headers / libraries are installed in
96 non-standard locations. Network interconnects are an easy example
97 to discuss -- Libfabric and OpenFabrics networks, for example, both
98 have supplemental headers and libraries that must be found before
99 Open MPI can build support for them. You must specify where these
100 files are with the appropriate options to configure. See the
101 listing of configure command-line switches, below, for more details.
102
103- The majority of Open MPI's documentation is here in this file, the
104 included man pages, and on the web site FAQ
105 (https://www.open-mpi.org/).
106
107- Note that Open MPI documentation uses the word "component"
108 frequently; the word "plugin" is probably more familiar to most
109 users. As such, end users can probably completely substitute the
110 word "plugin" wherever you see "component" in our documentation.
111 For what it's worth, we use the word "component" for historical
112 reasons, mainly because it is part of our acronyms and internal API
113 function calls.
114
115- The run-time systems that are currently supported are:
116 - rsh / ssh
117 - PBS Pro, Torque
118 - Platform LSF (v7.0.2 and later)
119 - SLURM
120 - Cray XE, XC, and XK
121 - Oracle Grid Engine (OGE) 6.1, 6.2 and open source Grid Engine
122
123- Systems that have been tested are:
124 - Linux (various flavors/distros), 64 bit (x86), with gcc, Absoft,
125 Intel, and Portland (*)
126 - macOS (10.12), 64 bit (x85_64) with XCode compilers
127
128 (*) Be sure to read the Compiler Notes, below.
129
130- Other systems have been lightly (but not fully tested):
131 - Linux (various flavors/distros), 32 bit, with gcc
132 - Cygwin 32 & 64 bit with gcc
133 - ARMv6, ARMv7, ARMv8 (aarch64)
134 - Other 64 bit platforms (e.g., Linux on PPC64)
135 - Oracle Solaris 10 and 11, 32 and 64 bit (SPARC, i386, x86_64),
136 with Oracle Solaris Studio 12.5
137 - OpenBSD. Requires configure options --enable-mca-no-build=patcher
138 and --disable-slopen with this release.
139 - Problems have been reported when building Open MPI on FreeBSD 11.1
140 using the clang-4.0 system compiler. A workaround is to build
141 Open MPI using the GNU compiler.
142
143Platform Notes
144--------------
145
146- N/A
147
148
149Compiler Notes
150--------------
151
152- Open MPI requires a C99-capable compiler to build.
153
154- Mixing compilers from different vendors when building Open MPI
155 (e.g., using the C/C++ compiler from one vendor and the Fortran
156 compiler from a different vendor) has been successfully employed by
157 some Open MPI users (discussed on the Open MPI user's mailing list),
158 but such configurations are not tested and not documented. For
159 example, such configurations may require additional compiler /
160 linker flags to make Open MPI build properly.
161
162- In general, the latest versions of compilers of a given vendor's
163 series have the least bugs. We have seen cases where Vendor XYZ's
164 compiler version A.B fails to compile Open MPI, but version A.C
165 (where C>B) works just fine. If you run into a compile failure, you
166 might want to double check that you have the latest bug fixes and
167 patches for your compiler.
168
169- Users have reported issues with older versions of the Fortran PGI
170 compiler suite when using Open MPI's (non-default) --enable-debug
171 configure option. Per the above advice of using the most recent
172 version of a compiler series, the Open MPI team recommends using the
173 latest version of the PGI suite, and/or not using the --enable-debug
174 configure option. If it helps, here's what we have found with some
175 (not comprehensive) testing of various versions of the PGI compiler
176 suite:
177
178 pgi-8 : NO known good version with --enable-debug
179 pgi-9 : 9.0-4 known GOOD
180 pgi-10: 10.0-0 known GOOD
181 pgi-11: NO known good version with --enable-debug
182 pgi-12: 12.10 known BAD with -m32, but known GOOD without -m32
183 (and 12.8 and 12.9 both known BAD with --enable-debug)
184 pgi-13: 13.9 known BAD with -m32, 13.10 known GOOD without -m32
185 pgi-15: 15.10 known BAD with -m32
186
187- Similarly, there is a known Fortran PGI compiler issue with long
188 source directory path names that was resolved in 9.0-4 (9.0-3 is
189 known to be broken in this regard).
190
191- Open MPI does not support the PGI compiler suite on OS X or MacOS.
192 See issues below for more details:
193 https://github.com/open-mpi/ompi/issues/2604
194 https://github.com/open-mpi/ompi/issues/2605
195
196- OpenSHMEM Fortran bindings do not support the `no underscore` Fortran
197 symbol convention. IBM's xlf compilers build in that mode by default.
198 As such, IBM's xlf compilers cannot build/link the OpenSHMEM Fortran
199 bindings by default. A workaround is to pass FC="xlf -qextname" at
200 configure time to force a trailing underscore. See the issue below
201 for more details:
202 https://github.com/open-mpi/ompi/issues/3612
203
204- MPI applications that use the mpi_f08 module on PowerPC platforms
205 (tested ppc64le) will likely experience runtime failures if:
206 - they are using a GNU linker (ld) version after v2.25.1 and before v2.28,
207 -and-
208 - they compiled with PGI (tested 17.5) or XL (tested v15.1.5) compilers.
209 This was noticed on Ubuntu 16.04 which uses the 2.26.1 version of ld by
210 default. However, this issue impacts any OS using a version of ld noted
211 above. This GNU linker regression will be fixed in version 2.28.
212 Below is a link to the GNU bug on this issue:
213 https://sourceware.org/bugzilla/show_bug.cgi?id=21306
214 The XL compiler will include a fix for this issue in a future release.
215
216- On NetBSD-6 (at least AMD64 and i386), and possibly on OpenBSD,
217 libtool misidentifies properties of f95/g95, leading to obscure
218 compile-time failures if used to build Open MPI. You can work
219 around this issue by ensuring that libtool will not use f95/g95
220 (e.g., by specifying FC=<some_other_compiler>, or otherwise ensuring
221 a different Fortran compiler will be found earlier in the path than
222 f95/g95), or by disabling the Fortran MPI bindings with
223 --disable-mpi-fortran.
224
225- On OpenBSD/i386, if you configure with
226 --enable-mca-no-build=patcher, you will also need to add
227 --disable-dlopen. Otherwise, odd crashes can occur
228 nondeterministically.
229
230- Absoft 11.5.2 plus a service pack from September 2012 (which Absoft
231 says is available upon request), or a version later than 11.5.2
232 (e.g., 11.5.3), is required to compile the Fortran mpi_f08
233 module.
234
235- Open MPI does not support the Sparc v8 CPU target. However,
236 as of Solaris Studio 12.1, and later compilers, one should not
237 specify -xarch=v8plus or -xarch=v9. The use of the options
238 -m32 and -m64 for producing 32 and 64 bit targets, respectively,
239 are now preferred by the Solaris Studio compilers. GCC may
240 require either "-m32" or "-mcpu=v9 -m32", depending on GCC version.
241
242- It has been noticed that if one uses CXX=sunCC, in which sunCC
243 is a link in the Solaris Studio compiler release, that the OMPI
244 build system has issue with sunCC and does not build libmpi_cxx.so.
245 Therefore the make install fails. So we suggest that one should
246 use CXX=CC, which works, instead of CXX=sunCC.
247
248- If one tries to build OMPI on Ubuntu with Solaris Studio using the C++
249 compiler and the -m32 option, you might see a warning:
250
251 CC: Warning: failed to detect system linker version, falling back to
252 custom linker usage
253
254 And the build will fail. One can overcome this error by either
255 setting LD_LIBRARY_PATH to the location of the 32 bit libraries (most
256 likely /lib32), or giving LDFLAGS="-L/lib32 -R/lib32" to the configure
257 command. Officially, Solaris Studio is not supported on Ubuntu Linux
258 distributions, so additional problems might be incurred.
259
260- Open MPI does not support the gccfss compiler (GCC For SPARC
261 Systems; a now-defunct compiler project from Sun).
262
263- At least some versions of the Intel 8.1 compiler seg fault while
264 compiling certain Open MPI source code files. As such, it is not
265 supported.
266
267- The Intel 9.0 v20051201 compiler on IA64 platforms seems to have a
268 problem with optimizing the ptmalloc2 memory manager component (the
269 generated code will segv). As such, the ptmalloc2 component will
270 automatically disable itself if it detects that it is on this
271 platform/compiler combination. The only effect that this should
272 have is that the MCA parameter mpi_leave_pinned will be inoperative.
273
274- It has been reported that the Intel 9.1 and 10.0 compilers fail to
275 compile Open MPI on IA64 platforms. As of 12 Sep 2012, there is
276 very little (if any) testing performed on IA64 platforms (with any
277 compiler). Support is "best effort" for these platforms, but it is
278 doubtful that any effort will be expended to fix the Intel 9.1 /
279 10.0 compiler issuers on this platform.
280
281- Early versions of the Intel 12.1 Linux compiler suite on x86_64 seem
282 to have a bug that prevents Open MPI from working. Symptoms
283 including immediate segv of the wrapper compilers (e.g., mpicc) and
284 MPI applications. As of 1 Feb 2012, if you upgrade to the latest
285 version of the Intel 12.1 Linux compiler suite, the problem will go
286 away.
287
288- Early versions of the Portland Group 6.0 compiler have problems
289 creating the C++ MPI bindings as a shared library (e.g., v6.0-1).
290 Tests with later versions show that this has been fixed (e.g.,
291 v6.0-5).
292
293- The Portland Group compilers prior to version 7.0 require the
294 "-Msignextend" compiler flag to extend the sign bit when converting
295 from a shorter to longer integer. This is is different than other
296 compilers (such as GNU). When compiling Open MPI with the Portland
297 compiler suite, the following flags should be passed to Open MPI's
298 configure script:
299
300 shell$ ./configure CFLAGS=-Msignextend CXXFLAGS=-Msignextend \
301 --with-wrapper-cflags=-Msignextend \
302 --with-wrapper-cxxflags=-Msignextend ...
303
304 This will both compile Open MPI with the proper compile flags and
305 also automatically add "-Msignextend" when the C and C++ MPI wrapper
306 compilers are used to compile user MPI applications.
307
308- It has been reported that Pathscale 5.0.5 and 6.0.527 compilers
309 give an internal compiler error when trying to Open MPI.
310
311- Using the MPI C++ bindings with older versions of the Pathscale
312 compiler on some platforms is an old issue that seems to be a
313 problem when Pathscale uses a back-end GCC 3.x compiler. Here's a
314 proposed solution from the Pathscale support team (from July 2010):
315
316 The proposed work-around is to install gcc-4.x on the system and
317 use the pathCC -gnu4 option. Newer versions of the compiler (4.x
318 and beyond) should have this fixed, but we'll have to test to
319 confirm it's actually fixed and working correctly.
320
321 We don't anticipate that this will be much of a problem for Open MPI
322 users these days (our informal testing shows that not many users are
323 still using GCC 3.x). Contact Pathscale support if you continue to
324 have problems with Open MPI's C++ bindings.
325
326 Note the MPI C++ bindings have been deprecated by the MPI Forum and
327 may not be supported in future releases.
328
329- As of July 2017, the Pathscale compiler suite apparently has no
330 further commercial support, and it does not look like there will be
331 further releases. Any issues discovered regarding building /
332 running Open MPI with the Pathscale compiler suite therefore may not
333 be able to be resolved.
334
335- Using the Absoft compiler to build the MPI Fortran bindings on Suse
336 9.3 is known to fail due to a Libtool compatibility issue.
337
338- MPI Fortran API support has been completely overhauled since the
339 Open MPI v1.5/v1.6 series.
340
341 ********************************************************************
342 ********************************************************************
343 *** There is now only a single Fortran MPI wrapper compiler and a
344 *** single Fortran OpenSHMEM wrapper compiler: mpifort and oshfort,
345 *** respectively. mpif77 and mpif90 still exist, but they are
346 *** symbolic links to mpifort.
347 ********************************************************************
348 *** Similarly, Open MPI's configure script only recognizes the FC
349 *** and FCFLAGS environment variables (to specify the Fortran
350 *** compiler and compiler flags, respectively). The F77 and FFLAGS
351 *** environment variables are IGNORED.
352 ********************************************************************
353 ********************************************************************
354
355 As a direct result, it is STRONGLY recommended that you specify a
356 Fortran compiler that uses file suffixes to determine Fortran code
357 layout (e.g., free form vs. fixed). For example, with some versions
358 of the IBM XLF compiler, it is preferable to use FC=xlf instead of
359 FC=xlf90, because xlf will automatically determine the difference
360 between free form and fixed Fortran source code.
361
362 However, many Fortran compilers allow specifying additional
363 command-line arguments to indicate which Fortran dialect to use.
364 For example, if FC=xlf90, you may need to use "mpifort --qfixed ..."
365 to compile fixed format Fortran source files.
366
367 You can use either ompi_info or oshmem_info to see with which Fortran
368 compiler Open MPI was configured and compiled.
369
370 There are up to three sets of Fortran MPI bindings that may be
371 provided depending on your Fortran compiler):
372
373 - mpif.h: This is the first MPI Fortran interface that was defined
374 in MPI-1. It is a file that is included in Fortran source code.
375 Open MPI's mpif.h does not declare any MPI subroutines; they are
376 all implicit.
377
378 - mpi module: The mpi module file was added in MPI-2. It provides
379 strong compile-time parameter type checking for MPI subroutines.
380
381 - mpi_f08 module: The mpi_f08 module was added in MPI-3. It
382 provides many advantages over the mpif.h file and mpi module. For
383 example, MPI handles have distinct types (vs. all being integers).
384 See the MPI-3 document for more details.
385
386 *** The mpi_f08 module is STRONGLY is recommended for all new MPI
387 Fortran subroutines and applications. Note that the mpi_f08
388 module can be used in conjunction with the other two Fortran
389 MPI bindings in the same application (only one binding can be
390 used per subroutine/function, however). Full interoperability
391 between mpif.h/mpi module and mpi_f08 module MPI handle types
392 is provided, allowing mpi_f08 to be used in new subroutines in
393 legacy MPI applications.
394
395 Per the OpenSHMEM specification, there is only one Fortran OpenSHMEM
396 binding provided:
397
398 - shmem.fh: All Fortran OpenSHMEM programs **should** include
399 'shmem.fh', and Fortran OpenSHMEM programs that use constants
400 defined by OpenSHMEM **MUST** include 'shmem.fh'.
401
402 The following notes apply to the above-listed Fortran bindings:
403
404 - All Fortran compilers support the mpif.h/shmem.fh-based bindings,
405 with one exception: the MPI_SIZEOF interfaces will only be present
406 when Open MPI is built with a Fortran compiler that support the
407 INTERFACE keyword and ISO_FORTRAN_ENV. Most notably, this
408 excludes the GNU Fortran compiler suite before version 4.9.
409
410 - The level of support provided by the mpi module is based on your
411 Fortran compiler.
412
413 If Open MPI is built with a non-GNU Fortran compiler, or if Open
414 MPI is built with the GNU Fortran compiler >= v4.9, all MPI
415 subroutines will be prototyped in the mpi module. All calls to
416 MPI subroutines will therefore have their parameter types checked
417 at compile time.
418
419 If Open MPI is built with an old gfortran (i.e., < v4.9), a
420 limited "mpi" module will be built. Due to the limitations of
421 these compilers, and per guidance from the MPI-3 specification,
422 all MPI subroutines with "choice" buffers are specifically *not*
423 included in the "mpi" module, and their parameters will not be
424 checked at compile time. Specifically, all MPI subroutines with
425 no "choice" buffers are prototyped and will receive strong
426 parameter type checking at run-time (e.g., MPI_INIT,
427 MPI_COMM_RANK, etc.).
428
429 Similar to the mpif.h interface, MPI_SIZEOF is only supported on
430 Fortran compilers that support INTERFACE and ISO_FORTRAN_ENV.
431
432 - The mpi_f08 module has been tested with the Intel Fortran compiler
433 and gfortran >= 4.9. Other modern Fortran compilers likely also
434 work.
435
436 Many older Fortran compilers do not provide enough modern Fortran
437 features to support the mpi_f08 module. For example, gfortran <
438 v4.9 does provide enough support for the mpi_f08 module.
439
440 You can examine the output of the following command to see all
441 the Fortran features that are/are not enabled in your Open MPI
442 installation:
443
444 shell$ ompi_info | grep -i fort
445
446
447General Run-Time Support Notes
448------------------------------
449
450- The Open MPI installation must be in your PATH on all nodes (and
451 potentially LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH), if libmpi/libshmem
452 is a shared library), unless using the --prefix or
453 --enable-mpirun-prefix-by-default functionality (see below).
454
455- Open MPI's run-time behavior can be customized via MPI Component
456 Architecture (MCA) parameters (see below for more information on how
457 to get/set MCA parameter values). Some MCA parameters can be set in
458 a way that renders Open MPI inoperable (see notes about MCA
459 parameters later in this file). In particular, some parameters have
460 required options that must be included.
461
462 - If specified, the "btl" parameter must include the "self"
463 component, or Open MPI will not be able to deliver messages to the
464 same rank as the sender. For example: "mpirun --mca btl tcp,self
465 ..."
466 - If specified, the "btl_tcp_if_exclude" parameter must include the
467 loopback device ("lo" on many Linux platforms), or Open MPI will
468 not be able to route MPI messages using the TCP BTL. For example:
469 "mpirun --mca btl_tcp_if_exclude lo,eth1 ..."
470
471- Running on nodes with different endian and/or different datatype
472 sizes within a single parallel job is supported in this release.
473 However, Open MPI does not resize data when datatypes differ in size
474 (for example, sending a 4 byte MPI_DOUBLE and receiving an 8 byte
475 MPI_DOUBLE will fail).
476
477
478MPI Functionality and Features
479------------------------------
480
481- All MPI-3 functionality is supported.
482
483- Rank reordering support is available using the TreeMatch library. It
484 is activated for the graph and dist_graph topologies.
485
486- When using MPI deprecated functions, some compilers will emit
487 warnings. For example:
488
489 shell$ cat deprecated_example.c
490 #include <mpi.h>
491 void foo(void) {
492 MPI_Datatype type;
493 MPI_Type_struct(1, NULL, NULL, NULL, &type);
494 }
495 shell$ mpicc -c deprecated_example.c
496 deprecated_example.c: In function 'foo':
497 deprecated_example.c:4: warning: 'MPI_Type_struct' is deprecated (declared at /opt/openmpi/include/mpi.h:1522)
498 shell$
499
500- MPI_THREAD_MULTIPLE is supported with some exceptions. Note that
501 Open MPI must be configured with --enable-mpi-thread-multiple to get
502 this level of thread safety support.
503
504 The following PMLs support MPI_THREAD_MULTIPLE:
505 - cm (see list (1) of supported MTLs, below)
506 - ob1 (see list (2) of supported BTLs, below)
507 - ucx
508 - yalla
509
510 (1) The cm PML and the following MTLs support MPI_THREAD_MULTIPLE:
511 - MXM
512 - ofi (Libfabric)
513 - portals4
514
515 (2) The ob1 PML and the following BTLs support MPI_THREAD_MULTIPLE:
516 - openib (see exception below)
517 - self
518 - sm
519 - smcuda
520 - tcp
521 - ugni
522 - usnic
523 - vader (shared memory)
524
525 The openib BTL's RDMACM based connection setup mechanism is also not
526 thread safe. The default UDCM method should be used for
527 applications requiring MPI_THREAD_MULTIPLE support.
528
529 Currently, MPI File operations are not thread safe even if MPI is
530 initialized for MPI_THREAD_MULTIPLE support.
531
532- MPI_REAL16 and MPI_COMPLEX32 are only supported on platforms where a
533 portable C datatype can be found that matches the Fortran type
534 REAL*16, both in size and bit representation.
535
536- The "libompitrace" library is bundled in Open MPI and is installed
537 by default (it can be disabled via the --disable-libompitrace
538 flag). This library provides a simplistic tracing of select MPI
539 function calls via the MPI profiling interface. Linking it in to
540 your application via (e.g., via -lompitrace) will automatically
541 output to stderr when some MPI functions are invoked:
542
543 shell$ cd examples/
544 shell$ mpicc hello_c.c -o hello_c -lompitrace
545 shell$ mpirun -np 1 hello_c
546 MPI_INIT: argc 1
547 Hello, world, I am 0 of 1
548 MPI_BARRIER[0]: comm MPI_COMM_WORLD
549 MPI_FINALIZE[0]
550 shell$
551
552 Keep in mind that the output from the trace library is going to
553 stderr, so it may output in a slightly different order than the
554 stdout from your application.
555
556 This library is being offered as a "proof of concept" / convenience
557 from Open MPI. If there is interest, it is trivially easy to extend
558 it to printf for other MPI functions. Pull requests on github.com
559 would be greatly appreciated.
560
561OpenSHMEM Functionality and Features
562------------------------------------
563
564- All OpenSHMEM-1.3 functionality is supported.
565
566
567MPI Collectives
568---------------
569
570- The "fca" coll component: the Mellanox Fabric Collective Accelerator
571 (FCA) is a solution for offloading collective operations from the
572 MPI process onto Mellanox QDR InfiniBand switch CPUs and HCAs.
573
574- The "cuda" coll component provides CUDA-aware support for the
575 reduction type collectives with GPU buffers. This component is only
576 compiled into the library when the library has been configured with
577 CUDA-aware support. It intercepts calls to the reduction
578 collectives, copies the data to staging buffers if GPU buffers, then
579 calls underlying collectives to do the work.
580
581OpenSHMEM Collectives
582---------------------
583
584- The "fca" scoll component: the Mellanox Fabric Collective
585 Accelerator (FCA) is a solution for offloading collective operations
586 from the MPI process onto Mellanox QDR InfiniBand switch CPUs and
587 HCAs.
588
589- The "basic" scoll component: Reference implementation of all
590 OpenSHMEM collective operations.
591
592
593Network Support
594---------------
595
596- There are four main MPI network models available: "ob1", "cm",
597 "yalla", and "ucx". "ob1" uses BTL ("Byte Transfer Layer")
598 components for each supported network. "cm" uses MTL ("Matching
599 Tranport Layer") components for each supported network. "yalla"
600 uses the Mellanox MXM transport. "ucx" uses the OpenUCX transport.
601
602 - "ob1" supports a variety of networks that can be used in
603 combination with each other:
604
605 - OpenFabrics: InfiniBand, iWARP, and RoCE
606 - Loopback (send-to-self)
607 - Shared memory
608 - TCP
609 - Intel Phi SCIF
610 - SMCUDA
611 - Cisco usNIC
612 - uGNI (Cray Gemini, Aries)
613 - vader (XPMEM, Linux CMA, Linux KNEM, and copy-in/copy-out shared
614 memory)
615
616 - "cm" supports a smaller number of networks (and they cannot be
617 used together), but may provide better overall MPI performance:
618
619 - Intel Omni-Path PSM2
620 - Intel True Scale PSM (QLogic InfiniPath)
621 - OpenFabrics Interfaces ("libfabric" tag matching)
622 - Portals 4
623
624 Open MPI will, by default, choose to use "cm" when one of the
625 above transports can be used, unless OpenUCX or MXM support is
626 detected, in which case the "ucx" or "yalla" PML will be used
627 by default. Otherwise, "ob1" will be used and the corresponding
628 BTLs will be selected. Users can force the use of ob1 or cm if
629 desired by setting the "pml" MCA parameter at run-time:
630
631 shell$ mpirun --mca pml ob1 ...
632 or
633 shell$ mpirun --mca pml cm ...
634
635- Similarly, there are two OpenSHMEM network models available: "ucx",
636 and "ikrit":
637 - "ucx" interfaces directly with UCX;
638 - "ikrit" interfaces directly with Mellanox MXM.
639
640- UCX is the Unified Communication X (UCX) communication library
641 (http://www.openucx.org/).
642 This is an open-source project developed in collaboration between
643 industry, laboratories, and academia to create an open-source
644 production grade communication framework for data centric and
645 high-performance applications.
646 UCX currently supports:
647 - OFA Verbs;
648 - Cray's uGNI;
649 - NVIDIA CUDA drivers.
650
651- MXM is the Mellanox Messaging Accelerator library utilizing a full
652 range of IB transports to provide the following messaging services
653 to the upper level MPI/OpenSHMEM libraries:
654
655 - Usage of all available IB transports
656 - Native RDMA support
657 - Progress thread
658 - Shared memory communication
659 - Hardware-assisted reliability
660
661- The usnic BTL is support for Cisco's usNIC device ("userspace NIC")
662 on Cisco UCS servers with the Virtualized Interface Card (VIC).
663 Although the usNIC is accessed via the OpenFabrics Libfabric API
664 stack, this BTL is specific to Cisco usNIC devices.
665
666- uGNI is a Cray library for communicating over the Gemini and Aries
667 interconnects.
668
669- The OpenFabrics Enterprise Distribution (OFED) software package v1.0
670 will not work properly with Open MPI v1.2 (and later) due to how its
671 Mellanox InfiniBand plugin driver is created. The problem is fixed
672 OFED v1.1 (and later).
673
674- Better memory management support is available for OFED-based
675 transports using the "ummunotify" Linux kernel module. OFED memory
676 managers are necessary for better bandwidth when re-using the same
677 buffers for large messages (e.g., benchmarks and some applications).
678
679 Unfortunately, the ummunotify module was not accepted by the Linux
680 kernel community (and is still not distributed by OFED). But it
681 still remains the best memory management solution for MPI
682 applications that used the OFED network transports. If Open MPI is
683 able to find the <linux/ummunotify.h> header file, it will build
684 support for ummunotify and include it by default. If MPI processes
685 then find the ummunotify kernel module loaded and active, then their
686 memory managers (which have been shown to be problematic in some
687 cases) will be disabled and ummunotify will be used. Otherwise, the
688 same memory managers from prior versions of Open MPI will be used.
689 The ummunotify Linux kernel module can be downloaded from:
690
691 http://lwn.net/Articles/343351/
692
693- The use of fork() with OpenFabrics-based networks (i.e., the openib
694 BTL) is only partially supported, and only on Linux kernels >=
695 v2.6.15 with libibverbs v1.1 or later (first released as part of
696 OFED v1.2), per restrictions imposed by the OFED network stack.
697
698- Linux "knem" support is used when the "vader" or "sm" (shared
699 memory) BTLs are compiled with knem support (see the --with-knem
700 configure option) and the knem Linux module is loaded in the running
701 kernel. If the knem Linux kernel module is not loaded, the knem
702 support is (by default) silently deactivated during Open MPI jobs.
703
704 See http://runtime.bordeaux.inria.fr/knem/ for details on Knem.
705
706- Linux Cross-Memory Attach (CMA) or XPMEM is used by the vader
707 shared-memory BTL when the CMA/XPMEM libraries are installedm,
708 respectively. Linux CMA and XPMEM are similar (but different)
709 mechanisms for Open MPI to utilize single-copy semantics for shared
710 memory.
711
712Open MPI Extensions
713-------------------
714
715- An MPI "extensions" framework is included in Open MPI, but is not
716 enabled by default. See the "Open MPI API Extensions" section below
717 for more information on compiling and using MPI extensions.
718
719- The following extensions are included in this version of Open MPI:
720
721 - affinity: Provides the OMPI_Affinity_str() routine on retrieving
722 a string that contains what resources a process is bound to. See
723 its man page for more details.
724 - cr: Provides routines to access to checkpoint restart routines.
725 See ompi/mpiext/cr/mpiext_cr_c.h for a listing of available
726 functions.
727 - cuda: When the library is compiled with CUDA-aware support, it
728 provides two things. First, a macro
729 MPIX_CUDA_AWARE_SUPPORT. Secondly, the function
730 MPIX_Query_cuda_support that can be used to query for support.
731 - example: A non-functional extension; its only purpose is to
732 provide an example for how to create other extensions.
733
734===========================================================================
735
736Building Open MPI
737-----------------
738
739If you have checked out a DEVELOPER'S COPY of Open MPI (i.e., you
740cloned from Git), you really need to read the HACKING file before
741attempting to build Open MPI. Really.
742
743If you have downloaded a tarball, then things are much simpler.
744Open MPI uses a traditional configure script paired with "make" to
745build. Typical installs can be of the pattern:
746
747shell$ ./configure [...options...]
748shell$ make [-j N] all install
749 (use an integer value of N for parallel builds)
750
751There are many available configure options (see "./configure --help"
752for a full list); a summary of the more commonly used ones is included
753below.
754
755Note that for many of Open MPI's --with-<foo> options, Open MPI will,
756by default, search for header files and/or libraries for <foo>. If
757the relevant files are found, Open MPI will built support for <foo>;
758if they are not found, Open MPI will skip building support for <foo>.
759However, if you specify --with-<foo> on the configure command line and
760Open MPI is unable to find relevant support for <foo>, configure will
761assume that it was unable to provide a feature that was specifically
762requested and will abort so that a human can resolve out the issue.
763
764Additionally, if a search directory is specified in the form
765--with-<foo>=<dir>, Open MPI will:
766
7671. Search for <foo>'s header files in <dir>/include.
7682. Search for <foo>'s library files:
769 2a. If --with-<foo>-libdir=<libdir> was specified, search in
770 <libdir>.
771 2b. Otherwise, search in <dir>/lib, and if they are not found
772 there, search again in <dir>/lib64.
7733. If both the relevant header files and libraries are found:
774 3a. Open MPI will build support for <foo>.
775 3b. If the root path where the <foo> libraries are found is neither
776 "/usr" nor "/usr/local", Open MPI will compile itself with
777 RPATH flags pointing to the directory where <foo>'s libraries
778 are located. Open MPI does not RPATH /usr/lib[64] and
779 /usr/local/lib[64] because many systems already search these
780 directories for run-time libraries by default; adding RPATH for
781 them could have unintended consequences for the search path
782 ordering.
783
784INSTALLATION OPTIONS
785
786--prefix=<directory>
787 Install Open MPI into the base directory named <directory>. Hence,
788 Open MPI will place its executables in <directory>/bin, its header
789 files in <directory>/include, its libraries in <directory>/lib, etc.
790
791--disable-shared
792 By default, Open MPI and OpenSHMEM build shared libraries, and all
793 components are built as dynamic shared objects (DSOs). This switch
794 disables this default; it is really only useful when used with
795 --enable-static. Specifically, this option does *not* imply
796 --enable-static; enabling static libraries and disabling shared
797 libraries are two independent options.
798
799--enable-static
800 Build MPI and OpenSHMEM as static libraries, and statically link in
801 all components. Note that this option does *not* imply
802 --disable-shared; enabling static libraries and disabling shared
803 libraries are two independent options.
804
805 Be sure to read the description of --without-memory-manager, below;
806 it may have some effect on --enable-static.
807
808--disable-wrapper-rpath
809 By default, the wrapper compilers (e.g., mpicc) will enable "rpath"
810 support in generated executables on systems that support it. That
811 is, they will include a file reference to the location of Open MPI's
812 libraries in the application executable itself. This means that
813 the user does not have to set LD_LIBRARY_PATH to find Open MPI's
814 libraries (e.g., if they are installed in a location that the
815 run-time linker does not search by default).
816
817 On systems that utilize the GNU ld linker, recent enough versions
818 will actually utilize "runpath" functionality, not "rpath". There
819 is an important difference between the two:
820
821 "rpath": the location of the Open MPI libraries is hard-coded into
822 the MPI/OpenSHMEM application and cannot be overridden at
823 run-time.
824 "runpath": the location of the Open MPI libraries is hard-coded into
825 the MPI/OpenSHMEM application, but can be overridden at run-time
826 by setting the LD_LIBRARY_PATH environment variable.
827
828 For example, consider that you install Open MPI vA.B.0 and
829 compile/link your MPI/OpenSHMEM application against it. Later, you
830 install Open MPI vA.B.1 to a different installation prefix (e.g.,
831 /opt/openmpi/A.B.1 vs. /opt/openmpi/A.B.0), and you leave the old
832 installation intact.
833
834 In the rpath case, your MPI application will always use the
835 libraries from your A.B.0 installation. In the runpath case, you
836 can set the LD_LIBRARY_PATH environment variable to point to the
837 A.B.1 installation, and then your MPI application will use those
838 libraries.
839
840 Note that in both cases, however, if you remove the original A.B.0
841 installation and set LD_LIBRARY_PATH to point to the A.B.1
842 installation, your application will use the A.B.1 libraries.
843
844 This rpath/runpath behavior can be disabled via
845 --disable-wrapper-rpath.
846
847 If you would like to keep the rpath option, but not enable runpath
848 a different configure option is avalabile
849 --disable-wrapper-runpath.
850
851--enable-dlopen
852 Build all of Open MPI's components as standalone Dynamic Shared
853 Objects (DSO's) that are loaded at run-time (this is the default).
854 The opposite of this option, --disable-dlopen, causes two things:
855
856 1. All of Open MPI's components will be built as part of Open MPI's
857 normal libraries (e.g., libmpi).
858 2. Open MPI will not attempt to open any DSO's at run-time.
859
860 Note that this option does *not* imply that OMPI's libraries will be
861 built as static objects (e.g., libmpi.a). It only specifies the
862 location of OMPI's components: standalone DSOs or folded into the
863 Open MPI libraries. You can control whether Open MPI's libraries
864 are build as static or dynamic via --enable|disable-static and
865 --enable|disable-shared.
866
867--disable-show-load-errors-by-default
868 Set the default value of the mca_base_component_show_load_errors MCA
869 variable: the --enable form of this option sets the MCA variable to
870 true, the --disable form sets the MCA variable to false. The MCA
871 mca_base_component_show_load_errors variable can still be overridden
872 at run time via the usual MCA-variable-setting mechanisms; this
873 configure option simply sets the default value.
874
875 The --disable form of this option is intended for Open MPI packagers
876 who tend to enable support for many different types of networks and
877 systems in their packages. For example, consider a packager who
878 includes support for both the FOO and BAR networks in their Open MPI
879 package, both of which require support libraries (libFOO.so and
880 libBAR.so). If an end user only has BAR hardware, they likely only
881 have libBAR.so available on their systems -- not libFOO.so.
882 Disabling load errors by default will prevent the user from seeing
883 potentially confusing warnings about the FOO components failing to
884 load because libFOO.so is not available on their systems.
885
886 Conversely, system administrators tend to build an Open MPI that is
887 targeted at their specific environment, and contains few (if any)
888 components that are not needed. In such cases, they might want
889 their users to be warned that the FOO network components failed to
890 load (e.g., if libFOO.so was mistakenly unavailable), because Open
891 MPI may otherwise silently failover to a slower network path for MPI
892 traffic.
893
894--with-platform=FILE
895 Load configure options for the build from FILE. Options on the
896 command line that are not in FILE are also used. Options on the
897 command line and in FILE are replaced by what is in FILE.
898
899--with-libmpi-name=STRING
900 Replace libmpi.* and libmpi_FOO.* (where FOO is one of the fortran
901 supporting libraries installed in lib) with libSTRING.* and
902 libSTRING_FOO.*. This is provided as a convenience mechanism for
903 third-party packagers of Open MPI that might want to rename these
904 libraries for their own purposes. This option is *not* intended for
905 typical users of Open MPI.
906
907--enable-mca-no-build=LIST
908 Comma-separated list of <type>-<component> pairs that will not be
909 built. For example, "--enable-mca-no-build=btl-portals,oob-ud" will
910 disable building the portals BTL and the ud OOB component.
911
912NETWORKING SUPPORT / OPTIONS
913
914--with-fca=<directory>
915 Specify the directory where the Mellanox FCA library and
916 header files are located.
917
918 FCA is the support library for Mellanox switches and HCAs.
919
920--with-hcoll=<directory>
921 Specify the directory where the Mellanox hcoll library and header
922 files are located. This option is generally only necessary if the
923 hcoll headers and libraries are not in default compiler/linker
924 search paths.
925
926 hcoll is the support library for MPI collective operation offload on
927 Mellanox ConnectX-3 HCAs (and later).
928
929--with-knem=<directory>
930 Specify the directory where the knem libraries and header files are
931 located. This option is generally only necessary if the knem headers
932 and libraries are not in default compiler/linker search paths.
933
934 knem is a Linux kernel module that allows direct process-to-process
935 memory copies (optionally using hardware offload), potentially
936 increasing bandwidth for large messages sent between messages on the
937 same server. See http://runtime.bordeaux.inria.fr/knem/ for
938 details.
939
940--with-libfabric=<directory>
941 Specify the directory where the OpenFabrics Interfaces libfabric
942 library and header files are located. This option is generally only
943 necessary if the libfabric headers and libraries are not in default
944 compiler/linker search paths.
945
946 Libfabric is the support library for OpenFabrics Interfaces-based
947 network adapters, such as Cisco usNIC, Intel True Scale PSM, Cray
948 uGNI, etc.
949
950--with-libfabric-libdir=<directory>
951 Look in directory for the libfabric libraries. By default, Open MPI
952 will look in <libfabric directory>/lib and <libfabric
953 directory>/lib64, which covers most cases. This option is only
954 needed for special configurations.
955
956--with-mxm=<directory>
957 Specify the directory where the Mellanox MXM library and header
958 files are located. This option is generally only necessary if the
959 MXM headers and libraries are not in default compiler/linker search
960 paths.
961
962 MXM is the support library for Mellanox Network adapters.
963
964--with-mxm-libdir=<directory>
965 Look in directory for the MXM libraries. By default, Open MPI will
966 look in <mxm directory>/lib and <mxm directory>/lib64, which covers
967 most cases. This option is only needed for special configurations.
968
969--with-portals4=<directory>
970 Specify the directory where the Portals4 libraries and header files
971 are located. This option is generally only necessary if the Portals4
972 headers and libraries are not in default compiler/linker search
973 paths.
974
975 Portals is a low-level network API for high-performance networking
976 on high-performance computing systems developed by Sandia National
977 Laboratories, Intel Corporation, and the University of New Mexico.
978 The Portals 4 Reference Implementation is a complete implementation
979 of Portals 4, with transport over InfiniBand verbs and UDP.
980
981--with-portals4-libdir=<directory>
982 Location of libraries to link with for Portals4 support.
983
984--with-portals4-max-md-size=SIZE
985--with-portals4-max-va-size=SIZE
986 Set configuration values for Portals 4
987
988--with-psm=<directory>
989 Specify the directory where the QLogic InfiniPath / Intel True Scale
990 PSM library and header files are located. This option is generally
991 only necessary if the PSM headers and libraries are not in default
992 compiler/linker search paths.
993
994 PSM is the support library for QLogic InfiniPath and Intel TrueScale
995 network adapters.
996
997--with-psm-libdir=<directory>
998 Look in directory for the PSM libraries. By default, Open MPI will
999 look in <psm directory>/lib and <psm directory>/lib64, which covers
1000 most cases. This option is only needed for special configurations.
1001
1002--with-psm2=<directory>
1003 Specify the directory where the Intel Omni-Path PSM2 library and
1004 header files are located. This option is generally only necessary
1005 if the PSM2 headers and libraries are not in default compiler/linker
1006 search paths.
1007
1008 PSM is the support library for Intel Omni-Path network adapters.
1009
1010--with-psm2-libdir=<directory>
1011 Look in directory for the PSM2 libraries. By default, Open MPI will
1012 look in <psm2 directory>/lib and <psm2 directory>/lib64, which
1013 covers most cases. This option is only needed for special
1014 configurations.
1015
1016--with-scif=<dir>
1017 Look in directory for Intel SCIF support libraries
1018
1019--with-verbs=<directory>
1020 Specify the directory where the verbs (also known as OpenFabrics
1021 verbs, or Linux verbs, and previously known as OpenIB) libraries and
1022 header files are located. This option is generally only necessary
1023 if the verbs headers and libraries are not in default
1024 compiler/linker search paths.
1025
1026 The Verbs library usually implies operating system bypass networks,
1027 such as InfiniBand, usNIC, iWARP, and RoCE (aka "IBoIP").
1028
1029--with-verbs-libdir=<directory>
1030 Look in directory for the verbs libraries. By default, Open MPI
1031 will look in <verbs_directory>/lib and <verbs_ directory>/lib64,
1032 which covers most cases. This option is only needed for special
1033 configurations.
1034
1035--with-verbs-usnic
1036 Note that this option is no longer necessary in recent Linux distro
1037 versions. If your Linux distro uses the "rdma-core" package (instead
1038 of a standalone "libibverbs" package), not only do you not need this
1039 option, you shouldn't use it, either. More below.
1040
1041 This option will activate support in Open MPI for disabling a
1042 dire-sounding warning message from libibverbs that Cisco usNIC
1043 devices are not supported (because Cisco usNIC devices are supported
1044 through libfabric, not libibverbs). This libibverbs warning can
1045 also be suppressed by installing the "no op" libusnic_verbs plugin
1046 for libibverbs (see https://github.com/cisco/libusnic_verbs, or
1047 download binaries from cisco.com).
1048
1049 This option is disabled by default for two reasons:
1050
1051 1. It causes libopen-pal.so to depend on libibverbs.so, which is
1052 undesirable to many downstream packagers.
1053 2. As mentioned above, recent versions of the libibverbs library
1054 (included in the "rdma-core" package) do not have the bug that
1055 will emit dire-sounding warnings about usnic devices. Indeed,
1056 the --with-verbs-usnic option will enable code in Open MPI that
1057 is actually incompatible with rdma-core (i.e., cause Open MPI to
1058 fail to compile).
1059
1060 If you enable --with-verbs-usnic and your system uses the rdma-core
1061 package, configure will safely abort with a helpful message telling
1062 you that you should not use --with-verbs-usnic.
1063
1064--with-usnic
1065 Abort configure if Cisco usNIC support cannot be built.
1066
1067RUN-TIME SYSTEM SUPPORT
1068
1069--enable-mpirun-prefix-by-default
1070 This option forces the "mpirun" command to always behave as if
1071 "--prefix $prefix" was present on the command line (where $prefix is
1072 the value given to the --prefix option to configure). This prevents
1073 most rsh/ssh-based users from needing to modify their shell startup
1074 files to set the PATH and/or LD_LIBRARY_PATH for Open MPI on remote
1075 nodes. Note, however, that such users may still desire to set PATH
1076 -- perhaps even in their shell startup files -- so that executables
1077 such as mpicc and mpirun can be found without needing to type long
1078 path names. --enable-orterun-prefix-by-default is a synonym for
1079 this option.
1080
1081--enable-orte-static-ports
1082 Enable orte static ports for tcp oob (default: enabled).
1083
1084--with-alps
1085 Force the building of for the Cray Alps run-time environment. If
1086 Alps support cannot be found, configure will abort.
1087
1088--with-lsf=<directory>
1089 Specify the directory where the LSF libraries and header files are
1090 located. This option is generally only necessary if the LSF headers
1091 and libraries are not in default compiler/linker search paths.
1092
1093 LSF is a resource manager system, frequently used as a batch
1094 scheduler in HPC systems.
1095
1096 NOTE: If you are using LSF version 7.0.5, you will need to add
1097 "LIBS=-ldl" to the configure command line. For example:
1098
1099 ./configure LIBS=-ldl --with-lsf ...
1100
1101 This workaround should *only* be needed for LSF 7.0.5.
1102
1103--with-lsf-libdir=<directory>
1104 Look in directory for the LSF libraries. By default, Open MPI will
1105 look in <lsf directory>/lib and <lsf directory>/lib64, which covers
1106 most cases. This option is only needed for special configurations.
1107
1108--with-pmi
1109 Build PMI support (by default on non-Cray XE/XC systems, it is not
1110 built). On Cray XE/XC systems, the location of pmi is detected
1111 automatically as part of the configure process. For non-Cray
1112 systems, if the pmi2.h header is found in addition to pmi.h, then
1113 support for PMI2 will be built.
1114
1115--with-slurm
1116 Force the building of SLURM scheduler support.
1117
1118--with-sge
1119 Specify to build support for the Oracle Grid Engine (OGE) resource
1120 manager and/or the Open Grid Engine. OGE support is disabled by
1121 default; this option must be specified to build OMPI's OGE support.
1122
1123 The Oracle Grid Engine (OGE) and open Grid Engine packages are
1124 resource manager systems, frequently used as a batch scheduler in
1125 HPC systems.
1126
1127--with-tm=<directory>
1128 Specify the directory where the TM libraries and header files are
1129 located. This option is generally only necessary if the TM headers
1130 and libraries are not in default compiler/linker search paths.
1131
1132 TM is the support library for the Torque and PBS Pro resource
1133 manager systems, both of which are frequently used as a batch
1134 scheduler in HPC systems.
1135
1136MISCELLANEOUS SUPPORT LIBRARIES
1137
1138--with-blcr=<directory>
1139 Specify the directory where the Berkeley Labs Checkpoint / Restart
1140 (BLCR) libraries and header files are located. This option is
1141 generally only necessary if the BLCR headers and libraries are not
1142 in default compiler/linker search paths.
1143
1144 This option is only meaningful if the --with-ft option is also used
1145 to active Open MPI's fault tolerance behavior.
1146
1147--with-blcr-libdir=<directory>
1148 Look in directory for the BLCR libraries. By default, Open MPI will
1149 look in <blcr directory>/lib and <blcr directory>/lib64, which
1150 covers most cases. This option is only needed for special
1151 configurations.
1152
1153--with-dmtcp=<directory>
1154 Specify the directory where the Distributed MultiThreaded
1155 Checkpointing (DMTCP) libraries and header files are located. This
1156 option is generally only necessary if the DMTCP headers and
1157 libraries are not in default compiler/linker search paths.
1158
1159 This option is only meaningful if the --with-ft option is also used
1160 to active Open MPI's fault tolerance behavior.
1161
1162--with-dmtcp-libdir=<directory>
1163 Look in directory for the DMTCP libraries. By default, Open MPI
1164 will look in <dmtcp directory>/lib and <dmtcp directory>/lib64,
1165 which covers most cases. This option is only needed for special
1166 configurations.
1167
1168--with-libevent(=value)
1169 This option specifies where to find the libevent support headers and
1170 library. The following VALUEs are permitted:
1171
1172 internal: Use Open MPI's internal copy of libevent.
1173 external: Use an external libevent installation (rely on default
1174 compiler and linker paths to find it)
1175 <no value>: Same as "internal".
1176 <directory>: Specify the location of a specific libevent
1177 installation to use
1178
1179 By default (or if --with-libevent is specified with no VALUE), Open
1180 MPI will build and use the copy of libevent that it has in its
1181 source tree. However, if the VALUE is "external", Open MPI will
1182 look for the relevant libevent header file and library in default
1183 compiler / linker locations. Or, VALUE can be a directory tree
1184 where the libevent header file and library can be found. This
1185 option allows operating systems to include Open MPI and use their
1186 default libevent installation instead of Open MPI's bundled libevent.
1187
1188 libevent is a support library that provides event-based processing,
1189 timers, and signal handlers. Open MPI requires libevent to build;
1190 passing --without-libevent will cause configure to abort.
1191
1192--with-libevent-libdir=<directory>
1193 Look in directory for the libevent libraries. This option is only
1194 usable when building Open MPI against an external libevent
1195 installation. Just like other --with-FOO-libdir configure options,
1196 this option is only needed for special configurations.
1197
1198--with-hwloc(=value)
1199 Build hwloc support (default: enabled). This option specifies where
1200 to find the hwloc support headers and library. The following values
1201 are permitted:
1202
1203 internal: Use Open MPI's internal copy of hwloc.
1204 external: Use an external hwloc installation (rely on default
1205 compiler and linker paths to find it)
1206 <no value>: Same as "internal".
1207 <directory>: Specify the location of a specific hwloc
1208 installation to use
1209
1210 By default (or if --with-hwloc is specified with no VALUE), Open MPI
1211 will build and use the copy of hwloc that it has in its source tree.
1212 However, if the VALUE is "external", Open MPI will look for the
1213 relevant hwloc header files and library in default compiler / linker
1214 locations. Or, VALUE can be a directory tree where the hwloc header
1215 file and library can be found. This option allows operating systems
1216 to include Open MPI and use their default hwloc installation instead
1217 of Open MPI's bundled hwloc.
1218
1219 hwloc is a support library that provides processor and memory
1220 affinity information for NUMA platforms.
1221
1222--with-hwloc-libdir=<directory>
1223 Look in directory for the hwloc libraries. This option is only
1224 usable when building Open MPI against an external hwloc
1225 installation. Just like other --with-FOO-libdir configure options,
1226 this option is only needed for special configurations.
1227
1228--disable-hwloc-pci
1229 Disable building hwloc's PCI device-sensing capabilities. On some
1230 platforms (e.g., SusE 10 SP1, x86-64), the libpci support library is
1231 broken. Open MPI's configure script should usually detect when
1232 libpci is not usable due to such brokenness and turn off PCI
1233 support, but there may be cases when configure mistakenly enables
1234 PCI support in the presence of a broken libpci. These cases may
1235 result in "make" failing with warnings about relocation symbols in
1236 libpci. The --disable-hwloc-pci switch can be used to force Open
1237 MPI to not build hwloc's PCI device-sensing capabilities in these
1238 cases.
1239
1240 Similarly, if Open MPI incorrectly decides that libpci is broken,
1241 you can force Open MPI to build hwloc's PCI device-sensing
1242 capabilities by using --enable-hwloc-pci.
1243
1244 hwloc can discover PCI devices and locality, which can be useful for
1245 Open MPI in assigning message passing resources to MPI processes.
1246
1247--with-libltdl=<directory>
1248 Specify the directory where the GNU Libtool libltdl libraries and
1249 header files are located. This option is generally only necessary
1250 if the libltdl headers and libraries are not in default
1251 compiler/linker search paths.
1252
1253 Note that this option is ignored if --disable-dlopen is specified.
1254
1255--disable-libompitrace
1256 Disable building the simple "libompitrace" library (see note above
1257 about libompitrace)
1258
1259--with-valgrind(=<directory>)
1260 Directory where the valgrind software is installed. If Open MPI
1261 finds Valgrind's header files, it will include additional support
1262 for Valgrind's memory-checking debugger.
1263
1264 Specifically, it will eliminate a lot of false positives from
1265 running Valgrind on MPI applications. There is a minor performance
1266 penalty for enabling this option.
1267
1268MPI FUNCTIONALITY
1269
1270--with-mpi-param-check(=value)
1271 Whether or not to check MPI function parameters for errors at
1272 runtime. The following values are permitted:
1273
1274 always: MPI function parameters are always checked for errors
1275 never: MPI function parameters are never checked for errors
1276 runtime: Whether MPI function parameters are checked depends on
1277 the value of the MCA parameter mpi_param_check (default:
1278 yes).
1279 yes: Synonym for "always" (same as --with-mpi-param-check).
1280 no: Synonym for "none" (same as --without-mpi-param-check).
1281
1282 If --with-mpi-param is not specified, "runtime" is the default.
1283
1284--disable-mpi-thread-multiple
1285 Disable the MPI thread level MPI_THREAD_MULTIPLE (it is enabled by
1286 default).
1287
1288--enable-mpi-cxx
1289 Enable building the C++ MPI bindings (default: disabled).
1290
1291 The MPI C++ bindings were deprecated in MPI-2.2, and removed from
1292 the MPI standard in MPI-3.0.
1293
1294--enable-mpi-java
1295 Enable building of an EXPERIMENTAL Java MPI interface (disabled by
1296 default). You may also need to specify --with-jdk-dir,
1297 --with-jdk-bindir, and/or --with-jdk-headers. See README.JAVA.txt
1298 for details.
1299
1300 Note that this Java interface is INCOMPLETE (meaning: it does not
1301 support all MPI functionality) and LIKELY TO CHANGE. The Open MPI
1302 developers would very much like to hear your feedback about this
1303 interface. See README.JAVA.txt for more details.
1304
1305--enable-mpi-fortran(=value)
1306 By default, Open MPI will attempt to build all 3 Fortran bindings:
1307 mpif.h, the "mpi" module, and the "mpi_f08" module. The following
1308 values are permitted:
1309
1310 all: Synonym for "yes".
1311 yes: Attempt to build all 3 Fortran bindings; skip
1312 any binding that cannot be built (same as
1313 --enable-mpi-fortran).
1314 mpifh: Build mpif.h support.
1315 usempi: Build mpif.h and "mpi" module support.
1316 usempif08: Build mpif.h, "mpi" module, and "mpi_f08"
1317 module support.
1318 none: Synonym for "no".
1319 no: Do not build any MPI Fortran support (same as
1320 --disable-mpi-fortran). This is mutually exclusive
1321 with building the OpenSHMEM Fortran interface.
1322
1323--enable-mpi-ext(=<list>)
1324 Enable Open MPI's non-portable API extensions. If no <list> is
1325 specified, all of the extensions are enabled.
1326
1327 See "Open MPI API Extensions", below, for more details.
1328
1329--disable-mpi-io
1330 Disable built-in support for MPI-2 I/O, likely because an
1331 externally-provided MPI I/O package will be used. Default is to use
1332 the internal framework system that uses the ompio component and a
1333 specially modified version of ROMIO that fits inside the romio
1334 component
1335
1336--disable-io-romio
1337 Disable the ROMIO MPI-IO component
1338
1339--with-io-romio-flags=flags
1340 Pass flags to the ROMIO distribution configuration script. This
1341 option is usually only necessary to pass
1342 parallel-filesystem-specific preprocessor/compiler/linker flags back
1343 to the ROMIO system.
1344
1345--disable-io-ompio
1346 Disable the ompio MPI-IO component
1347
1348--enable-sparse-groups
1349 Enable the usage of sparse groups. This would save memory
1350 significantly especially if you are creating large
1351 communicators. (Disabled by default)
1352
1353OPENSHMEM FUNCTIONALITY
1354
1355--disable-oshmem
1356 Disable building the OpenSHMEM implementation (by default, it is
1357 enabled).
1358
1359--disable-oshmem-fortran
1360 Disable building only the Fortran OpenSHMEM bindings. Please see
1361 the "Compiler Notes" section herein which contains further
1362 details on known issues with various Fortran compilers.
1363
1364MISCELLANEOUS FUNCTIONALITY
1365
1366--without-memory-manager
1367 Disable building Open MPI's memory manager. Open MPI's memory
1368 manager is usually built on Linux based platforms, and is generally
1369 only used for optimizations with some OpenFabrics-based networks (it
1370 is not *necessary* for OpenFabrics networks, but some performance
1371 loss may be observed without it).
1372
1373 However, it may be necessary to disable the memory manager in order
1374 to build Open MPI statically.
1375
1376--with-ft=TYPE
1377 Specify the type of fault tolerance to enable. Options: LAM
1378 (LAM/MPI-like), cr (Checkpoint/Restart). Fault tolerance support is
1379 disabled unless this option is specified.
1380
1381--enable-peruse
1382 Enable the PERUSE MPI data analysis interface.
1383
1384--enable-heterogeneous
1385 Enable support for running on heterogeneous clusters (e.g., machines
1386 with different endian representations). Heterogeneous support is
1387 disabled by default because it imposes a minor performance penalty.
1388
1389 *** THIS FUNCTIONALITY IS CURRENTLY BROKEN - DO NOT USE ***
1390
1391--with-wrapper-cflags=<cflags>
1392--with-wrapper-cxxflags=<cxxflags>
1393--with-wrapper-fflags=<fflags>
1394--with-wrapper-fcflags=<fcflags>
1395--with-wrapper-ldflags=<ldflags>
1396--with-wrapper-libs=<libs>
1397 Add the specified flags to the default flags that used are in Open
1398 MPI's "wrapper" compilers (e.g., mpicc -- see below for more
1399 information about Open MPI's wrapper compilers). By default, Open
1400 MPI's wrapper compilers use the same compilers used to build Open
1401 MPI and specify a minimum set of additional flags that are necessary
1402 to compile/link MPI applications. These configure options give
1403 system administrators the ability to embed additional flags in
1404 OMPI's wrapper compilers (which is a local policy decision). The
1405 meanings of the different flags are:
1406
1407 <cflags>: Flags passed by the mpicc wrapper to the C compiler
1408 <cxxflags>: Flags passed by the mpic++ wrapper to the C++ compiler
1409 <fcflags>: Flags passed by the mpifort wrapper to the Fortran compiler
1410 <ldflags>: Flags passed by all the wrappers to the linker
1411 <libs>: Flags passed by all the wrappers to the linker
1412
1413 There are other ways to configure Open MPI's wrapper compiler
1414 behavior; see the Open MPI FAQ for more information.
1415
1416There are many other options available -- see "./configure --help".
1417
1418Changing the compilers that Open MPI uses to build itself uses the
1419standard Autoconf mechanism of setting special environment variables
1420either before invoking configure or on the configure command line.
1421The following environment variables are recognized by configure:
1422
1423CC - C compiler to use
1424CFLAGS - Compile flags to pass to the C compiler
1425CPPFLAGS - Preprocessor flags to pass to the C compiler
1426
1427CXX - C++ compiler to use
1428CXXFLAGS - Compile flags to pass to the C++ compiler
1429CXXCPPFLAGS - Preprocessor flags to pass to the C++ compiler
1430
1431FC - Fortran compiler to use
1432FCFLAGS - Compile flags to pass to the Fortran compiler
1433
1434LDFLAGS - Linker flags to pass to all compilers
1435LIBS - Libraries to pass to all compilers (it is rarely
1436 necessary for users to need to specify additional LIBS)
1437
1438PKG_CONFIG - Path to the pkg-config utility
1439
1440For example:
1441
1442 shell$ ./configure CC=mycc CXX=myc++ FC=myfortran ...
1443
1444*** NOTE: We generally suggest using the above command line form for
1445 setting different compilers (vs. setting environment variables and
1446 then invoking "./configure"). The above form will save all
1447 variables and values in the config.log file, which makes
1448 post-mortem analysis easier if problems occur.
1449
1450Note that if you intend to compile Open MPI with a "make" other than
1451the default one in your PATH, then you must either set the $MAKE
1452environment variable before invoking Open MPI's configure script, or
1453pass "MAKE=your_make_prog" to configure. For example:
1454
1455 shell$ ./configure MAKE=/path/to/my/make ...
1456
1457This could be the case, for instance, if you have a shell alias for
1458"make", or you always type "gmake" out of habit. Failure to tell
1459configure which non-default "make" you will use to compile Open MPI
1460can result in undefined behavior (meaning: don't do that).
1461
1462Note that you may also want to ensure that the value of
1463LD_LIBRARY_PATH is set appropriately (or not at all) for your build
1464(or whatever environment variable is relevant for your operating
1465system). For example, some users have been tripped up by setting to
1466use a non-default Fortran compiler via FC, but then failing to set
1467LD_LIBRARY_PATH to include the directory containing that non-default
1468Fortran compiler's support libraries. This causes Open MPI's
1469configure script to fail when it tries to compile / link / run simple
1470Fortran programs.
1471
1472It is required that the compilers specified be compile and link
1473compatible, meaning that object files created by one compiler must be
1474able to be linked with object files from the other compilers and
1475produce correctly functioning executables.
1476
1477Open MPI supports all the "make" targets that are provided by GNU
1478Automake, such as:
1479
1480all - build the entire Open MPI package
1481install - install Open MPI
1482uninstall - remove all traces of Open MPI from the $prefix
1483clean - clean out the build tree
1484
1485Once Open MPI has been built and installed, it is safe to run "make
1486clean" and/or remove the entire build tree.
1487
1488VPATH and parallel builds are fully supported.
1489
1490Generally speaking, the only thing that users need to do to use Open
1491MPI is ensure that <prefix>/bin is in their PATH and <prefix>/lib is
1492in their LD_LIBRARY_PATH. Users may need to ensure to set the PATH
1493and LD_LIBRARY_PATH in their shell setup files (e.g., .bashrc, .cshrc)
1494so that non-interactive rsh/ssh-based logins will be able to find the
1495Open MPI executables.
1496
1497===========================================================================
1498
1499Open MPI Version Numbers and Binary Compatibility
1500-------------------------------------------------
1501
1502Open MPI has two sets of version numbers that are likely of interest
1503to end users / system administrator:
1504
1505 * Software version number
1506 * Shared library version numbers
1507
1508Both are predicated on Open MPI's definition of "backwards
1509compatibility."
1510
1511NOTE: The version numbering conventions were changed with the release
1512 of v1.10.0. Most notably, Open MPI no longer uses an "odd/even"
1513 release schedule to indicate feature development vs. stable
1514 releases. See the README in releases prior to v1.10.0 for more
1515 information (e.g.,
1516 https://github.com/open-mpi/ompi/blob/v1.8/README#L1392-L1475).
1517
1518Backwards Compatibility
1519-----------------------
1520
1521Open MPI version Y is backwards compatible with Open MPI version X
1522(where Y>X) if users can:
1523
1524 * Compile an MPI/OpenSHMEM application with version X, mpirun/oshrun
1525 it with version Y, and get the same user-observable behavior.
1526 * Invoke ompi_info with the same CLI options in versions X and Y and
1527 get the same user-observable behavior.
1528
1529Note that this definition encompasses several things:
1530
1531 * Application Binary Interface (ABI)
1532 * MPI / OpenSHMEM run time system
1533 * mpirun / oshrun command line options
1534 * MCA parameter names / values / meanings
1535
1536However, this definition only applies when the same version of Open
1537MPI is used with all instances of the runtime and MPI / OpenSHMEM
1538processes in a single MPI job. If the versions are not exactly the
1539same everywhere, Open MPI is not guaranteed to work properly in any
1540scenario.
1541
1542Backwards compatibility tends to work best when user applications are
1543dynamically linked to one version of the Open MPI / OSHMEM libraries,
1544and can be updated at run time to link to a new version of the Open
1545MPI / OSHMEM libraries.
1546
1547For example, if an MPI / OSHMEM application links statically against
1548the libraries from Open MPI vX, then attempting to launch that
1549application with mpirun / oshrun from Open MPI vY is not guaranteed to
1550work (because it is mixing vX and vY of Open MPI in a single job).
1551
1552Similarly, if using a container technology that internally bundles all
1553the libraries from Open MPI vX, attempting to launch that container
1554with mpirun / oshrun from Open MPI vY is not guaranteed to work.
1555
1556Software Version Number
1557-----------------------
1558
1559Official Open MPI releases use the common "A.B.C" version identifier
1560format. Each of the three numbers has a specific meaning:
1561
1562 * Major: The major number is the first integer in the version string
1563 Changes in the major number typically indicate a significant
1564 change in the code base and/or end-user functionality, and also
1565 indicate a break from backwards compatibility. Specifically: Open
1566 MPI releases with different major version numbers are not
1567 backwards compatibale with each other.
1568
1569 CAVEAT: This rule does not extend to versions prior to v1.10.0.
1570 Specifically: v1.10.x is not guaranteed to be backwards
1571 compatible with other v1.x releases.
1572
1573 * Minor: The minor number is the second integer in the version
1574 string. Changes in the minor number indicate a user-observable
1575 change in the code base and/or end-user functionality. Backwards
1576 compatibility will still be preserved with prior releases that
1577 have the same major version number (e.g., v2.5.3 is backwards
1578 compatible with v2.3.1).
1579
1580 * Release: The release number is the third integer in the version
1581 string. Changes in the release number typically indicate a bug
1582 fix in the code base and/or end-user functionality. For example,
1583 if there is a release that only contains bug fixes and no other
1584 user-observable changes or new features, only the third integer
1585 will be increased (e.g., from v4.3.0 to v4.3.1).
1586
1587The "A.B.C" version number may optionally be followed by a Quantifier:
1588
1589 * Quantifier: Open MPI version numbers sometimes have an arbitrary
1590 string affixed to the end of the version number. Common strings
1591 include:
1592
1593 o aX: Indicates an alpha release. X is an integer indicating the
1594 number of the alpha release (e.g., v1.10.3a5 indicates the 5th
1595 alpha release of version 1.10.3).
1596 o bX: Indicates a beta release. X is an integer indicating the
1597 number of the beta release (e.g., v1.10.3b3 indicates the 3rd
1598 beta release of version 1.10.3).
1599 o rcX: Indicates a release candidate. X is an integer indicating
1600 the number of the release candidate (e.g., v1.10.3rc4 indicates
1601 the 4th release candidate of version 1.10.3).
1602
1603Nightly development snapshot tarballs use a different version number
1604scheme; they contain three distinct values:
1605
1606 * The git branch name from which the tarball was created.
1607 * The date/timestamp, in YYYYMMDDHHMM format.
1608 * The hash of the git commit from which the tarball was created.
1609
1610For example, a snapshot tarball filename of
1611"openmpi-v2.x-201703070235-e4798fb.tar.gz" indicates that this tarball
1612was created from the v2.x branch, on March 7, 2017, at 2:35am GMT,
1613from git hash e4798fb.
1614
1615Shared Library Version Number
1616-----------------------------
1617
1618The GNU Libtool official documentation details how the versioning
1619scheme works. The quick version is that the shared library versions
1620are a triple of integers: (current,revision,age), or "c:r:a". This
1621triple is not related to the Open MPI software version number. There
1622are six simple rules for updating the values (taken almost verbatim
1623from the Libtool docs):
1624
1625 1. Start with version information of "0:0:0" for each shared library.
1626
1627 2. Update the version information only immediately before a public
1628 release of your software. More frequent updates are unnecessary,
1629 and only guarantee that the current interface number gets larger
1630 faster.
1631
1632 3. If the library source code has changed at all since the last
1633 update, then increment revision ("c:r:a" becomes "c:r+1:a").
1634
1635 4. If any interfaces have been added, removed, or changed since the
1636 last update, increment current, and set revision to 0.
1637
1638 5. If any interfaces have been added since the last public release,
1639 then increment age.
1640
1641 6. If any interfaces have been removed since the last public release,
1642 then set age to 0.
1643
1644Here's how we apply those rules specifically to Open MPI:
1645
1646 1. The above rules do not apply to MCA components (a.k.a. "plugins");
1647 MCA component .so versions stay unspecified.
1648
1649 2. The above rules apply exactly as written to the following
1650 libraries starting with Open MPI version v1.5 (prior to v1.5,
1651 libopen-pal and libopen-rte were still at 0:0:0 for reasons
1652 discussed in bug ticket #2092
1653 https://svn.open-mpi.org/trac/ompi/ticket/2092):
1654
1655 * libopen-rte
1656 * libopen-pal
1657 * libmca_common_*
1658
1659 3. The following libraries use a slightly modified version of the
1660 above rules: rules 4, 5, and 6 only apply to the official MPI and
1661 OpenSHMEM interfaces (functions, global variables). The rationale
1662 for this decision is that the vast majority of our users only care
1663 about the official/public MPI/OpenSHMEM interfaces; we therefore
1664 want the .so version number to reflect only changes to the
1665 official MPI/OpenSHMEM APIs. Put simply: non-MPI/OpenSHMEM API /
1666 internal changes to the MPI-application-facing libraries are
1667 irrelevant to pure MPI/OpenSHMEM applications.
1668
1669 * libmpi
1670 * libmpi_mpifh
1671 * libmpi_usempi_tkr
1672 * libmpi_usempi_ignore_tkr
1673 * libmpi_usempif08
1674 * libmpi_cxx
1675 * libmpi_java
1676 * liboshmem
1677
1678===========================================================================
1679
1680Checking Your Open MPI Installation
1681-----------------------------------
1682
1683The "ompi_info" command can be used to check the status of your Open
1684MPI installation (located in <prefix>/bin/ompi_info). Running it with
1685no arguments provides a summary of information about your Open MPI
1686installation.
1687
1688Note that the ompi_info command is extremely helpful in determining
1689which components are installed as well as listing all the run-time
1690settable parameters that are available in each component (as well as
1691their default values).
1692
1693The following options may be helpful:
1694
1695--all Show a *lot* of information about your Open MPI
1696 installation.
1697--parsable Display all the information in an easily
1698 grep/cut/awk/sed-able format.
1699--param <framework> <component>
1700 A <framework> of "all" and a <component> of "all" will
1701 show all parameters to all components. Otherwise, the
1702 parameters of all the components in a specific framework,
1703 or just the parameters of a specific component can be
1704 displayed by using an appropriate <framework> and/or
1705 <component> name.
1706--level <level>
1707 By default, ompi_info only shows "Level 1" MCA parameters
1708 -- parameters that can affect whether MPI processes can
1709 run successfully or not (e.g., determining which network
1710 interfaces to use). The --level option will display all
1711 MCA parameters from level 1 to <level> (the max <level>
1712 value is 9). Use "ompi_info --param <framework>
1713 <component> --level 9" to see *all* MCA parameters for a
1714 given component. See "The Modular Component Architecture
1715 (MCA)" section, below, for a fuller explanation.
1716
1717Changing the values of these parameters is explained in the "The
1718Modular Component Architecture (MCA)" section, below.
1719
1720When verifying a new Open MPI installation, we recommend running six
1721tests:
1722
17231. Use "mpirun" to launch a non-MPI program (e.g., hostname or uptime)
1724 across multiple nodes.
1725
17262. Use "mpirun" to launch a trivial MPI program that does no MPI
1727 communication (e.g., the hello_c program in the examples/ directory
1728 in the Open MPI distribution).
1729
17303. Use "mpirun" to launch a trivial MPI program that sends and
1731 receives a few MPI messages (e.g., the ring_c program in the
1732 examples/ directory in the Open MPI distribution).
1733
17344. Use "oshrun" to launch a non-OpenSHMEM program across multiple
1735 nodes.
1736
17375. Use "oshrun" to launch a trivial MPI program that does no OpenSHMEM
1738 communication (e.g., hello_shmem.c program in the examples/
1739 directory in the Open MPI distribution.)
1740
17416. Use "oshrun" to launch a trivial OpenSHMEM program that puts and
1742 gets a few messages. (e.g., the ring_shmem.c in the examples/
1743 directory in the Open MPI distribution.)
1744
1745If you can run all six of these tests successfully, that is a good
1746indication that Open MPI built and installed properly.
1747
1748===========================================================================
1749
1750Open MPI API Extensions
1751-----------------------
1752
1753Open MPI contains a framework for extending the MPI API that is
1754available to applications. Each extension is usually a standalone set
1755of functionality that is distinct from other extensions (similar to
1756how Open MPI's plugins are usually unrelated to each other). These
1757extensions provide new functions and/or constants that are available
1758to MPI applications.
1759
1760WARNING: These extensions are neither standard nor portable to other
1761MPI implementations!
1762
1763Compiling the extensions
1764------------------------
1765
1766Open MPI extensions are all enabled by default; they can be disabled
1767via the --disable-mpi-ext command line switch.
1768
1769Since extensions are meant to be used by advanced users only, this
1770file does not document which extensions are available or what they
1771do. Look in the ompi/mpiext/ directory to see the extensions; each
1772subdirectory of that directory contains an extension. Each has a
1773README file that describes what it does.
1774
1775Using the extensions
1776--------------------
1777
1778To reinforce the fact that these extensions are non-standard, you must
1779include a separate header file after <mpi.h> to obtain the function
1780prototypes, constant declarations, etc. For example:
1781
1782-----
1783#include <mpi.h>
1784#if defined(OPEN_MPI) && OPEN_MPI
1785#include <mpi-ext.h>
1786#endif
1787
1788int main() {
1789 MPI_Init(NULL, NULL);
1790
1791#if defined(OPEN_MPI) && OPEN_MPI
1792 {
1793 char ompi_bound[OMPI_AFFINITY_STRING_MAX];
1794 char current_binding[OMPI_AFFINITY_STRING_MAX];
1795 char exists[OMPI_AFFINITY_STRING_MAX];
1796 OMPI_Affinity_str(OMPI_AFFINITY_LAYOUT_FMT, ompi_bound,
1797 current_bindings, exists);
1798 }
1799#endif
1800 MPI_Finalize();
1801 return 0;
1802}
1803-----
1804
1805Notice that the Open MPI-specific code is surrounded by the #if
1806statement to ensure that it is only ever compiled by Open MPI.
1807
1808The Open MPI wrapper compilers (mpicc and friends) should
1809automatically insert all relevant compiler and linker flags necessary
1810to use the extensions. No special flags or steps should be necessary
1811compared to "normal" MPI applications.
1812
1813===========================================================================
1814
1815Compiling Open MPI Applications
1816-------------------------------
1817
1818Open MPI provides "wrapper" compilers that should be used for
1819compiling MPI and OpenSHMEM applications:
1820
1821C: mpicc, oshcc
1822C++: mpiCC, oshCC (or mpic++ if your filesystem is case-insensitive)
1823Fortran: mpifort, oshfort
1824
1825For example:
1826
1827 shell$ mpicc hello_world_mpi.c -o hello_world_mpi -g
1828 shell$
1829
1830For OpenSHMEM applications:
1831
1832 shell$ oshcc hello_shmem.c -o hello_shmem -g
1833 shell$
1834
1835All the wrapper compilers do is add a variety of compiler and linker
1836flags to the command line and then invoke a back-end compiler. To be
1837specific: the wrapper compilers do not parse source code at all; they
1838are solely command-line manipulators, and have nothing to do with the
1839actual compilation or linking of programs. The end result is an MPI
1840executable that is properly linked to all the relevant libraries.
1841
1842Customizing the behavior of the wrapper compilers is possible (e.g.,
1843changing the compiler [not recommended] or specifying additional
1844compiler/linker flags); see the Open MPI FAQ for more information.
1845
1846Alternatively, Open MPI also installs pkg-config(1) configuration
1847files under $libdir/pkgconfig. If pkg-config is configured to find
1848these files, then compiling / linking Open MPI programs can be
1849performed like this:
1850
1851 shell$ gcc hello_world_mpi.c -o hello_world_mpi -g \
1852 `pkg-config ompi-c --cflags --libs`
1853 shell$
1854
1855Open MPI supplies multiple pkg-config(1) configuration files; one for
1856each different wrapper compiler (language):
1857
1858------------------------------------------------------------------------
1859ompi Synonym for "ompi-c"; Open MPI applications using the C
1860 MPI bindings
1861ompi-c Open MPI applications using the C MPI bindings
1862ompi-cxx Open MPI applications using the C or C++ MPI bindings
1863ompi-fort Open MPI applications using the Fortran MPI bindings
1864------------------------------------------------------------------------
1865
1866The following pkg-config(1) configuration files *may* be installed,
1867depending on which command line options were specified to Open MPI's
1868configure script. They are not necessary for MPI applications, but
1869may be used by applications that use Open MPI's lower layer support
1870libraries.
1871
1872orte: Open MPI Run-Time Environment applications
1873opal: Open Portable Access Layer applications
1874
1875===========================================================================
1876
1877Running Open MPI Applications
1878-----------------------------
1879
1880Open MPI supports both mpirun and mpiexec (they are exactly
1881equivalent) to launch MPI applications. For example:
1882
1883 shell$ mpirun -np 2 hello_world_mpi
1884 or
1885 shell$ mpiexec -np 1 hello_world_mpi : -np 1 hello_world_mpi
1886
1887are equivalent. Some of mpiexec's switches (such as -host and -arch)
1888are not yet functional, although they will not error if you try to use
1889them.
1890
1891The rsh launcher (which defaults to using ssh) accepts a -hostfile
1892parameter (the option "-machinefile" is equivalent); you can specify a
1893-hostfile parameter indicating an standard mpirun-style hostfile (one
1894hostname per line):
1895
1896 shell$ mpirun -hostfile my_hostfile -np 2 hello_world_mpi
1897
1898If you intend to run more than one process on a node, the hostfile can
1899use the "slots" attribute. If "slots" is not specified, a count of 1
1900is assumed. For example, using the following hostfile:
1901
1902---------------------------------------------------------------------------
1903node1.example.com
1904node2.example.com
1905node3.example.com slots=2
1906node4.example.com slots=4
1907---------------------------------------------------------------------------
1908
1909 shell$ mpirun -hostfile my_hostfile -np 8 hello_world_mpi
1910
1911will launch MPI_COMM_WORLD rank 0 on node1, rank 1 on node2, ranks 2
1912and 3 on node3, and ranks 4 through 7 on node4.
1913
1914Other starters, such as the resource manager / batch scheduling
1915environments, do not require hostfiles (and will ignore the hostfile
1916if it is supplied). They will also launch as many processes as slots
1917have been allocated by the scheduler if no "-np" argument has been
1918provided. For example, running a SLURM job with 8 processors:
1919
1920 shell$ salloc -n 8 mpirun a.out
1921
1922The above command will reserve 8 processors and run 1 copy of mpirun,
1923which will, in turn, launch 8 copies of a.out in a single
1924MPI_COMM_WORLD on the processors that were allocated by SLURM.
1925
1926Note that the values of component parameters can be changed on the
1927mpirun / mpiexec command line. This is explained in the section
1928below, "The Modular Component Architecture (MCA)".
1929
1930Open MPI supports oshrun to launch OpenSHMEM applications. For
1931example:
1932
1933 shell$ oshrun -np 2 hello_world_oshmem
1934
1935OpenSHMEM applications may also be launched directly by resource
1936managers such as SLURM. For example, when OMPI is configured
1937--with-pmi and --with-slurm, one may launch OpenSHMEM applications via
1938srun:
1939
1940 shell$ srun -N 2 hello_world_oshmem
1941
1942===========================================================================
1943
1944The Modular Component Architecture (MCA)
1945
1946The MCA is the backbone of Open MPI -- most services and functionality
1947are implemented through MCA components. Here is a list of all the
1948component frameworks in Open MPI:
1949
1950---------------------------------------------------------------------------
1951
1952MPI component frameworks:
1953-------------------------
1954
1955bml - BTL management layer
1956coll - MPI collective algorithms
1957fbtl - file byte transfer layer: abstraction for individual
1958 read/write operations for OMPIO
1959fcoll - collective read and write operations for MPI I/O
1960fs - file system functions for MPI I/O
1961io - MPI I/O
1962mtl - Matching transport layer, used for MPI point-to-point
1963 messages on some types of networks
1964op - Back end computations for intrinsic MPI_Op operators
1965osc - MPI one-sided communications
1966pml - MPI point-to-point management layer
1967rte - Run-time environment operations
1968sharedfp - shared file pointer operations for MPI I/O
1969topo - MPI topology routines
1970vprotocol - Protocols for the "v" PML
1971
1972OpenSHMEM component frameworks:
1973-------------------------
1974
1975atomic - OpenSHMEM atomic operations
1976memheap - OpenSHMEM memory allocators that support the
1977 PGAS memory model
1978scoll - OpenSHMEM collective operations
1979spml - OpenSHMEM "pml-like" layer: supports one-sided,
1980 point-to-point operations
1981sshmem - OpenSHMEM shared memory backing facility
1982
1983
1984Back-end run-time environment (RTE) component frameworks:
1985---------------------------------------------------------
1986
1987dfs - Distributed file system
1988errmgr - RTE error manager
1989ess - RTE environment-specific services
1990filem - Remote file management
1991grpcomm - RTE group communications
1992iof - I/O forwarding
1993notifier - System-level notification support
1994odls - OpenRTE daemon local launch subsystem
1995oob - Out of band messaging
1996plm - Process lifecycle management
1997ras - Resource allocation system
1998rmaps - Resource mapping system
1999rml - RTE message layer
2000routed - Routing table for the RML
2001rtc - Run-time control framework
2002schizo - OpenRTE personality framework
2003state - RTE state machine
2004
2005Miscellaneous frameworks:
2006-------------------------
2007
2008allocator - Memory allocator
2009backtrace - Debugging call stack backtrace support
2010btl - Point-to-point Byte Transfer Layer
2011dl - Dynamic loading library interface
2012event - Event library (libevent) versioning support
2013hwloc - Hardware locality (hwloc) versioning support
2014if - OS IP interface support
2015installdirs - Installation directory relocation services
2016memchecker - Run-time memory checking
2017memcpy - Memory copy support
2018memory - Memory management hooks
2019mpool - Memory pooling
2020patcher - Symbol patcher hooks
2021pmix - Process management interface (exascale)
2022pstat - Process status
2023rcache - Memory registration cache
2024sec - Security framework
2025shmem - Shared memory support (NOT related to OpenSHMEM)
2026timer - High-resolution timers
2027
2028---------------------------------------------------------------------------
2029
2030Each framework typically has one or more components that are used at
2031run-time. For example, the btl framework is used by the MPI layer to
2032send bytes across different types underlying networks. The tcp btl,
2033for example, sends messages across TCP-based networks; the openib btl
2034sends messages across OpenFabrics-based networks.
2035
2036Each component typically has some tunable parameters that can be
2037changed at run-time. Use the ompi_info command to check a component
2038to see what its tunable parameters are. For example:
2039
2040 shell$ ompi_info --param btl tcp
2041
2042shows some of the parameters (and default values) for the tcp btl
2043component (use --level to show *all* the parameters; see below).
2044
2045Note that ompi_info only shows a small number a component's MCA
2046parameters by default. Each MCA parameter has a "level" value from 1
2047to 9, corresponding to the MPI-3 MPI_T tool interface levels. In Open
2048MPI, we have interpreted these nine levels as three groups of three:
2049
2050 1. End user / basic
2051 2. End user / detailed
2052 3. End user / all
2053
2054 4. Application tuner / basic
2055 5. Application tuner / detailed
2056 6. Application tuner / all
2057
2058 7. MPI/OpenSHMEM developer / basic
2059 8. MPI/OpenSHMEM developer / detailed
2060 9. MPI/OpenSHMEM developer / all
2061
2062Here's how the three sub-groups are defined:
2063
2064 1. End user: Generally, these are parameters that are required for
2065 correctness, meaning that someone may need to set these just to
2066 get their MPI/OpenSHMEM application to run correctly.
2067 2. Application tuner: Generally, these are parameters that can be
2068 used to tweak MPI application performance.
2069 3. MPI/OpenSHMEM developer: Parameters that either don't fit in the
2070 other two, or are specifically intended for debugging /
2071 development of Open MPI itself.
2072
2073Each sub-group is broken down into three classifications:
2074
2075 1. Basic: For parameters that everyone in this category will want to
2076 see.
2077 2. Detailed: Parameters that are useful, but you probably won't need
2078 to change them often.
2079 3. All: All other parameters -- probably including some fairly
2080 esoteric parameters.
2081
2082To see *all* available parameters for a given component, specify that
2083ompi_info should use level 9:
2084
2085 shell$ ompi_info --param btl tcp --level 9
2086
2087These values can be overridden at run-time in several ways. At
2088run-time, the following locations are examined (in order) for new
2089values of parameters:
2090
20911. <prefix>/etc/openmpi-mca-params.conf
2092
2093 This file is intended to set any system-wide default MCA parameter
2094 values -- it will apply, by default, to all users who use this Open
2095 MPI installation. The default file that is installed contains many
2096 comments explaining its format.
2097
20982. $HOME/.openmpi/mca-params.conf
2099
2100 If this file exists, it should be in the same format as
2101 <prefix>/etc/openmpi-mca-params.conf. It is intended to provide
2102 per-user default parameter values.
2103
21043. environment variables of the form OMPI_MCA_<name> set equal to a
2105 <value>
2106
2107 Where <name> is the name of the parameter. For example, set the
2108 variable named OMPI_MCA_btl_tcp_frag_size to the value 65536
2109 (Bourne-style shells):
2110
2111 shell$ OMPI_MCA_btl_tcp_frag_size=65536
2112 shell$ export OMPI_MCA_btl_tcp_frag_size
2113
21144. the mpirun/oshrun command line: --mca <name> <value>
2115
2116 Where <name> is the name of the parameter. For example:
2117
2118 shell$ mpirun --mca btl_tcp_frag_size 65536 -np 2 hello_world_mpi
2119
2120These locations are checked in order. For example, a parameter value
2121passed on the mpirun command line will override an environment
2122variable; an environment variable will override the system-wide
2123defaults.
2124
2125Each component typically activates itself when relevant. For example,
2126the usNIC component will detect that usNIC devices are present and
2127will automatically be used for MPI communications. The SLURM
2128component will automatically detect when running inside a SLURM job
2129and activate itself. And so on.
2130
2131Components can be manually activated or deactivated if necessary, of
2132course. The most common components that are manually activated,
2133deactivated, or tuned are the "BTL" components -- components that are
2134used for MPI point-to-point communications on many types common
2135networks.
2136
2137For example, to *only* activate the TCP and "self" (process loopback)
2138components are used for MPI communications, specify them in a
2139comma-delimited list to the "btl" MCA parameter:
2140
2141 shell$ mpirun --mca btl tcp,self hello_world_mpi
2142
2143To add shared memory support, add "vader" into the command-delimited
2144list (list order does not matter):
2145
2146 shell$ mpirun --mca btl tcp,vader,self hello_world_mpi
2147
2148(there is an "sm" shared memory BTL, too, but "vader" is a newer
2149generation of shared memory support; by default, "vader" will be used
2150instead of "sm")
2151
2152To specifically deactivate a specific component, the comma-delimited
2153list can be prepended with a "^" to negate it:
2154
2155 shell$ mpirun --mca btl ^tcp hello_mpi_world
2156
2157The above command will use any other BTL component other than the tcp
2158component.
2159
2160===========================================================================
2161
2162Common Questions
2163----------------
2164
2165Many common questions about building and using Open MPI are answered
2166on the FAQ:
2167
2168 https://www.open-mpi.org/faq/
2169
2170===========================================================================
2171
2172Got more questions?
2173-------------------
2174
2175Found a bug? Got a question? Want to make a suggestion? Want to
2176contribute to Open MPI? Please let us know!
2177
2178When submitting questions and problems, be sure to include as much
2179extra information as possible. This web page details all the
2180information that we request in order to provide assistance:
2181
2182 https://www.open-mpi.org/community/help/
2183
2184User-level questions and comments should generally be sent to the
2185user's mailing list (users@lists.open-mpi.org). Because of spam, only
2186subscribers are allowed to post to this list (ensure that you
2187subscribe with and post from *exactly* the same e-mail address --
2188joe@example.com is considered different than
2189joe@mycomputer.example.com!). Visit this page to subscribe to the
2190user's list:
2191
2192 http://lists.open-mpi.org/mailman/listinfo/users
2193
2194Developer-level bug reports, questions, and comments should generally
2195be sent to the developer's mailing list (devel@lists.open-mpi.org).
2196Please do not post the same question to both lists. As with the
2197user's list, only subscribers are allowed to post to the developer's
2198list. Visit the following web page to subscribe:
2199
2200 http://lists.open-mpi.org/mailman/listinfo/devel
2201
2202Make today an Open MPI day!
2203