doc/installguide/install.tex.vin

\documentclass[dvipdfm,11pt]{article}
\usepackage[dvipdfm]{hyperref} % Upgraded url package
\parskip=.1in

% Formatting conventions for contributors
%
% A quoting mechanism is needed to set off things like file names, command
% names, code fragments, and other strings that would confuse the flow of
% text if left undistinguished from preceding and following text.  In this
% document we use the LaTeX macro '\texttt' to indicate such text in the
% source, which normally produces, when used as in '\texttt{special text}',
% the typewriter font.

% It is particularly easy to use this convention if one is using emacs as
% the editor and LaTeX mode within emacs for editing LaTeX documents.  In
% such a case the key sequence ^C^F^T (hold down the control key and type
% 'cft') produces '\texttt{}' with the cursor positioned between the
% braces, ready for the special text to be typed.  The closing brace can
% be skipped over by typing ^e (go to the end of the line) if entering
% text or ^C-} to just move the cursor past the brace.

% LaTeX mode is usually loaded automatically.  At Argonne, one way to
% get several useful emacs tools working for you automatically is to put
% the following in your .emacs file.

% (require 'tex-site)
% (setq LaTeX-mode-hook '(lambda ()
%                        (auto-fill-mode 1)
%                        (flyspell-mode 1)
%                        (reftex-mode 1)
%                        (setq TeX-command "latex")))

%
% When updating the version number of MPICH, make sure that you change *all*
% instances of the version.  Search for mpich- and check each match.  Some
% are within a verbatim statement and will need to be changed for each update.
%

\begin{document}
\markright{MPICH Installer's Guide}
\title{{\bf MPICH Installer's Guide}\thanks{This work was supported by the
    Mathematical, Information, and Computational Sciences Division
    subprogram of the Office of Advanced Scientific Computing Research,
    SciDAC Program, Office of Science, U.S. Department of Energy, under
    Contract DE-AC02-06CH11357.}\\
  Version %MPICH_VERSION%\\
  Mathematics and Computer Science Division\\
  Argonne National Laboratory}

\author{
Abdelhalim Amer \and Pavan Balaji \and Sudheer Chunduri \and Giuseppe
Congui \and Huansong Fu \and William Gropp \and Yanfei Guo \and
Shintaro Iwasaki \and Travis Koehring \and Rob Latham \and Kavitha
Madhu \and Kaiming Ouyang \and Ken Raffenetti \and Min Si \and Rajeev
Thakur \and Rohit Zambre \and Hui Zhou
}

\maketitle
\cleardoublepage

\pagenumbering{roman}
\tableofcontents
\clearpage

\pagenumbering{arabic}
\pagestyle{headings}

%% Here is a basic outline for the document.   (Bill's original outline)

%% 0. Quick start with ``best practices''.  Each step has a reference to
%% more detailed information later in the document.
%% 1. Acquiring and unpacking.  Using a ``fast'' directory location and
%%    VPATH
%% 1a. Reporting problems
%% 2. Choosing a device (defer a detailed discussion of each until later)
%% 3. configure, make, and install.  Always use --prefix
%% show only basic options for configure
%%    3a. Optional include of device-specific information
%%    3b. Optional include of pm-specific information
%%    3c. Optional ``fast'' version
%%    3d. Shared libraries
%% 4. Testing and benchmarking
%% 4a. make testing
%% 4b. Getting, building, and using mpptest and netpipe
%% 5. Special options
%% 6. Troubleshooting
%% Appendix:
%% A. Summary of configure options (particularly the enable and with options)


\section{Introduction}
\label{sec:intro}
This manual describes how to obtain and install MPICH, the MPI
implementation from Argonne National Laboratory.  (Of course, if you are
reading this, chances are good that you have already obtained it and
found this document, among others, in its \texttt{doc} subdirectory.)
This \emph{Guide} will explain how to install MPICH so that you and others can use it to
run MPI applications.  Some particular features are different
if you have system administration privileges (can become ``root'' on a
Unix system), and these are explained here.  It is not necessary to have
such privileges to build and install MPICH.  In the event of problems,
send mail to \texttt{discuss@mpich.org}.  Once MPICH is
installed, details on how to run MPI jobs are covered in the \emph{MPICH
User's Guide}, found in this same \texttt{doc} subdirectory.

MPICH has many options.  We will first go through a recommended,
``standard'' installation in a step-by-step fashion, and later describe
alternative possibilities.


\section{Quick Start}
\label{sec:quick}

In this section we describe a ``default'' set of installation steps.
It uses the default set of configuration options, which builds the
\texttt{nemesis} communication device and the \texttt{Hydra} process
manager, for languages C, C++, Fortran-77, and Fortran-90 (if those
compilers exist), with compilers chosen automatically from the user's
environment, without tracing and debugging options.  It uses the
\texttt{VPATH} feature of \texttt{make}, so that the build process can
take place on a local disk for speed.

\subsection{Prerequisites}
\label{sec:prerequisites}

For the default installation, you will need:
\begin{enumerate}
\item A copy of the distribution, \texttt{mpich.tar.gz}.
\item A C compiler.
\item A Fortran-77, Fortran-90, and/or C++ compiler if you wish to write
  MPI programs in any of these languages.
\item Any one of a number of Unix operating systems, such as IA32-Linux.
  MPICH is most extensively tested on Linux;  there remain some
  difficulties on systems to which we do not currently have access.  Our
  \texttt{configure} script attempts to adapt MPICH to new systems.
\end{enumerate}
Configure will check for these prerequisites and try to work around
deficiencies if possible.  (If you don't have Fortran, you will
still be able to use MPICH, just not with Fortran applications.)


\subsection{From A Standing Start to Running an MPI Program}
\label{sec:steps}
Here are the steps from obtaining MPICH through running your own
parallel program on multiple machines.

\begin{enumerate}
\item
Unpack the tar file.
\begin{verbatim}
    tar xfz mpich.tar.gz
\end{verbatim}
If your tar doesn't accept the z option, use
\begin{verbatim}
    gunzip -c mpich.tar.gz | tar xf -
\end{verbatim}
Let us assume that the directory where you do this is
\texttt{/home/you/libraries}.  It will now contain a subdirectory named
\texttt{mpich-%MPICH_VERSION%}.

\item
Choose an installation directory (the default is \texttt{/usr/local/bin)}:
\begin{verbatim}
    mkdir /home/you/mpich-install
\end{verbatim}
It will be most convenient if this directory is shared by all of the
machines where you intend to run processes.  If not, you will have
to duplicate it on the other machines after installation.  Actually, if
you leave out this step, the next step will create the directory for you.

\item
Choose a build directory.  Building will proceed \emph{much} faster if
your build directory is on a file system local to the machine on which
the configuration and compilation steps are executed.  It is preferable
that this also be separate from the source directory, so that the
source directories remain
clean and can be reused to build other copies on other machines.
\begin{verbatim}
    mkdir /tmp/you/mpich-%MPICH_VERSION%
\end{verbatim}

\item
Choose any configure options.  See
Section~\ref{sec:important-configure-options} for a description of the most
important options to consider.

\item
Configure MPICH, specifying the installation directory, and running
the \texttt{configure} script in the source directory:
\begin{verbatim}
    cd  /tmp/you/mpich-%MPICH_VERSION%
    /home/you/libraries/mpich-%MPICH_VERSION%/configure \
            -prefix=/home/you/mpich-install |& tee c.txt
\end{verbatim}
where the \texttt{$\backslash$} means that this is really one line.  (On
\texttt{sh} and its derivatives, use \verb+2>&1 | tee c.txt+
instead of \verb+|& tee c.txt+).  Other configure options are
described below.  Check the \texttt{c.txt} file to make sure
everything went well.  Problems should be self-explanatory, but if not,
send \texttt{c.txt} to \texttt{discuss@mpich.org}.
The file \texttt{config.log} is created by \texttt{configure} and
contains a record of the tests that \texttt{configure} performed.  It
is normal for some tests recorded in \texttt{config.log} to fail.

\item
Build MPICH:
\begin{verbatim}
    make |& tee m.txt       (for csh and tcsh)
      OR
    make 2>&1 | tee m.txt   (for bash and sh)
\end{verbatim}
This step should succeed if there were no problems with the preceding
step.  Check file \texttt{m.txt}. If there were problems,
do a \texttt{make clean} and then run make again with \texttt{VERBOSE=1}
\begin{verbatim}
    make VERBOSE=1 |& tee m.txt       (for csh and tcsh)
      OR
    make VERBOSE=1 2>&1 | tee m.txt   (for bash and sh)
\end{verbatim}
and then send \texttt{m.txt} and \texttt{c.txt} to
\texttt{discuss@mpich.org}.

\item
Install the MPICH commands:
\begin{verbatim}
    make install |& tee mi.txt
\end{verbatim}
This step collects all required executables and scripts in the \texttt{bin}
subdirectory of the directory specified by the prefix argument to
configure.

(For users who want an install directory structure compliant to
GNU coding standards (i.e., documentation files go to
\texttt{\$\{datarootdir\}/doc/\$\{PACKAGE\}}, architecture independent
read-only files go to \texttt{\$\{datadir\}/\$\{PACKAGE\}}), replace
\texttt{make install} by

\begin{verbatim}
       make install PACKAGE=mpich-<versrion>
\end{verbatim}

and corresponding installcheck step should be

\begin{verbatim}
       make installcheck PACKAGE=mpich-<version>
\end{verbatim}

Setting PACKAGE in \texttt{make install} or \texttt{make installcheck} step
is optional and unnecessary for typical MPI users.)

\item
Add the \texttt{bin} subdirectory of the installation directory to your path:
\begin{verbatim}
    setenv PATH /home/you/mpich-install/bin:$PATH
\end{verbatim}
for \texttt{csh} and \texttt{tcsh}, or
\begin{verbatim}
    export PATH=/home/you/mpich-install/bin:$PATH
\end{verbatim}
for \texttt{bash}, and
\begin{verbatim}
    PATH=/home/you/mpich-install/bin:$PATH
    export PATH
\end{verbatim}
for \texttt{sh} (the \texttt{bash} syntax may work for more recent
implementations of \texttt{sh}).  % export var=value is an extension and
                                  % is not in original sh
Check that everything is in order at
this point by doing
\begin{verbatim}
    which mpicc
    which mpiexec
\end{verbatim}
All should refer to the commands in the \texttt{bin} subdirectory of your
install directory.  It is at this point that you will need to
duplicate this directory on your other machines if it is not
in a shared file system. % Just having NFS doesn't mean that directories
                         % are consistencly cross-mounted.

\item Check that you can reach these machines with \texttt{ssh} or
\texttt{rsh} without entering a password.  You can test by doing

\begin{verbatim}
    ssh othermachine date
\end{verbatim}

or

\begin{verbatim}
    rsh othermachine date
\end{verbatim}

If you cannot get this to work without entering a password, you will
need to configure \texttt{ssh} or \texttt{rsh} so that this can be
done.

\item Test the setup you just created:

\begin{verbatim}
    mpiexec -f machinefile -n <number> hostname
\end{verbatim}

The machinefile contains the list of hosts you want to run the
executable on.

\begin{verbatim}
  % cat machinefile
     host1     # Run 1 process on host1
     host2:4   # Run 4 processes on host2
     host3:2   # Run 2 processes on host3
     host4:1   # Run 1 process on host4
\end{verbatim}

\item
Now we will run an MPI job, using the \texttt{mpiexec} command as specified
in the MPI standard.

As part of the build process for MPICH, a simple program to compute the value
of $\pi$ by numerical integration is created in the
\texttt{mpich-%MPICH_VERSION%/examples} directory.  If the current directory is the top
level MPICH build directory, then you can run this program with
\begin{verbatim}
    mpiexec -n 5 -f machinefile ./examples/cpi
\end{verbatim}
The \texttt{cpi} example will tell you which hosts it is running on.

There are many options for \texttt{mpiexec}, by which multiple
executables can be run, hosts can be specified, separate command-line
arguments and environment variables can be passed to different
processes, and working directories and search paths for executables
can be specified.  Do
\begin{verbatim}
    mpiexec --help
\end{verbatim}
for details. A typical example is:
\begin{verbatim}
    mpiexec -f machinefile -n 1 ./main : -n 19 ./child
\end{verbatim}
to ensure that the process with rank 0 runs on your workstation.

The arguments between `:'s in this syntax are called ``argument sets,''
since they apply to a set of processes.  More argments are described in
the \textit{User's Guide}.

\end{enumerate}

If you have completed all of the above steps, you have successfully
installed MPICH and run an MPI example.

%
% DO NOT REMOVE THE FOLLOWING SECTION.  IF NECESSARY, UPDATE IT.
\subsection{Selecting the Compilers}
\label{sec:compilers}
The MPICH configure step will attempt to find the C, C++, and Fortran
compilers for you, but if you either want to override the default or
need to specify a compiler that configure doesn't recognize, you can
specify them on the command line using these variables
\begin{description}
\item[CC]The C compiler.
\item[CXX]The C++ compiler.  Use \texttt{--disable-cxx} if you do not
  want to build the MPI C++ interface
\item[F77]The Fortran 77 compiler (for the original MPI Fortran
  bindings).  Use \texttt{--disable-f77} if you do not want to build
  either the Fortran 77 or Fortran 90 MPI interfaces
\item[FC]The Fortran 90 (or later) compiler.  Use
  \texttt{--disable-fc} if you do not want to build the Fortran 90 MPI
  interfaces.  Note that in previous versions of MPICH, the variable
  name was \texttt{F90}.  As Fortran has had 3 major releases since
  Fortran 90 (95, 2003, and 2008), most tools, including those built
  with GNU autotools, have or are changing to use \texttt{FC} instead
  of \texttt{F90}.
\end{description}
For example, to select the Intel compilers instead of the GNU
compilers on a system with both, use
\begin{verbatim}
    ./configure CC=icc CXX=icpc F77=ifort FC=ifort ...
\end{verbatim}
Note the use of the same Fortran compiler, \texttt{ifort}, for both
Fortran 77 and Fortran 90; this is an increasingly common choice.

\subsection{Compiler Optimization Levels}

MPICH can be configured with two sets of compiler flags:
\texttt{CFLAGS}, \texttt{CXXFLAGS}, \texttt{FFLAGS}, \texttt{FCFLAGS}
(abbreviated as \texttt{xFLAGS}) and \texttt{MPICHLIB\_CFLAGS},
\texttt{MPICHLIB\_CXXFLAGS}, \texttt{MPICHLIB\_FFLAGS},
\texttt{MPICHLIB\_FCFLAGS} (abbreviated as \texttt{MPICHLIB\_xFLAGS})
for compilation; \texttt{LDFLAGS} and \texttt{MPICHLIB\_LDFLAGS} for linking.
All these flags can be set as part of configure command or
through environment variables.
(\texttt{CPPFLAGS} stands for C preprocessor flags, which should NOT be set)

Both \texttt{xFLAGS} and \texttt{MPICHLIB\_xFLAGS} affect the compilation
of the MPICH libraries. However, only \texttt{xFLAGS} is appended to
MPI wrapper scripts, \texttt{mpicc} and friends.

MPICH libraries are built with default compiler optimization, \texttt{-O2},
which can be modified by \texttt{--enable-fast} configure option.  For instance,
\texttt{--disable-fast} disables the default optimization option.
\texttt{--enable-fast=O<n>} sets default compiler optimization as
\texttt{-O<n>} (note that this assumes that the compiler accepts this
format).
For more details of \texttt{--enable-fast}, see the output of
\texttt{configure --help}.
Any other complicated optimization flags for MPICH libraries have
to be set throught \texttt{MPICHLIB\_xFLAGS}.  \texttt{CFLAGS} and friends
are empty by default.

For example, to build a production MPICH environment with \texttt{-O3} for all
language bindings, one can simply do

\begin{verbatim}
  ./configure --enable-fast=all,O3
\end{verbatim}

or

\begin{verbatim}
  ./configure --enable-fast=all MPICHLIB_CFLAGS=-O3 \
                                MPICHLIB_FFLAGS=-O3 \
                                MPICHLIB_CXXFLAGS=-O3 \
                                MPICHLIB_FCFLAGS=-O3
\end{verbatim}

This will cause the MPICH libraries to be built with \texttt{-O3},
and \texttt{-O3} will
not be included in the \texttt{mpicc} and other MPI wrapper script.

\subsection{Common Non-Default Configuration Options}
\label{sec:non-default}

A brief discussion of some of the \texttt{configure} options is found in
Section~\ref{configure-options}.  Here we comment on some of the most
commonly used options.
% FIXME: This section needs much more work.

\subsubsection{The Most Important Configure Options}
\label{sec:important-configure-options}
\begin{description}
\item[--prefix]Set the installation directories for MPICH.
\item[--enable-debuginfo]Provide access to the message queues for
  debuggers such as Totalview.
\item[--enable-g]Build MPICH with various debugging options.  This is
  of interest primarily to MPICH developers.  The options
\begin{verbatim}
  --enable-g=dbg,mem,log
\end{verbatim}
 are recommended in that case.
\item[--enable-fast]Configure MPICH for fastest performance at the
  expense of error reporting and other program development aids.  This
  is recommended only for getting the best performance out of proven
  production applications, and for benchmarking.
\item[--enable-shared]Build MPICH with shared libraries.
  MPICH will try to automatically detect the type of shared library
  support required. See Section~\ref{sec:shared-libraries} for more details.
\item[--with-pm]Select the process manager.  The default is
  \texttt{hydra}; also useful are \texttt{gforker} and \texttt{remshell}.  You can build with
  all three process managers by specifying
\begin{verbatim}
  --with-pm=hydra:gforker:remshell
\end{verbatim}
\item[--with-java]Set the location of Java installation.  This option
is necessary only if the default Java installation in your PATH does
not contain a valid Java installation for Jumpshot, e.g.
\begin{verbatim}
  --with-java=/opt/jdk1.6.0
\end{verbatim}
\end{description}


\subsubsection{Using the Absoft Fortran compilers with MPICH}
\label{sec:absoft}

For best results, it is important to force the Absoft Fortran compilers to
make all routine names monocase.  In addition, if lower case is chosen
(this will match common use by many programs), you must also tell the the
Absoft compiles to append an underscore to global names in order to access
routines such as \texttt{getarg} (\texttt{getarg} is not used by MPICH but is
used in some
of the tests and is often used in application programs).  We recommend
configuring MPICH with the following options
\begin{verbatim}

setenv F77 f77
setenv FFLAGS "-f -N15"
setenv FCFLAGS "-YALL_NAMES=LCS -YEXT_SFX=_"

./configure ....

\end{verbatim}


\subsection{Shared Libraries}
\label{sec:shared-libraries}

To have shared libraries created when MPICH is built, specify the
following when MPICH is configured:

\begin{verbatim}
    configure --enable-shared
\end{verbatim}


\subsection{What to Tell the Users}
\label{sec:telling}

Now that MPICH has been installed, the users have to be informed of how
to use it.  Part of this is covered in the \emph{User's Guide}.  Other
things users need to know are covered here.


\section{Migrating from MPICH1}
\label{sec:migrating}

MPICH is an all-new rewrite of MPICH1.  Although the basic steps for
installation have remained the same (\texttt{configure}, \texttt{make},
\texttt{make install}), a number of things have changed.  In this
section we attempt to point out what you may be used to in MPICH1 that
are now different in MPICH.

\subsection{Configure Options}
\label{sec:configure-options}

The arguments to \texttt{configure} are different in MPICH1 and MPICH;
the \texttt{Installer's Guide} discusses \texttt{configure}.  In
particular, the newer \texttt{configure} in MPICH does not support the
\verb+-cc=<compiler-name>+ (or \texttt{-fc}, \texttt{-c++}, or
\texttt{-f90}) options.  Instead, many of the items that could be
specified in the command line to configure in MPICH1 must now be set by
defining an environment variable.  E.g., while MPICH1 allowed
\begin{verbatim}
    ./configure -cc=pgcc
\end{verbatim}
MPICH requires
\begin{verbatim}
    ./configure CC=pgcc
\end{verbatim}
Basically, every
option to the MPICH-1 configure that does not start with
\texttt{--enable} or \texttt{--with} is not available as a configure
option in MPICH.  Instead, environment variables must be used.  This is
consistent (and required) for use of version 2 GNU \texttt{autoconf}.

\subsection{Other Differences}
Other differences between MPICH1 and MPICH include the handling of
process managers and the choice of communication device.

For example, the new process managers have a new format and slightly
different semantics for the \texttt{-machinefile} option.  Assume that
you type this data into a file named \texttt{machfile}:

\begin{verbatim}
    bp400:2
    bp401:2
    bp402:2
    bp403:2
\end{verbatim}

If you then run a parallel job with this machinefile, you would expect
ranks 0 and 1 to run on bp400 because it says to run 2 processes there
before going on to bp401.  Ranks 2 and 3 would run on bp401, and rank
4 on bp402, e.g.:

\begin{verbatim}
    mpiexec -l -machinefile machfile -n 5 hostname
\end{verbatim}

produces:
\begin{verbatim}
    0: bp400
    1: bp400
    2: bp401
    3: bp401
    4: bp402
\end{verbatim}

\section{Choosing the Communication Device}
\label{sec:choose-device}

MPICH is designed to be build with many different communication devices,
allowing an implementation to be tuned for different communication fabrics.  A
simple communication device, known as ``ch3'' (for the third version of the
``channel'' interface) is provided with MPICH and is the default choice.

The ch3 device itself supports a variety of communication methods.  These are
specified by providing the name of the method after a colon in the
\texttt{--with-device} configure option.  For example,
\texttt{--with-device=ch3:sock} selects the (older) socket-base communication
method.
Methods supported by the MPICH group include:

\begin{description}
\item[ch3:nemesis]This method is our new, high performance method. It
  has been made the default communication channel starting the 1.1
  release of MPICH.  It uses shared-memory to send messages between
  processes on the same node and the network for processes between
  nodes.  Currently sockets and Myrinet-MX are supported networks.  It
  supports \texttt{MPI\_THREAD\_MULTIPLE} and other levels of thread
  safety.
\item[ch3:sock]This method uses sockets for all communications between
  processes. It supports \texttt{MPI\_THREAD\_MULTIPLE} and other
  levels of thread safety.
\end{description}

Most installations should use the default \texttt{ch3:nemesis} method
for best performance. For platforms that are not supported by nemesis,
the \texttt{ch3:sock} method is suggested.

MPICH is designed to efficiently support all types of systems.  The
\texttt{ch3:nemesis} device is the primary focus of the MPICH group,
but other research groups and computer vendors can and have developed
both their own ch3 ``channels'' as well as complete communication
``devices'' in place of ch3.


%
% Add any configure options that are specific to each method here.  Also
% any environment variables that might apply (e.g., MPICH_INTERFACE_HOSTNAME
% or MPICH_PORT_RANGE.
%
%\subsection{Special Features of Each Communication Device}
%\subsubsection{ch3:nemesis}

\section{Installing and Managing Process Managers}
\label{sec:process-managers}
MPICH has been designed to work with multiple process managers; that
is, although you can start MPICH jobs with \texttt{mpiexec}, there are
different mechanisms by which your processes are started.  An interface
(called PMI) isolates the MPICH library code from the process manager.
Currently three process managers are distributed with MPICH
\begin{description}
\item[hydra] This is the default process manager tha natively uses the
existing daemons on the system such as ssh, slurm, pbs.
\item[gforker] This is a simple process manager that creates all
  processes on a single machine.  It is useful both for debugging and
  for running on shared memory multiprocessors.
\end{description}

\subsection{hydra} \label{sec:hydra} \texttt{hydra} is the default
process manager that launches processes using the native daemons
present on the system such as ssh, slurm, pbs, etc. To configure with
the \texttt{hydra} process manager, use

\begin{verbatim}
    configure --with-pm=hydra ...
\end{verbatim}


\subsection{gforker}
\label{sec:forker}
\texttt{gforker} is a simple process manager that runs all processes on
a single node; it's version of \texttt{mpiexec} uses the system
\texttt{fork} and \texttt{exec} calls to create the new processes.
To configure with the \texttt{gforker} process manager, use
\begin{verbatim}
    configure --with-pm=gforker ...
\end{verbatim}

\section{Testing}
\label{sec:testing}
Once MPICH has been installed, you can test it by running some of the example
programs in the \texttt{examples} directory.  A more thorough test can be run
with the command \texttt{make testing}.  This will produce a summary on
standard output, along with an XML version of the test results in
\texttt{mpich/test/mpi}. In addition, running \texttt{make testing} from the
top-level (\texttt{mpich}) directory will run tests of the commands, such as
\texttt{mpicc} and \texttt{mpiexec}, that are included with MPICH.

Other MPI test suites are available from
\url{http://www.mcs.anl.gov/mpi/mpi-test/tsuite.html}.  As part of the MPICH
development, we run the MPICH1, MPICH, C++, and Intel test suites every night
and post the results on
\url{http://www.mpich.org/static/cron/tests/}.
Other tests are run on an occasional basis.

% \subsection{Using the Intel Test Suite}
% \label{sec:intel}

% These instructions are local to our test environment at Argonne.

% How to run a select set of tests from the Intel test suite:

% \begin{small}
% \begin{verbatim}
% 1) checkout the Intel test suite (svn co https://svn.mcs.anl.gov/repos/mpi/test_suites/intel/trunk IntelMPITEST) (outside users
%    can access the most recent version of the test suite from the
%    test suite web page).

% 2) create a testing directory separate from the IntelMPITEST source
% directory

% 3) cd into that testing directory

% 4) run "<ITS_SRC_DIR>/configure --with-mpich=<MPICH_INSTALL_DIR>", where
% <ITS_SRC_DIR> is the path to the directory Intel test suite source (e.g.,
% /home/toonen/Projects/MPI-Tests/IntelMPITEST) and <MPICH_INSTALL_DIR> is
% the directory containing your MPICH installation

% 5) mkdir Test; cd Test

% 6) find tests in <ITS_SRC_DIR>/{c,fortran} that you are interested in
% running and place the test names in a file.  For example:

% % ( cd /home/toonen/Projects/MPI-Tests/IntelMPITEST/Test ; \
%     find {c,fortran} -name 'node.*' -print | grep 'MPI_Test'
%     | sed -e 's-/node\..*$--' ) |& tee testlist
% Test/c/nonblocking/functional/MPI_Test
% Test/c/nonblocking/functional/MPI_Testall
% Test/c/nonblocking/functional/MPI_Testany
% Test/c/nonblocking/functional/MPI_Testsome
% Test/c/persist_request/functional/MPI_Test_p
% Test/c/persist_request/functional/MPI_Testall_p
% Test/c/persist_request/functional/MPI_Testany_p
% Test/c/persist_request/functional/MPI_Testsome_p
% Test/c/probe_cancel/functional/MPI_Test_cancelled_false
% Test/fortran/nonblocking/functional/MPI_Test
% Test/fortran/nonblocking/functional/MPI_Testall
% Test/fortran/nonblocking/functional/MPI_Testany
% Test/fortran/nonblocking/functional/MPI_Testsome
% Test/fortran/persist_request/functional/MPI_Test_p
% Test/fortran/persist_request/functional/MPI_Testall_p
% Test/fortran/persist_request/functional/MPI_Testany_p
% Test/fortran/persist_request/functional/MPI_Testsome_p
% Test/fortran/probe_cancel/functional/MPI_Test_cancelled_false
% %

% 7) run the tests using ../bin/mtest:

% % ../bin/mtest -testlist testlist -np 6 |& tee mtest.log
% %

% NOTE: some programs hang if less they are run with less than 6 processes.

% 8) examine the summary.xml file.  look for '<STATUS>fail</STATUS>' to see if
% any failures occurred.  (search for '>fail<' works as well)

% \end{verbatim}
% \end{small}

\section{Benchmarking}
\label{sec:benchmarking}

There are many benchmarking programs for MPI implementations.  Three
that we use are \texttt{mpptest}
(\url{http://www.mcs.anl.gov/mpi/mpptest}), \texttt{netpipe}
(\url{http://www.scl.ameslab.gov/netpipe}), and \texttt{SkaMPI}
(\url{http://liinwww.ira.uka.de/~skampi}).  Each of these has
different strengths and weaknesses and reveals different properties of
the MPI implementation.

In addition, the
MPICH test suite contains a few programs to test for performance artifacts in
the directory \texttt{test/mpi/perf}.  An example of a performance artifact is
markedly different performance for the same operation when performed in two
different ways.  For example, using an MPI datatype for a non-contiguous
transfer should not be much slower than packing the data into a contiguous
buffer, sending it as a contiguous buffer, and then unpacking it into the
destination buffer.  An example of this from the MPI-1 standard illustrates
the use of MPI datatypes to transpose a matrix ``on the fly,'' and one test in
\texttt{test/mpi/perf} checks that the MPI implementation performs well in
this case.

\section{All Configure Options}
\label{configure-options}

To get the latest list of all the configure options recognized by the
top-level configure, use:
\begin{verbatim}
    configure --help
\end{verbatim}
Not all of these options may be fully supported yet.

% Fixme:  Explain some of them, at least the ones that we ask the
% users to try.

\paragraph{Notes on the configure options.}
The \texttt{--with-htmldir} and \texttt{--with-docdir} options specify the
directories into which the documentation will be installed by \texttt{make
  install}.

\appendix

\end{document}

%
% Comments on subclassing the document
% We can use \ifdevname ... \fi and \ifpmname ... \fi, as in
% \ifdevchiii .. \fi and \ifpmmpd ... \fi
% (these will still need to be defined)
% There should also be a way to select ``all'' in such a way that the
% document can still flow well, such as
% \ifdevall ... \else \ifdevchiii \else \ifdevmm \fi \fi \fi