1\documentclass[dvipdfm,11pt]{article}
2\usepackage[dvipdfm]{hyperref} % Upgraded url package
3\parskip=.1in
4
5\begin{document}
6\markright{MPICH Logging}
7\title{MPICH Logging\\
8Version 0.1\\
9DRAFT of \today\\
10Mathematics and Computer Science Division\\
11Argonne National Laboratory}
12
13\author{David Ashton}
14
15
16\maketitle
17
18\cleardoublepage
19
20\pagenumbering{arabic}
21\pagestyle{headings}
22
23
24\section{Introduction}
25\label{sec:introduction}
26
27This manual assumes that MPICH has already been installed.  For
28instructions on how to install MPICH, see the MPICH Installer's Guide,
29or the README in the top-level MPICH directory.  This manual will
30explain how the internal logging macros are generated and how the user
31can generate log files viewable in Jumpshot.
32
33
34\section{Configuring mpich to create log files}
35\label{sec:configuring}
36
37When users run configure they can specify logging options.  There are three
38configure options to control logging.
39
40\begin{description}
41\item[\texttt{--enable-timing=<timing\_type>}]\mbox{}\\
42Add this option to enable timing.  The two options for timing\_type are
43\texttt{log} and \texttt{log\_detailed}.  The \texttt{log} option will log
44only the MPI functions.  The
45\texttt{log\_detailed} will log every function in mpich.  This option gives
46fine grained logging information and also creates large log files.  It must
47be used in conjunction with a timer-type that can log very short intervals
48on the order of 100's of nanoseconds.
49
50\item[\texttt{--with-logging=<logger>}]\mbox{}\\
51Specify the logging library to use.  Currently the only logger option is \texttt{rlog}.
52
53\item[\texttt{--enable-timer-type=<timer\_type>}]\mbox{}\\
54Specify the timer type.  The options are
55\begin{itemize}
56\item \texttt{gethrtime} -
57Solaris timer (Solaris systems only)
58\item \texttt{clock\_gettime} -
59Posix timer (where available)
60\item \texttt{gettimeofday} -
61Most Unix systems
62\item \texttt{linux86\_cycle} -
63Linux x86 cycle counter*
64\item \texttt{linuxalpha\_cycle} -
65Like linux86\_cycle, but for Linux Alpha*
66\item \texttt{gcc\_ia64\_cycle} -
67IA64 cycle counter*
68\end{itemize}
69* Note that CPU cycle counters count cycles, not elapsed time.
70Because processor frequencies are variable, especially with modern
71power-aware hardware, these are not always reliable for timing and so
72should only be used if you're sure you know what you're doing.
73
74\end{description}
75
76Here is an example:
77\begin{verbatim}
78mpich/configure
79    --enable-timing=log
80    --with-logging=rlog
81    --enable-timer-type=gettimeofday
82    ...
83\end{verbatim}
84
85\section{Generating log files}
86\label{sec:genlogs}
87Run your mpi application to create intermediate \texttt{.irlog} files.
88
89\begin{verbatim}
90mpicc myapp.c -o myapp
91mpiexec -n 3 myapp
92\end{verbatim}
93There will be .irlog files created for each process:
94\begin{verbatim}
95log0.irlog
96log1.irlog
97log2.irlog
98\end{verbatim}
99
100\section{RLOG tools}
101\label{sec:tools}
102For performance reasons each process produces a local intermediate log file
103that needs to be merged into a single rlog file.  Use the rlog tools to merge
104the \texttt{.irlog} files into an \texttt{.rlog} file.  The rlog tools are
105found in \texttt{mpich\_build/src/util/logging/rlog}. Currently they are not
106copied to the install directory.
107
108\begin{description}
109\item[\texttt{irlog2rlog}]\mbox{}\\
110This tool combines the intermediate \texttt{.irlog} files into a single
111\texttt{.rlog} file. The usage is: ``\texttt{irlog2rlog outname.rlog
112input0.irlog input1.irlog ...}'' A shortcut is provided: ``\texttt{irlog2rlog
113outname.rlog <num\_files>}''.  Execute \texttt{irlog2rlog} without any
114parameters to see the usage options.
115
116\item[\texttt{printrlog}]\mbox{}\\
117This tool prints the contents of an \texttt{.rlog} file.
118
119\item[\texttt{printirlog}]\mbox{}\\
120This tool prints the contents of an \texttt{.irlog} file.
121\end{description}
122
123Continuing the example from the previous section:
124\begin{verbatim}
125irlog2rlog myapp.rlog 3
126\end{verbatim}
127will convert \texttt{log0.irlog}, \texttt{log1.irlog} and \texttt{log2.irlog}
128to \texttt{myapp.rlog}.
129
130\section{Viewing log files}
131This section describes how to view a log file
132
133\texttt{.rlog} files can be printed from a command shell using the
134\texttt{printrlog} tool but the more interesting way to view the log files
135is from Jumpshot.  Jumpshot displays slog2 files and has a built in converter
136to convert \texttt{.rlog} files to \texttt{.slog2} files.  Start Jumpshot and
137open your \texttt{.rlog} file.  Jumpshot will ask you if you want to convert
138the file and you say yes.
139
140\section{Logging state code generation}
141\label{sec:genstates}
142
143This section can be skipped by users.  It describes the internal scripts used
144to develop the logging macros.
145
146This is how the \texttt{maint/genstates} script works:
147
148\begin{enumerate}
149\item \texttt{autogen.sh} creates \texttt{genstates} from
150\texttt{genstates.in} replacing \texttt{@PERL@} with the appropriate path to
151perl and then runs \texttt{genstates}.
152
153\item \texttt{genstates} finds all \texttt{.i}, \texttt{.h} and \texttt{.c} files
154in the mpich directory tree, searches for \texttt{\_STATE\_DECL} in each
155file and builds a list of all the MPID\_STATEs.  It validates that the states
156start in a \texttt{\_STATE\_DECL} statement, followed by a \texttt{FUNC\_ENTER}
157statement, and then at least one \texttt{FUNC\_EXIT} statement.  Errors are printed
158out if the code does not follow this format except for macros.  State declarations
159in macros are assumed to be correct.
160
161\item \texttt{genstates} finds all the \texttt{describe\_states.txt}
162files anywhere in the mpich tree.  \texttt{describe\_states.txt} files are
163optional and are used to set the output name of the state and its associated
164color.
165
166\item The \texttt{describe\_states.txt} file format is this:
167\begin{verbatim}
168MPID_STATE_XXX <user string for the state> <optional rgb color>
169\end{verbatim}
170 Here is an example line:
171\begin{verbatim}
172 MPID_STATE_MPI_SEND MPI_Send 0 0 255
173\end{verbatim}
174If you don't specify a state in a \texttt{describe\_states.txt} file then
175the state user name will be automatically created by stripping off the
176\texttt{MPID\_STATE\_} prefix and the color will be assigned a random value.
177
178\item \texttt{genstates} ouputs \texttt{mpich/src/include/mpiallstates.h}
179with this \texttt{enum} in it:
180\begin{verbatim}
181enum MPID_TIMER_STATE
182{
183    MPID_STATE_XXX,
184    ...
185};
186\end{verbatim}
187
188\item \texttt{genstates} outputs
189\texttt{mpich/src/util/logging/describe\_states.c} with the
190\texttt{MPIR\_Describe\_timer\_states()} function in it.  Currently, only
191the rlog version of \texttt{MPIR\_Describe\_timer\_states()} is generated.
192
193\end{enumerate}
194
195\end{document}
196