1 MPE (Multi-Processing Environment) 2 ---------------------------------- 3 4 Version 2.5.0. Oct, 2009 5 6 Mathematics and Computer Science Division 7 Argonne National Laboratory 8 9II. CONFIGURATION 10----------------- 11 12Please read the CONFIGURATION section II. a) and II. b) in INSTALL to get 13yourself familar with the Configuration Model of MPE. 14 15III. INSTALLATION INSTRUCTIONS 16------------------------------ 17 18This file describes the Cross-Compilation build instructions for MPE. 19If you don't need cross-compilation support or you don't know what 20cross-compilation is, you probably don't need to read this. INSTALL 21is all you need to know. 22 23**** The following is NO longer true in this MPE release **** 24This release of MPE has better support of cross-compilation and does 25not require a 2-step process, i.e. no cross-spec file is needed. 26Cross compilation is similar to native compilation except the addition 27of --host and --build as required by autoconf(i.e. configure). 28In short, cross-compile requires specification of CC/F77/MPI_CC/MPI_F77, 29--host and --build as part of configure command. 30More documentation will be added later. 31************************************************************** 32 33For those who need cross-compilation support, 7 to 10 variables are 34needed. These variables are determined through some runtime tests on 35cross-compiled host (i.e. the host where the libraries/executable are 36being built for). The exact number of variables depends on if the 37MPE's internal Fortran to C MPI profiling wrapper library needs to be 38built. 7 of the 10 environmental variables are related to the C compiler 39of the cross-compiled platform. 40 41CROSS_SIZEOF_CHAR : sizeof(char), e.g. 1 42CROSS_SIZEOF_SHORT : sizeof(short), e.g. 2 43CROSS_SIZEOF_INT : sizeof(int), e.g. 4 44CROSS_SIZEOF_LONG : sizeof(long), e.g. 8 45CROSS_SIZEOF_LONG_LONG : sizeof(long long), e.g. 8 46CROSS_SIZEOF_VOID_P : sizeof(void *), e.g. 8 47CROSS_BIGENDIAN : true if it is a big endian machine, false otherwise. 48 49If your MPI implementation does NOT provide any Fortran to C MPI profiling 50wrapper library, then you need to specify 3 more environmental variables. 51The following environmental variables specify properties of MPI 52implementation if F2CMPI_LIBS is defined, i.e MPE's internal Fortran 53to C MPI profiling library is used. These variables are related to 54how Fortran primitive type looks like in C. 55 56CROSS_MPI_STATUS_SIZE : value of MPI_STATUS_SIZE defined in mpif.h, i.e. 57 sizeof(MPI_STATUS_SIZE)/sizeof(MPI_Fint), e.g. 4 58CROSS_FORTRAN2C_TRUE : value of fortran's .TRUE. in C program, e.g. 1 59CROSS_FORTRAN2C_FALSE : value of fortran's .FALSE. in C program, e.g. 0 60 61 62 63III. a) Cross-Compile Specification 64----------------------------------- 65 66In order to simplify and avoid any ambiguity in the determination of these 67values, a small subpackage called "cross_spec" within MPE can be used. 68This goal of this subpackage is to generate a cross-compile specification 69file which will contain all the relevant CROSS_xxx variables listed above. 70The subapackage has its own independent configure and accepts a similar set 71of configure variables and options as MPE's configure. If you are not 72familar with MPE's configure variables and options, please see INSTALL file's 73section II. a) and II. b). Or you can do "mpe2/cross_spec/configure --help" 74for a list of the cross_spec's options and environment variables. 75The default cross-compile specification file generated by cross_spec 76is called cross_spec.txt which can be overriden by option --with-spec. 77 78Let say we are cross-compiling MPE for x86_64 (64bit linux) backend nodes 79on a 32bit linux frontend box which has the following cross-compilers: 80 81 the serial C cross-compiler : /usr/local/bin/lans64_cc, 82 the serial Fortran cross-compiler : /usr/local/bin/lans64_fc, 83 the MPI C cross-compiler : /usr/local/bin/lans64_mpicc, 84 the MPI Fortran cross-compiler : /usr/local/bin/lans64_mpif90 85 86Usually the frontend will have a command to run executable in the backend, 87let say this is called "/usr/local/bin/lans64_run -np 1", where "-np 1" 88means the executable will be run on 1 node. 89 90Now pick a build directory <build_spec_dir> to build cross_spec, be sure 91not set <build_spec_dir> as <mpe2_src_dir>. Doing so will cause conflict 92between cross_spec's Makefile and mpe2's main Makefile. 93 94One can configure cross_spec as follows: 95 96 mkdir <build_spec_dir> 97 cd <build_spec_dir> 98 <mpe2_src_dir>/cross_spec/configure --host=x86_64-linux \ 99 AR=ar RANLIB=ranlib \ 100 CC=/usr/local/bin/lans64_cc \ 101 F77=/usr/local/bin/lans64_fc \ 102 MPI_CC=/usr/local/bin/lans64_mpicc \ 103 MPI_F77=/usr/local/bin/lans64_mpif90 \ 104 MPERUN="/usr/local/bin/lans64_run -np 1" 105 make 106 make run 107 108Where --host=x86_64_linux informs configure we are building for 64bit linux. 109Notice there isn't a --prefix option in the configure command, as it isn't 110necessary to install this subpackage anywhere. Here AR and RANLIB are used 111in the configure command to highlight these variables can be added for 112platform which has specific AR and RANLIB for the backend nodes. The MPERUN 113allows users to run the tests on the backend through Makefile, i.e. through 114the "make run" commad. Without MPERUN or there isn't a command to launch 115executable to run on the backend, the "make run" will execute the tests 116locally on the frontend. In that case, one has to launch the tests separately 117by hand. 118 119There are 2 executables and 1 script created by cross_spec. They are 120cross_cc, cross_f2c and cross_env. 121 122 cross_cc: an executable that specifies various C compiler features 123 in the backend. (Has to run on the backend) 124 cross_f2c: an executable that specifies various Fortran to C features 125 in the backend. (Has to run on the backend) 126 cross_env: a script that specifies the configure options used in 127 cross_spec and these options will be used in cross-compiling 128 MPE again. (run on the frontend) 129 130If running of the cross_cc, cross_f2c and cross_env returns successfully, 131a cross_spec.txt file will be generated, an example of file is like 132 133# C compiler runtime characteristics... 134CROSS_SIZEOF_CHAR=1 135CROSS_SIZEOF_SHORT=2 136CROSS_SIZEOF_INT=4 137CROSS_SIZEOF_LONG=8 138CROSS_SIZEOF_LONG_LONG=8 139CROSS_SIZEOF_VOID_P=8 140CROSS_BIGENDIAN=false 141# Fortran to C runtime characteristics... 142CROSS_MPI_STATUS_SIZE=5 143CROSS_FORTRAN2C_TRUE=1 144CROSS_FORTRAN2C_FALSE=0 145# C and Fortran development tool variables... 146MPI_CC=/usr/local/bin/lans64_mpicc 147CC=/usr/local/bin/lans64_cc 148MPI_F77=/usr/local/bin/lans64_mpif90 149F77=/usr/local/bin/lans64_fc 150AR=ar 151RANLIB=ranlib 152 153Any other configure variables, e.g. CFLAGS, for MPE configure 154can be appended in the cross_spec.txt, as they will be proprogangated 155down to other subpackages within MPE configure. 156 157 158 159III. b) Cross-Compilation With "cross_spec.txt" 160----------------------------------------------- 161 162With cross_spec.txt generated by cross_spec subpackage, cross_compile 163MPE becomes easy. One can do it as follows: 164 165 mkdir <build_mpe2_for_backend> 166 cd <build_mpe2_for_backend> 167 <mpe2_src_dir>/configure --prefix=<mpe2_install_dir> \ 168 --host=x86_64_linux \ 169 --with-cross-spec=<build_spec_dir>/cross_spec.txt \ 170 MPERUN="/usr/local/bin/lans64_run -np 4" 171 make 172 make install 173 make installcheck-all 174 175Here AR and RANLIB are added for illustration purposes. 176 177Where --host is used to inform configure what type of host to be built for 178the backend. Again, MPERUN is optional here. With MPERUN, the 179"make installcheck-all" command will launch runtime tests to the backend 180to check if every selected components are built correctly. Notice that 181the MPERUN command here is different from the one used in cross_spec, since 182we want to run multiple processes job in the "installcheck-all". 183 184 185 instructions for BlueGeneLight (BG/L) 186 ------------------------------------- 187 188 **** Cross-Spec build **** 189 mkdir <build_spec_dir> 190 cd <build_spec_dir> 191 <mpe2_src_dir>/cross_spec/configure CC=blrts_xlc F77=blrts_xlf \ 192 MPI_CC=mpixlc MPI_F77=mpixlf77 \ 193 --host=powerpc-unknown-linux-gnu \ 194 --build=powerpc-pc-linux-gnu 195 make 196 cqsub -n 1 -t 5 cross_cc 197 cqsub -n 1 -t 5 cross_f2c 198 cross_env 199 200 Here --host is set to powerpc-unknown-linux-gnu even BG/L is not linux 201 Since we don't know what is the correct host-type for BG/L, --host 202 is arbitary set to powerpc-unknown-linux-gnu and should be different 203 from --build value so the configure knows we are doing cross-compilation. 204 Also make sure that "cqsub ... cross_cc" is finished __before__ 205 "cqsub ... cross_f2c", otherwise the cross_spec.txt may have overlapped 206 output from both cross_cc and cross_f2c at the same time. 207 208 **** MPE2 build **** 209 mkdir <build_mpe2_for_backend> 210 cd <build_mpe2_for_backend> 211 <mpe2_src_dir>/configure --prefix=<mpe2_install_dir> \ 212 --host=powerpc-unknown-linux-gnu \ 213 --build=powerpc-pc-linux-gnu \ 214 --with-cross-spec=<build_spec_dir>/cross_spec.txt \ 215 --with-java=/usr/lib/IBMJava2/jre \ 216 --disable-misc --disable-graphics \ 217 --disable-sample --disable-rlog 218 make 219 make install 220 make installcheck 221 222 instructions for BlueGene P (BG/P) 223 ------------------------------------- 224 225 **** Cross-Spec build **** 226 mkdir <build_spec_dir> 227 cd <build_spec_dir> 228 <mpe2_src_dir>/cross_spec/configure CC=bgxlc_r F77=bgxlf_r \ 229 MPI_CC=mpixlc_r MPI_F77=mpixlf77_r \ 230 --host=powerpc-unknown-linux-gnu \ 231 --build=powerpc-pc-linux-gnu 232 233 ** the "_r" is used to enable thread support in the compiler ** 234 ** mpixlc_r/mpixlf_r are in /bgsys/drivers/ppcfloor/comm/bin ** 235 236 make 237 cqsub -n 1 -t 5 cross_cc 238 cqsub -n 1 -t 5 cross_f2c 239 cross_env 240 241 Here --host is set to powerpc-unknown-linux-gnu even BG/P is not linux 242 Since we don't know what is the correct host-type for BG/P, --host 243 is arbitary set to powerpc-unknown-linux-gnu and should be different 244 from --build value so the configure knows we are doing cross-compilation. 245 Also make sure that "cqsub ... cross_cc" is finished __before__ 246 "cqsub ... cross_f2c", otherwise the cross_spec.txt may have overlapped 247 output from both cross_cc and cross_f2c at the same time. 248 249 **** MPE2 build **** 250 mkdir <build_mpe2_for_backend> 251 cd <build_mpe2_for_backend> 252 <mpe2_src_dir>/configure --prefix=<mpe2_install_dir> \ 253 --host=powerpc-unknown-linux-gnu \ 254 --build=powerpc-pc-linux-gnu \ 255 --with-cross-spec=<build_spec_dir>/cross_spec.txt\ 256 --disable-safePMPI \ 257 --disable-misc --disable-graphics \ 258 --disable-sample --disable-rlog 259 make 260 make install 261 make installcheck 262 263 264 instructions for SiCortex (SC5812) 265 ------------------------------------- 266 267 On the i686 linux cross-compiling frontend (bblogin) 268 269 **** Cross-Spec build **** 270 mkdir <build_spec_dir> 271 cd <build_spec_dir> 272 <mpe2_src_dir>/cross_spec/configure \ 273 CC="scpathcc -n32" \ 274 F77="scpathf95 -fno-second-underscore -n32" \ 275 MPI_CC="scmpicc --pathscale -n32" \ 276 MPI_F77="scmpif90 -fno-second-underscore -n32" \ 277 LIBS=-lpthread \ 278 --host=mips64el-gentoo-linux-gnu \ 279 --build=x86_64_linux 280 make 281 282 The above build is for n32 binary. For 64-bit binary, replace -n32 by -64. 283 Now log onto sicortex's login node (fd-login) 284 to launch programs with srun to get the cross_spec.txt 285 286 srun -n 1 cross_cc 287 srun -n 1 cross_f2c 288 srun -n 1 cross_env 289 290 **** MPE2 build **** 291 mkdir <build_mpe2_for_backend> 292 cd <build_mpe2_for_backend> 293 <mpe2_src_dir>/configure --prefix=<mpe2_install_dir> \ 294 --host=mips64el-gentoo-linux-gnu \ 295 --build=x86-linux-gnu_linux \ 296 --with-cross-spec=<build_spec_dir>/cross_spec.txt\ 297 --disable-safePMPI 298 make 299 make install 300 make installcheck 301 302 303 instructions for Cray X1E (Phoenix) 304 ---------------------------------- 305 306 On the i686 linux cross-compiling frontend (robin) 307 308 **** Cross-Spec build **** 309 mkdir <build_spec_dir> 310 cd <build_spec_dir> 311 <mpe2_src_dir>/cross_spec/configure CC=cc F77="ftn -s real64 -dp" \ 312 MPI_CC=cc MPI_F77="ftn -s real64 -dp" \ 313 --host=craynv-cray-unicosmp3.0.X \ 314 --build=i686-pc-linux-gnu 315 make 316 317 # Now create a cross_tests.cmd as follows: 318 cat > cross_tests.cmd <<EOF 319#!/bin/sh 320#PBS -l walltime=5:00,mppe=1 321#PBS -j oe 322#PBS -m abe 323#PBS -A xxxxx 324 325# job starts in home directory, cd to the submission directory 326cd ${PBS_O_WORKDIR} 327 328echo "running cross_cc" 329aprun -n 1 ./cross_cc 330 331echo "running cross_f2c" 332aprun -n 1 ./cross_f2c 333 334EOF 335 336 qsub cross_tests.cmd 337 # After the job is done, then run cross_env on the front-end 338 ./cross_env 339 340 341 Now make sure that java (SUN or IBM's jdk) is in your path. 342 343 **** MPE2 build **** 344 mkdir <build_mpe2_for_backend> 345 cd <build_mpe2_for_backend> 346 <mpe2_src_dir>/configure --prefix=<mpe2_install_dir> \ 347 --host=craynv-cray-unicosmp3.0.X \ 348 --build=i686-pc-linux-gnu \ 349 --with-cross-spec=<build_spec_dir>/cross_spec.txt \ 350 --disable-misc --disable-graphics \ 351 --disable-threadlogging 352 353 --disable-threadlogging disables the pthread logging support since Cray 354 backend does not seem to support pthread calls. 355 356 make 357 make install 358 make installcheck 359 360 361 instructions for Cray XT4 (jaguar) 362 ---------------------------------- 363 364 Compiling for the catamount(no thread support) on the frontend 365 366 **** Cross-Spec build **** 367 mkdir <build_spec_dir> 368 cd <build_spec_dir> 369 <mpe2_src_dir>/cross_spec/configure CC="cc --target=catamount" \ 370 F77="ftn --target=catamount" \ 371 MPI_CC="cc --target=catamount" \ 372 MPI_F77="ftn --target=catamount" \ 373 --host=x86_64-catamount-linux \ 374 --build=x86_64-linux 375 make 376 377# allocate an interactive PBS job slot to run cross_cc and cross_f2c 378 qsub -A XXXXX -V -I -lwalltime=00:15:00,size=2 379 380# cd to where cross_cc and cross_f2c re 381 cd <mpe2_src_dir>/cross_spec 382 383# run cross_cc and cross_f2c 384 yod -sz 1 cross_cc 385 yod -sz 1 cross_f2c 386 387# Now make sure that java (SUN or IBM's jdk) is in your path if it is available 388 389 **** MPE2 build **** 390 mkdir <build_mpe2_for_backend> 391 cd <build_mpe2_for_backend> 392 <mpe2_src_dir>/configure --prefix=<mpe2_install_dir> \ 393 --host=x86_64-catamount-linux \ 394 --build=x86_64-linux \ 395 --with-cross-spec=<build_spec_dir>/cross_spec.txt \ 396 CC="cc --target=catamount" \ 397 F77="ftn --target=catamount" \ 398 MPI_CC="cc --target=catamount" \ 399 MPI_F77="ftn --target=catamount" 400 make 401 make install 402 403 404 405 406 407III. c) Cross-Compilation Without "cross_spec.txt" 408-------------------------------------------------- 409 410The section contains some examples of cross-compiling MPE on older platforms: 411 412For cross-compilation with an existing version of MPICH(or LAM). e.g. ASCI-Red. 413 414*** If the Fortran to C MPI profiling wrapper library exists (it is called 415 libfmpich.a in MPICH and liblamf77mpi.a in LAM), do 416 417 setenv CC cicc 418 setenv F77 cif77 419 setenv CROSS_SIZEOF_CHAR 1 420 setenv CROSS_SIZEOF_SHORT 2 421 setenv CROSS_SIZEOF_INT 4 422 setenv CROSS_SIZEOF_LONG_LONG 8 423 setenv CROSS_BIGENDIAN false 424 ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \ 425 --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \ 426 --with-f2cmpilibs=-lfmpich \ 427 --disable-graphics 428 make 429 make install PREFIX=${MPE_INSTALL_DIR} 430 431Here assume mpicc/mpif77 scripts exist. If not, mpicc should include lines: 432 433#! /bin/sh 434cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi> 435 436 437mpif77 can be composed by including the following line 438 439#! /bin/sh 440cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi> 441 442 443<-lmpi> refers to various MPI libraries included in your MPI implementation. 444 445 446*** If the Fortran to C MPI profiling wrapper library does NOT exist, 447 then the MPE's own Fortran to C MPI profiling wrapper library needs 448 to be built. 3 additional environment variables are needed. They 449 are CROSS_MPI_STATUS_SIZE, CROSS_FORTRAN2C_TRUE & CROSS_FORTRAN2C_FALSE 450 451 setenv CC cicc 452 setenv F77 cif77 453 setenv CROSS_SIZEOF_CHAR 1 454 setenv CROSS_SIZEOF_SHORT 2 455 setenv CROSS_SIZEOF_INT 4 456 setenv CROSS_SIZEOF_LONG_LONG 8 457 setenv CROSS_BIGENDIAN false 458 setenv CROSS_MPI_STATUS_SIZE 4 459 setenv CROSS_FORTRAN2C_TRUE 1 460 setenv CROSS_FORTRAN2C_FALSE 0 461 ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \ 462 --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \ 463 --disable-graphics 464 make 465 make install PREFIX=${MPE_INSTALL_DIR} 466 467 468For Cray X1E MPI implementation: cross-compilation at a linux front-end(robin) 469(e.g. phoenix at Oak Ridge) 470 471 ${MPE_SRC_DIR}/configure \ 472 MPI_F77="ftn -s real64 -dp" \ 473 F77="ftn -s real64 -dp" \ 474 MPI_CC=cc \ 475 CC=$HOME/phoenix/bin/gcc-robin \ 476 --host=craynv-cray-unicosmp3.0.X \ 477 --build=i686-pc-linux-gnu \ 478 CROSS_SIZEOF_CHAR=1 \ 479 CROSS_SIZEOF_SHORT=2 \ 480 CROSS_SIZEOF_INT=4 \ 481 CROSS_SIZEOF_LONG_LONG=8 \ 482 CROSS_BIGENDIAN=true \ 483 CROSS_MPI_STATUS_SIZE=6 \ 484 CROSS_FORTRAN2C_TRUE=1 \ 485 CROSS_FORTRAN2C_FALSE=0 \ 486 --disable-misc --disable-graphics --disable-collchk \ 487 --disable-sample --disable-rlog 488