                      MPE (Multi-Processing Environment)
                      ----------------------------------

                          Version 2.4.3. November, 2006

                    Mathematics and Computer Science Division
                           Argonne National Laboratory

II. CONFIGURATION
-----------------

MPE can be configured and installed as an extension to most MPI standard
-compliant MPI implementations, e.g. MPICH-2, MPICH, LAM/MPI, Open/MPI,
SGI's MPI, HP-UX's MPI and IBM's MPI.  It has been integrated seamlessly
into MPICH and MPICH-2 distributions, so MPE will be installed automatically
during MPICH-X's installation process.  



II. a)  Configuration Model
---------------------------

MPE is designed to be used as an extension to an existing MPI implementation,
so its configuration model assumes a general MPI development environment.  
Here are the some of the variables that MPE configure reads, some are read 
as environmental variables and some are read from the command line arguments 
to configure.

CC          : C compiler used to create serial executable,
              e.g. xlc for IBM MPI.
MPI_CC      : C compiler used to compile MPI program and to create parallel 
              executable,
              e.g. mpcc for IBM MPI, or mpicc for MPICH.
CFLAGS      : compiler flags for CC and MPI_CC.

F77         : F77 compiler used to create serial executable,
              e.g. xlf for IBM MPI.
MPI_F77     : F77 compiler used to compile MPI program and to create parallel
              executable,
              e.g. mpxlf for IBM MPI, or mpif77 for MPICH.
FFLAGS      : compiler flags for F77 and MPI_F77.

MPI_INC     : compiler's include flag (with prefix "-I") for MPI_CC/MPI_F77, 
              e.g. "-I/usr/include" for mpi.h on IRIX64.
MPI_LIBS    : compiler's library flag (with prefix "-L" for library path and
              prefix "-l" for each library name) needed by MPI_CC/MPI_F77,
              e.g. "-L/usr/lib -lmpi" for libmpi.a on IRIX64.
F2CMPI_LIBS : compiler's library flag for Fortran to C MPI profiling wrapper
              library, e.g. "-lfmpich" when MPI_CC=mpicc & MPI_F77=mpif77
              for MPICH.

MPERUN      : MPI application launch command. This variable is necessary only
              if user wants to run "make check", "make installcheck-all" or
              "$sbindir/mpecheckinstall -run" and the default MPERUN value,
              "$bindir/mpiexec -n 4", is not correct.  This variable does
              not launch the process manager if it is required to run
              the MPI application.  Therefore, user needs to make sure that
              the MPI runtime environment has been set up correctly before
              invoking "make check", "make installcheck-all" or
              "$sbindir/mpecheckinstall -run".

Among above listed variables, CC, MPI_CC, F77 and MPI_F77 are usually set
by the corresponding environmental variables.  The rest can be set through 
command line arguments to configure.  In some MPI implementations, like 
HP-UX's, MPI_CC and MPI_F77 are reserved for use by the MPI implementation,
use the configure options to set MPI_CC and MPI_F77 instead.

For those who requires cross-compilation support, another 5 to 8 
more environmental variables are needed depends on if one needs to use the
MPE's internal Fortran to C MPI profiling wrapper library.  If your MPI
implementation does NOT provide any Fortran to C MPI profiling wrapper
library, then you need to specify at least following 5 environmental
variables regarding the properties of running environment in the executable
platform.

CROSS_SIZEOF_CHAR      : sizeof(char), e.g. 1
CROSS_SIZEOF_SHORT     : sizeof(short), e.g. 2
CROSS_SIZEOF_INT       : sizeof(int), e.g. 4
CROSS_SIZEOF_LONG_LONG : sizeof(long long), e.g. 8
CROSS_BIGENDIAN        : true if it is a big endian machine, false otherwise.

The following 3 environmental variables specify properties of MPI
implementation if F2CMPI_LIBS is defined, i.e MPE's internal Fortran
to C MPI profiling library is used.

CROSS_MPI_STATUS_SIZE  : value of MPI_STATUS_SIZE defined in mpif.h, 
                         i.e. sizeof(MPI_STATUS_SIZE)/sizeof(MPI_Fint), e.g. 4
CROSS_FORTRAN2C_TRUE   : value of fortran's .TRUE. in C program, e.g. 1
CROSS_FORTRAN2C_FALSE  : value of fortran's .FALSE. in C program, e.g. 0

II. b)  Build Options and Features
----------------------------------

MPE's configure is written using autoconf 2, and supports VPATH style install  
process.  It means the actual source directory and the building directory
can be in 2 different locations.  This allows the same source directory to be
used to build multiple versions of MPE with different options and still won't 
mess up the original source.  It is highly recommended that user should do a 
VPATH build.  Also MPE involves several different independent packages, in 
order to create a tightly integrated environment for user, it is recommended 
that user should do a "make install" to install the MPE in a separate directory
after the build is done.  The benefit is that all utility programs will be in
bin/, all libraries will be in lib/ and all graphic tools will be nicely 
organized in share/ ...  Finally, to test if the MPE installation is working
correctly, user is urged to do "make installcheck" or the demanding test
"make installcheck-all MPERUN='<path_to_mpiexec> -n 4'" in the build directory.

There are 2 types of configure options.

	1)  MPI implementation and User options
	2)  Generic configure flags supplied by autoconf 2

For a list of flags/switches for type 1 (not type 2) in MPE, do
"<mpe2>/configure --help=recursive".

For a more concise list of command line arguments like
--with-xxx and --enable-xxx, do
"<mpe2>/configure --help=recursive | grep "^--" | sort - | uniq"

The following is not a complete list but some of the more important ones.
Generic flags:
--prefix=INSTALL_DIR            Specifies the final install directory for 
                                "make install".  All libraries, utility
                                programs, graphic programs and examples
                                are installed in a standard directory
                                structure without files created in the
                                building process.

--x-includes=X_INC              Specifies the directory where X include
                                files are located.  This is used when
                                configure has trouble in locating X in
                                user system.

--x-libraries=X_LIBS            Specifies the directory where X libraries
                                are located.  This is used when configure
                                has trouble in locating X in user system.
	

MPI implementation Options:
Besides the following command line options, the serial C and Fortran compilers
can be specified through typical environmental variables CC and F77.

--with-mpicc=MPI_CC             Specify MPI C compiler to generate parallel
                                executable, e.g. mpcc for AIX.  This can be
                                specified through environmental parameter
                                MPI_CC.

--with-mpif77=MPI_F77           Specify MPI F77 compiler to generate parallel
                                executable, e.g. mpxlf for AIX.  This can be
                                specified through environmental parameter
                                MPI_F77.

--with-mpiinc=MPI_INC           Specify compiler's include flag for MPI 
                                include directory,
                                e.g. "-I/pkgs/MPI/include" for mpi.h 

--with-mpilibs=MPI_LIBS         Specify compiler's library flag for MPI 
                                libraries,
                                e.g. "-L/pkgs/MPI/lib -lpmpich -lmpich"

--enable-f77                    Enable the compilation of routines that 
                                require a Fortran compiler.  If configuring 
                                with MPICH, the configure in the top-level 
                                MPICH directory will choose the appropriate 
                                value for you.  However, it can be overridden.
                                The default is yes, --enable-f77.

--enable-f2cmpilib              Enable the building of MPE's internal Fortran 
                                to C MPI wrapper library.  The default is yes,
                                --enable-f2cmpilib

--with-f2cmpilibs=F2CMPI_LIBS   Specify compiler's library flags for Fortran
                                to C MPI wrapper library.  Using this option 
                                will force --disable-f2cmpilib.
                                e.g. "-lfmpich" when configuring MPE for MPICH

Other User Options:
--enable-echo                   Turn on strong echoing.  The default is no,
                                --disable-echo.

--with-mpelibname=MPE_LIBNAME   Specify the MPE library name instead of the 
                                default 'mpe'.  e.g. if MPE_LIBNAME="MPE",
                                then the libraries generated will be libMPE.a,
                                liblMPE.a, libtMPE.a, libaMPE.a and
                                libMPE_f2cmpi.a.  This option is necessary
                                when configuring MPE for a existing and older
                                version of MPICH which has MPE installed.

--enable-misc                   build MPE miscellaneous routines, e.g.
                                MPE_Errors_xxx(), MPE_Seq_xxx(),
                                MPE_xxxTags() and MPE_GetHostName()....
                                The default enable=yes.

--enable-logging                build MPE logging routines related to
                                CLOG-2, e.g.  MPE_Log_xxx()...
                                The default enable=yes.

--enable-wrappers               build various MPI wrapper libraries.
                                MPI logging library, liblmpe.a;
                                MPI tracing library, libtmpe.a;
                                Fortran to C MPI library, libmpe_f2cmpi.a.

--enable-graphics               build MPE graphics routines.  If disabled
                                then the MPE routines that make use of X11
                                graphics will NOT be built.  This is
                                appropriate for systems that either do not
                                have the X11 include files or that do not
                                support X11 graphics.  The default enable=yes.

--enable-collchk                bild MPI Collective and Datatype
                                checking library, libmpe_collchk.a

--enable-slog2[=build]          Configure all the SLOG-2 tools.
                                --enable-slog2 without any argument will
                                use the prebuilt jar files.
                                --enable-slog2=build will rebuild all the
                                jar files from the source code.
                                The default is enable=yes.

--with-java=JAVA_HOME           Specify the path of the top-level directory 
                                of the Java, j2sdk, installation.  If this 
                                option or --with-java2 is not given, 
                                slog2sdk configure will try to locate j2sdk
                                for you to build slog2 related tools. 

--with-java2=JAVA_HOME          Specify the path of the top-level directory
                                of the Java, j2sdk, installation.  If this
                                option or --with-java2 is not given, 
                                slog2sdk configure will try to locate j2sdk
                                for you to build slog2 related tools. 

In order to achieve maximum performance benefit, it is recommended to
use latest Java 2, i.e. JDK-1.4.x to build Jumpshot-4. 




III. INSTALLATION INSTRUCTIONS
-------------------------------  

As noted earlier, the MPE library can be installed as part of the MPICH
configure or as an extension of an existing MPI implementation.  Below are
instructions and examples for typical installation of MPE on popular MPI 
implementations. 



III. a)  Configuring as part of the MPICH-X configure
-----------------------------------------------------

The configure in the MPICH directory will try to determine the necessary 
information and pass it to the MPE configure.  If no options are given, 
the MPE will automatically be configured by default.  However, the user 
can provide extra configuration information to MPE through old MPICH-1 
configure with the following options:

	-mpe_opts=MPE_OPTS

where MPE_OPTS is one or more of the choices in section II.  Multiple
instances of -mpe_opts are allowed to specify different options for 
the MPE configure.

The MPICH-2 configure works seamlessly with MPE's, so MPE configure
options can be passed down from MPICH-2 _without_ -mpe_opts.


III. b) Configuring as part of an existing MPI implementation
-------------------------------------------------------------

The following are some examples for configuring MPE for an existing
MPI implementation.

For SGI MPI, e.g. denali.mcs.anl.gov, do the following for default ABI, -n32

    setenv MAKE gmake
    ${MPE_SRC_DIR}/configure --with-mpilibs=-lmpi \
                             --with-java=/usr/java-1.1.6/usr/java
    make
    make install PREFIX=${MPE_INSTALL_DIR}
    make installcheck PREFIX=${MPE_INSTALL_DIR}

for 64 bits ABI, either set CC="cc -64" and F77="f77 -64" or add options
CFLAGS=-64 and FFLAGS=-64 to the configure options.


For IBM MPI, e.g. quad.mcs.anl.gov, do

    setenv MPI_CC mpcc
    setenv MPI_F77 mpxlf
    ${MPE_SRC_DIR}/configure --with-java=/homes/chan/pkgs/java/J1.1.8
    make
    make install PREFIX=${MPE_INSTALL_DIR}
    make installcheck PREFIX=${MPE_INSTALL_DIR}


For SUN MPI, do

    setenv MPI_CC mpcc
    setenv MPI_F77 mpf90
    ${MPE_SRC_DIR}/configure --with-mpilibs=-lmpi
    make
    make install PREFIX=${MPE_INSTALL_DIR}
    make installcheck PREFIX=${MPE_INSTALL_DIR}


For HP-UX's MPI implementation,

    ${MPE_SRC_DIR}/configure --with-mpicc=mpicc \
                             --with-mpif77=mpif77 \
                             --with-flib_path_leader="-Wl,-L"
    make
    make install PREFIX=${MPE_INSTALL_DIR}
    make installcheck PREFIX=${MPE_INSTALL_DIR}

If MPE's Fortran support on HP-UX's MPI is NOT working, try using
HP-UX's libfmpi.a if it is there.  Here is the configure options.

    ${MPE_SRC_DIR}/configure --with-mpicc=mpicc \
                             --with-mpif77=mpif77 \
                             --with-flib_path_leader="-Wl,-L" \
                             --with-f2cmpilibs=-lfmpi
    make
    make install PREFIX=${MPE_INSTALL_DIR}
    make installcheck PREFIX=${MPE_INSTALL_DIR}

note: configure option --with-flib_path_leader="-Wl,-L" may not needed.


For LAM/MPI implementation, do

    setenv MPI_CC ${LAM_INSTALL_DIR}/bin/mpicc
    setenv MPI_F77 ${LAM_INSTALL_DIR}/bin/mpif77
    ${MPE_SRC_DIR}/configure --with-mpilibs="-L${LAM_INSTALL_DIR}/lib -lpmpi" \
                             --with-java=/sandbox/jdk1.3.1
    make
    make install PREFIX=${MPE_INSTALL_DIR}
    make installcheck PREFIX=${MPE_INSTALL_DIR}

    optional tests:
    "launch the process manager": e.g. lamboot.
    make installcheck-all PREFIX=${MPE_INSTALL_DIR} \
                          MPERUN="<path_to_mpirun> -np 4"

Using MPE with LAM for fortran MPI program is not working until recently.  
Configure options listed above enable MPE's internal Fortran to C MPI library.
To use LAM's Fortran to C MPI library in LAM 6.3.3 or later, liblamf77mpi.a, 
do

    setenv MPI_CC ${LAM_INSTALL_DIR}/bin/mpicc
    setenv MPI_F77 ${LAM_INSTALL_DIR}/bin/mpif77
    ${MPE_SRC_DIR}/configure --with-mpilibs="-L${LAM_INSTALL_DIR}/lib -lpmpi" \
                             --with-f2cmpilibs=-llamf77mpi \
                             --with-java2=/sandbox/jdk1.3.1
    make
    make install PREFIX=${MPE_INSTALL_DIR}
    make installcheck PREFIX=${MPE_INSTALL_DIR}

LAM 6.5.6 to 6.5.9 has a bug on autoconf variable, WORDS_BIGENDIAN, that
interferes with MPE's configure in the correct detection of endianiness
of the machine, e.g. enabling byte swapping on little endian machine,
e.g. intel box.  For details, see

http://www.lam-mpi.org/MailArchives/lam/2002/09/4893.php
http://www.lam-mpi.org/MailArchives/lam/2002/09/4894.php
http://www.lam-mpi.org/MailArchives/lam/2002/09/4895.php
http://www.lam-mpi.org/MailArchives/lam/2002/09/4896.php

Preferred Solution: Upgrade to newer version of LAM.
         Or
Temporary Solution: Hand edit src/logging/mpe_logging_conf.h to define
                    correct WORDS_BIGENDIAN.

LAM 7.0 or later has included libpmpi.a into libmpi.a, so --with-mpilibs
may not be needed.

LAM 7.1.2b23 or older has a bug on MPI_*_c2f( MPI_*_NULL ) that causes
seg. fault, see
http://www.lam-mpi.org/MailArchives/lam/2005/08/11073.php
So fortran LAM/MPI program that uses Isend/Irecv cannot use libmpe_f2cmpi.a,
(the reason is that MPI_Wait on the request from Isend/Irecv returns
MPI_REQUEST_NULL which could not be translated by MPI_Request_c2f() in
libmpe_f2cmpi.a)

Preferred Solution: Upgrade to newer version of LAM, 7.1.2b24 or later
         Or
Temporary Solution: Configured with --with-f2cmpilibs=-llamf77mpi. 


For OpenMPI, do

    ${MPE_SRC_DIR}/configure MPI_CC=${OPENMPI_INSTALL_DIR}/bin/mpicc \
                             CC=<C_COMPILER_used_to_build_OpenMPI> \
                             MPI_F77=${OPENMPI_INSTALL_DIR}/bin/mpif77 \
                             F77=<FORTRAN_COMPILER_used_to_build_OpenMPI> \
                             --with-java2=/usr/local/j2sdk-1.4.2
                             --prefix=<MPE_INSTALL_DIR>
    make
    make install
    make installcheck
    make installcheck-all MPERUN="${OPENMPI_INSTALL_DIR}/bin/mpiexec -n 4"

OpenMPI-1.0 does not instantiate MPI_F_STATUS(ES)_IGNORE, so fortran program
may not be able to use MPI_STATUS(ES)_IGNORE with OpenMPI+MPE.

See http://www.open-mpi.org/community/lists/users/2005/11/0380.php


For prebuilt version of MPICH-1.1.2 or MPICH-1.2.0, e.g. donner, do

    setenv MPI_CC ${MPICH_INSTALL_DIR}/bin/mpicc
    setenv MPI_F77 ${MPICH_INSTALL_DIR}/bin/mpif77
    ${MPE_SRC_DIR}/configure  --with-f2cmpilibs=-lfmpich \
                              --with-mpelibname=newMPE \
                              --with-java=/sandbox/j2sdk1.4.2
    make
    make install PREFIX=${MPE_INSTALL_DIR}
    make installcheck PREFIX=${MPE_INSTALL_DIR}

It is important to use the configure option --with-mpelibname to specify a 
different MPE library name than the default "mpe" when configuring MPE for
older MPICH.  Without this option, the linkage tests in MPE would most likely
use the old MPE libraries in the MPICH instead of the newly built MPE 
libraries in resolving the MPE symbols.  Also the option --with-f2cmpilibs
forces MPE to use the Fortran to C MPI wrapper library in previous
version of MPICH.  Not using --with-f2cmpilibs would cause MPE to use
its own Fortran to C MPI profiling wrapper library. 


For cross-compilation with an existing version of MPICH(or LAM). e.g. ASCI-Red.

*** If the Fortran to C MPI profiling wrapper library exists (it is called
    libfmpich.a in MPICH and liblamf77mpi.a in LAM), do 

    setenv CC cicc
    setenv F77 cif77
    setenv CROSS_SIZEOF_CHAR 1
    setenv CROSS_SIZEOF_SHORT 2
    setenv CROSS_SIZEOF_INT 4
    setenv CROSS_SIZEOF_LONG_LONG 8
    setenv CROSS_BIGENDIAN false 
    ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \
                             --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \
                             --with-f2cmpilibs=-lfmpich \
                             --disable-graphics
    make
    make install PREFIX=${MPE_INSTALL_DIR}

Here assume mpicc/mpif77 scripts exist.  If not, mpicc should include lines:

#! /bin/sh
cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi>


mpif77 can be composed by including the following line

#! /bin/sh
cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi>


<-lmpi> refers to various MPI libraries included in your MPI implementation.


*** If the Fortran to C MPI profiling wrapper library does NOT exist,
    then the MPE's own Fortran to C MPI profiling wrapper library needs
    to be built.  3 additional environmental variables are needed.  They
    are CROSS_MPI_STATUS_SIZE, CROSS_FORTRAN2C_TRUE & CROSS_FORTRAN2C_FALSE

    setenv CC cicc
    setenv F77 cif77
    setenv CROSS_SIZEOF_CHAR 1
    setenv CROSS_SIZEOF_SHORT 2
    setenv CROSS_SIZEOF_INT 4
    setenv CROSS_SIZEOF_LONG_LONG 8
    setenv CROSS_BIGENDIAN false 
    setenv CROSS_MPI_STATUS_SIZE 4 
    setenv CROSS_FORTRAN2C_TRUE  1 
    setenv CROSS_FORTRAN2C_FALSE 0
    ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \
                             --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \
                             --disable-graphics
    make
    make install PREFIX=${MPE_INSTALL_DIR}


For Cray X1E MPI implementation: cross-compilation at a linux front-end
(e.g. phoenix at Oak Ridge)

    ${MPE_SRC_DIR}/configure                                              \
    MPI_F77="ftn -s real64 -dp"                                           \
    F77="ftn -s real64 -dp"                                               \
    MPI_CC=cc                                                             \
    CC=$HOME/phoenix/bin/gcc-robin                                        \
    --host=craynv-cray-unicosmp3.0.X                                      \
    --build=i686-pc-linux-gnu                                             \
    CROSS_SIZEOF_CHAR=1                                                   \
    CROSS_SIZEOF_SHORT=2                                                  \
    CROSS_SIZEOF_INT=4                                                    \
    CROSS_SIZEOF_LONG_LONG=8                                              \
    CROSS_BIGENDIAN=true                                                  \
    CROSS_MPI_STATUS_SIZE=24                                              \
    CROSS_FORTRAN2C_TRUE=1                                                \
    CROSS_FORTRAN2C_FALSE=0                                               \
    --disable-misc --disable-graphics --disable-collchk                   \
    --disable-sample --disable-rlog


For BlueGene/Light's MPI implementation: If the compilers are set up correctly

    ${MPE_SRC_DIR}/configure                                               \
    MPI_CC=blrts_xlc MPI_F77=blrts_xlf CC=xlc F77=xlf                      \
    MPI_INC="-I/bgl/BlueLight/ppcfloor/bglsys/include"                     \
    MPI_LIBS="-L/bgl/BlueLight/ppcfloor/bglsys/lib                         \
             -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts -lc"       \
    CFLAGS=-O3 FFLAGS=-O3                                                  \
    --with-java=/usr/lib/IBMJava2/jre                                      \
    --disable-misc --disable-graphics --disable-sample --disable-rlog

i.e. make sure that blrts_xlc uses the same xlc as listed in CC, the same
     applies to blrts_xlf.

If the above does not work, try the following:

for older XL compilers, do

    ${MPE_SRC_DIR}/configure                                               \
    MPI_CC="/opt/ibmcmp/vac/7.0/bin/xlc                                    \
           -F /etc/opt/ibmcmp/vac/7.0/blrts-vac.cfg"                       \
    MPI_F77="/opt/ibmcmp/xlf/9.1/bin/xlf                                   \
           -F /etc/opt/ibmcmp/xlf/9.1/blrts-xlf.cfg"                       \
    CC="/opt/ibmcmp/vac/7.0/bin/xlc -F /etc/opt/ibmcmp/vac/7.0/vac.cfg"    \
    F77="/opt/ibmcmp/xlf/9.1/bin/xlf -F /etc/opt/ibmcmp/xlf/9.1/xlf.cfg"   \
    MPI_INC="-I/bgl/BlueLight/ppcfloor/bglsys/include"                     \
    MPI_LIBS="-L/bgl/BlueLight/ppcfloor/bglsys/lib                         \
             -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts -lc"       \
    --with-java=/usr/lib/IBMJava2/jre                                      \
    --disable-misc --disable-graphics --disable-sample --disable-rlog      \
    CFLAGS=-O3 FFLAGS=-O3  

For newer XL compilers, do

    ${MPE_SRC_DIR}/configure                                               \
    CC="/soft/apps/ibmcmp-20050414/vac/7.0/bin/xlc                         \
       -F /soft/apps/ibmcmp-20050414/etc/opt/ibmcmp/vac/7.0/vac.cfg"       \
    F77="/soft/apps/ibmcmp-20050414/xlf/9.1/bin/xlf                        \
        -F /soft/apps/ibmcmp-20050414/etc/opt/ibmcmp/xlf/9.1/xlf.cfg"      \
    MPI_CC="/soft/apps/ibmcmp-20050414/vac/7.0/bin/xlc -F                  \
        /soft/apps/ibmcmp-20050414/etc/opt/ibmcmp/vac/7.0/blrts-vac.cfg"   \
    MPI_F77="/soft/apps/ibmcmp-20050414/xlf/9.1/bin/xlf -F                 \
        /soft/apps/ibmcmp-20050414/etc/opt/ibmcmp/xlf/9.1/blrts-xlf.cfg"   \
    MPI_INC="-I/bgl/BlueLight/DRV100_2005-050311PM/ppc/bglsys/include"     \
    MPI_LIBS="-L/bgl/BlueLight/DRV100_2005-050311PM/ppc/bglsys/lib         \
             -lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts -lc"       \
    CFLAGS=-O3 FFLAGS=-O3                                                  \
    --with-java=/usr/lib/IBMJava2/jre                                      \
    --disable-misc --disable-graphics --disable-sample --disable-rlog

Preferred approach: MPI_INC and MPI_LIBS should be included in some kind of
mpicc/mpif77 scripts, e.g. mpicc_ibm(or mpxlc) and mpif77_ibm(or mpxlf)
wrappers

     ${MPE_SRC_DIR}/configure                                              \
     MPI_CC=/home/chan/bin/mpicc_ibm                                       \
     MPI_F77=/home/chan/bin/mpif77_ibm                                     \
     CC=/opt/ibmcmp/vac/7.0/bin/xlc                                        \
     F77=/opt/ibmcmp/xlf/9.1/bin/xlf                                       \
     --with-java=/usr/lib/IBMJava2/jre                                     \
     --disable-misc --disable-graphics --disable-sample --disable-rlog

After the successful configure, do make and make install.

     make
     make install PREFIX=${MPE_INSTALL_DIR}


For older MacOSX's MPICH2 which is built with gcc and gfortran:

    ${MPE_SRC_DIR}/configure MPI_CC=${MPICH2_INSTALL_DIR}/bin/mpicc      \
                             MPI_F77=${MPICH2_INSTALL_DIR}/bin/mpif77    \
                             CC=gcc F77=gfortran                         \
                             MPI_LIBS="  -lSystemStubs -lSystem -ldl"    \
                             --prefix=<MPE_INSTALL_DIR>
    make
    make install
    make installcheck
    make installcheck-all MPERUN="${MPICH2_INSTALL_DIR}/bin/mpiexec -n 4"

where MPI_LIBS="  -lSystemStubs -lSystem -ldl" is needed to get fortran
program to link correctly if the mpif77 does not have the linking flags.
This seems to be necessary for some older gcc/gfortran combination on MaxOSX.


III. c) Miscellaneous Info about the Build System.
--------------------------------------------------

3 extra variables, MPE_COPTS, MPE_FOPTS, and MPE_LDOPTS, are included in 
almost all of MPE's Makefiles to mimic the usage of CFLAGS, FFLAGS and LDFLAGS.
   MPE_COPTS  is the CFLAGS equivalent,
   MPE_FOPTS  is the FFLAGS equivalent,
   MPE_LDOPTS is the LDFLAGS equivalent.
These variables are not set in configures and Makefiles.  So it allows
end-user to quickly set one of this variables in Makefile through environmental
variable without rerun configure.  For instance, if user forgets to set
CFLAGS=-g in configuring MPE, one can do "make MPE_COPTS=-g" instead and 
this can be done selectively in any subpackages.
