wiki:OrioPFLOTRAN
Last modified 7 years ago Last modified on 02/07/12 10:55:29

PFLOTRAN Tuning Notes

Building

Last known working versions: March 29, 2011.

PETSc

  • Latest PETSc (has to be development version), checked out from repository or nightly tarball; set PETSC_DIR env. variable to top-level PETSc directory.
  • Configure (note that for development, debugging should *not* be disabled), for example:

On MCS workstations:

./configure --with-mpi-dir=/soft/apps/packages/mpich2-1.2.1p1-intel --with-x=0 --with-debugging=0 \
--with-blas-lapack-dir=/soft/com/packages/intel/12.0/137/mkl --CFLAGS="-O3 -g" \
--FFLAGS="-O3 -g" --download-hypre=1 PETSC_ARCH=mpich2-intel-O3 --with-hdf5=1 \
--download-hdf5=1 --prefix=/homes/norris/software/petsc-dev-pflotran

Alternate configure (to enable MPE logging, MVAPICH2 1.4.1 or later is required):

./configure --with-mpi-dir=/sandbox/chan/mpich2_work/install_132p1_intel --with-x11=0 --with-debugging=1\
--with-blas-lapack-dir=/soft/com/packages/intel/12.0/137/mkl --CFLAGS="-openmp -g -O3"\
--FFLAGS="-openmp -g -O3" --with-mpi-dir=/sandbox/chan/mpich2_work/install_132p1_intel\
--download-hypre=1 PETSC_ARCH=mpich2-intel-openmp-O3-mpe --with-hdf5=1 --download-hdf5=1

On cookie:

./configure --with-mpi-dir=/disks/soft/mpich2-1.2-intel --with-x=0 --with-debugging=0 \
--with-blas-lapack-dir=/disks/soft/intel/mkl/10.0.4.023 --CFLAGS="-O3 -g" \
--FFLAGS="-O3 -g" --download-hypre=1 PETSC_ARCH=mpich2-intel-O3 --with-hdf5=1 \
--download-hdf5=1
  • After the PETSc build is finished, manually modify ${PETSC_DIR}/${PETSC_ARCH}/conf/petscvariables, e.g.:
  • For C code: replace "-lmpe" with "-llmpe -lmpe"
  • For Fortran code: replace "-lmpe" with "-lmpe_f2cmpi -llmpe -lmpe"
  • If the compiler warns about -O overriding -O3 when you do make, you may want to edit $PETSC_DIR/$PETSC_ARCH/conf/petscvars and remove the -O from all compile flag variables (leave the -O3 -g).

On MCS workstations, PETSc (petsc-dev-peri, which reflects petsc-dev as of 3/29/11) was built on crank (/sandbox/norris/petsc-dev-peri) and installed in ~norris/software/petsc-pflotran. Available PETSC_ARCH values are mpich2-intel-O3 and mpich2-intel-debug.

PFLOTRAN

  • Latest version from repository
  • Set the PETSC_DIR and PETSC_ARCH environment variables, e.g., on MCS workstations:
    setenv PETSC_DIR /home/norris/software/petsc-dev-pflotran
    setenv PETSC_ARCH mpich2-intel-O3
    

or for bash

export PETSC_DIR=/home/norris/software/petsc-dev-pflotran
export PETSC_ARCH=mpich2-intel-O3
  • Build in the src/ directory:
make pflotran hdf5-vamsi=1

or just

make pflotran
  • Run Pflotran with MPE logging enabled
    • ./pflotran -vecscatter_alltoall
  • There is a version checked out and built on 3/29/2011 in /home/norris/research/apps/pflotran/src/pflotran on MCS workstations.
  • To run with sample inputs, copy the /sandbox/norris/pflotran_inputs directory from crank to wherever you wish to run
    • cd to the particular experiment directory, e.g., 24x24x30
    • Run the pflotran executable, e.g., mpiexec -np 8 /path/to/pflotran
    • You can use any valid PETSc options, e.g., -log_summary -- for a list of options, run pflotran with -help
  • You can run longer or shorted by changing the simulation time in pflotran.in for the given experiment, e.g.,
:=========================== times ============================================
TIME
FINAL_TIME 1.0d0 y
INITIAL_TIMESTEP_SIZE 1.d0 h
MAXIMUM_TIMESTEP_SIZE 1.d0 h
/

is for a simulated year and

:=========================== times ============================================
TIME
FINAL_TIME 100. h
INITIAL_TIMESTEP_SIZE 1.d-3 h
MAXIMUM_TIMESTEP_SIZE 1.d0 h
/

is for 100 simulated hours.

Additional Information

Attachments