Detailed PETSc installation instructions are at this link: http://www.mcs.anl.gov/petsc/documentation/installation.html
Below are sample instructions for installing PETSc on hyak. Please modify the instructions appropriately. Here ABC is your group name and xyz is your userid. You may also want to download and build the latest version rather than the version we use in this example.
Use below command on the login node to get a build node:
On ikt.hyak and mox.hyak:
srun -p build --time=3:00:00 --mem=10G --pty /bin/bash
mkdir /gscratch/ABC/xyz/petscstuff mkdir /gscratch/ABC/xyz/petscinstall cd /gscratch/ABC/xyz/petscstuff wget http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.5.3.tar.gz tar -xvf petsc-3.5.3.tar.gz cd /gscratch/ABC/xyz/petscstuff/petsc-3.5.3
GNU Compiler Build Instructions
Use below command to find the latest gcc and impi version on ikt.hyak or mox.hyak:
module avail gcc
In below instructions use the latest gcc and impi version from output of above module command. Also modify appropriately "--with-mpi-dir=/sw/intel/impi/5.0.2.044/intel64" .
On ikt.hyak, use below instructions:
module load gcc_4.4.7-impi_5.0.2 ./configure --download-fblaslapack --prefix=/gscratch/ABC/xyz/petscinstall \ --with-mpi-dir=/sw/intel/impi/5.0.2.044/intel64 make PETSC_DIR=/gscratch/ABC/xyz/petscstuff/petsc-3.5.3 PETSC_ARCH=arch-linux2-c-debug all make PETSC_DIR=/gscratch/ABC/xyz/petscstuff/petsc-3.5.3 PETSC_ARCH=arch-linux2-c-debug install
On mox.hyak, use below instructions:
module gcc_4.8.5-impi_2017
./configure --download-fblaslapack --prefix=/gscratch/ABC/xyz/petscinstall \
--with-mpi-dir=/gscratch/sw/intel-201703/compilers_and_libraries_2017.2.174/linux/mpi/intel64
make PETSC_DIR=/gscratch/ABC/xyz/petscstuff/petsc-3.5.3 PETSC_ARCH=arch-linux2-c-debug all
make PETSC_DIR=/gscratch/ABC/xyz/petscstuff/petsc-3.5.3 PETSC_ARCH=arch-linux2-c-debug install
Please refer to Hyak's sample job scripts:
For ikt.hyak and mox.hyak:
When you run PETSc in your slurm script, you should include the same module you used when you built PETSc.
Intel Compiler Build Instructions
The Intel suite of software development products generally produce the fastest code on Hyak. Use below command to find the latest icc and impi version on ikt.hyak or mox.hyak:
module avail icc
In below instructions use the latest icc and impi version from output of above module command.
module load icc_<version>-impi_<version> ./configure --prefix=/gscratch/ABC/xyz/petsc-intel --with-mpi-dir=$I_MPI_ROOT --with-blas-lapack-dir=$MKLROOT \ --with-scalapack-lib="-L$\{MKLROOT\}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-cc=$CC \ --with-cxx=$CXX --with-fc=$FC PETSC_ARCH=linux-gnu-intel make PETSC_ARCH=linux-gnu-intel PETSC_DIR=$PWD all
Please refer to Hyak's sample job scripts:
For ikt.hyak and mox.hyak:
When you run PETSc in your slurm scripts, you should include the same module you used when you built PETSc.