Below xyz is your group name and abc is your userid.

Compiling MPI program:

See also Hyak Intel MPI

Load one of below modules.

Intel:

Intel Mox:

module load icc_18-impi_2018

Intel Ikt:

module load icc_18-impi_2018

gcc:

gcc Mox:

module load gcc_4.8.5-impi_2017 

gcc Ikt:

module load gcc_4.4.7-impi_5.1.2


Compile your program using one of below compilers:

intel C compiler is mpiicc

intel C++ compiler is mpiicpc

intel Fortran compiler is mpiifort

gnu C compiler is mpigcc

gnu C++ compiler is mpigxx

gnu Fortran compiler mpifc

generic C compiler is mpicc

generic C++ compiler is mpiCC

generic Fortran compiler mpif90

Example:

mpicc my_mpi_program.c -o my_mpi_program

Above generic means it could be intel or gnu compiler depending on what module was loaded. Below command shows details of mpicc and the compiler used:

mpicc -show

Use below command to find location of MPI:

 which mpirun

Running MPI program:

(More details are at https://slurm.schedmd.com/mpi_guide.html)

Interactive MPI:

Below must be same module which was used to compile the program.

  module load icc_17-impi_2017

 

For ikt, use below line. Do not use below line for mox.

 

  export MX_RCACHE=0

 



While running in an interactive node you must give the -np option. The -np option is not required for batch jobs.

Use full path or cd to the appropriate directory and use "mpirun -np 28 ./my_mpi_program".
 

   mpirun -np 28 /gscratch/xyz/abc/my_mpi_dir/my_mpi_program

Batch MPI:

Below is a sample MPI Job Script myscript.slurm. Change options appropriately and submit from the login node using "sbatch myscript.slurm".


#!/bin/bash

## Job Name

#SBATCH --job-name=my_mpi_job

## Allocation Definition 
#SBATCH --account=xyz 
#SBATCH --partition=xyz

## Resources

## Total number of nodes

#SBATCH --nodes=2   

##Number of cores per node

#SBATCH --ntasks-per-node=28

## Walltime (2 hours)

#SBATCH --time=2:00:00

## Memory per node

#SBATCH --mem=100G

## Specify the working directory for this job

## Make this directory before submiting this job.

#SBATCH --chdir=/gscratch/xyz/abc/my_mpi_dir

##turn on e-mail notification

#SBATCH --mail-type=ALL

#SBATCH --mail-user=your_email_address


#For ikt, use below line. Delete below line for mox.

export MX_RCACHE=0


#Below must be same module which was used to compile the program.

module load icc_17-impi_2017

#Use full path or cd to the appropriate directory and use "mpirun ./my_mpi_program".

#Here in batch mode we do not need to use -np option with mpirun.

mpirun /gscratch/xyz/abc/my_mpi_dir/my_mpi_progr