Child pages
  • Mox_gpu
Skip to end of metadata
Go to start of metadata

The stf group (HPC club) has Tesla P100 GPU nodes on mox.hyak. A Tesla P100 GPU has compute capability 6.0.

Interactive usage:

If you belong to the stf group (HPC club) then use below command to get a regular GPU node:

srun -p stf-gpu -A stf --nodes=1 --mem=120G  --time=1:00:00 --gres=gpu:P100:1 --pty /bin/bash

 If you belong to the stf group (HPC club) then use below command to get an interactive GPU node:

srun -p stf-int-gpu -A stf --nodes=1 --mem=120G  --time=1:00:00 --gres=gpu:P100:1 --pty /bin/bash


If you belong to a group which does not own GPU nodes then use below command to get an interactive GPU node. Note that in below ckpt queue (checkpoint queue), your job can get interrupted at anytime. For example, the interruption can happen when a user belonging to the group which owns the node submits a job.

srun -p ckpt -A xyz-ckpt --nodes=1 --mem=120G  --time=1:00:00 --gres=gpu:P100:1 --pty /bin/bash

Batch usage:

If you belong to the stf group, you can submit batch jobs which use the GPU nodes, by adding below lines to your sbatch script:

#SBATCH --partition=stf-gpu

#SBATCH --account=stf

#SBATCH --gres=gpu:P100:1


If you belong to a group which does not own GPU nodes, you can submit batch jobs which use the GPU nodes, by adding below lines to your sbatch script:

#SBATCH --partition=ckpt

#SBATCH --account=xyz-ckpt

#SBATCH --gres=gpu:P100:1

Nvidia MPS daemon configuration:

On Batch nodes you have the choice of running in MPS modes with the MPS daemon running:

  • gpu_prohibited: PROHIBITED – the GPU is not available for compute applications. 
  • gpu_exclusive_process: EXCLUSIVE_PROCESS – the GPU is assigned to only one process at a time, and individual process threads may submit work to the GPU concurrently.
  • gpu_default: DEFAULT – multiple processes can use the GPU simultaneously. Individual threads of each process may submit work to the GPU simultaneously.

OR you can request EXCLUSIVE_PROCESS or DEFAULT modes without the Nvidia MPS daemon: gpu_no_mps_exclusive_process or gpu_no_mps_default

On the interactive partition gpu nodes (sinfo -p stf-int-gpu) this will default to running in the EXCLUSIVE_PROCESS configuration to better support multiple users.

On Batch nodes it will default to the DEFAULT nvidia compute mode without MPS  daemon running if not constraint is requested. 

On ckpt partition GPU jobs if the job happens to land on an interactive partition gpu node then it will only set it EXCLUSIVE_PROCESS gpu compute mode with MPS running.

To request these features you will need to add a constraint request to you sbatch or srun schedule request with one of the following options: gpu_default, gpu_exclusive_process, gpu_no_mps_default, gpu_no_mps_exclusive_process, or gpu_prohibited.

Example:

#SBATCH --constraint=gpu_default

or

srun -p stf-gpu -A stf --nodes=1 --mem=120G  --time=1:00:00 --gres=gpu:P100:1 --constraint=gpu_default --pty /bin/bash

Compiling GPU CUDA code:

Get an interactive GPU node as shown above.

Use below command to verify that the node has a Nvidia GPU:

lspci | grep -i nvidia

Use below command to get access to the Nvidia compiler nvcc:

module load cuda/9.1.85.3

Use below command to verify that you can access nvcc:

nvcc --version

Compile the CUDA program myprogram.cu :

nvcc myprogram.cu -o myprogram

The above nvcc command uses a default GPU compute capability of 30.

The mox Tesla P100 GPUs have compute capability 6.0. Hence, you can use below command:

 nvcc myprogram.cu -o myprogram -arch=compute_60 -code=sm_60

Profile the CUDA executable myprogram:

nvprof ./myprogram

Use below command to find out details about the GPU:

nvidia-smi


  • No labels