High Level Differences from ikt

  1. Mox is an entirely separate cluster. They share nothing with one another.
  2. You only get what you ask for, regardless of the resources available on the node. If you ask for 1 CPU, you'll only get one. If you ask for 1GB of RAM, you'll only get 1GB.
  3. An allocation won't get the same set of nodes all the time, just access to the particular number of nodes to which they're entitled.
  4. No occasional preemption in ckpt (formerly bf queue) for the moment.
  5. Preempted jobs get 10s to do something smart before being killed and requeued.
  6. No direct lolo mounts right now
  7. Please report any problems to help@uw.edu with Hyak as the first word in the subject. Please also let us know you're using mox not ikt.


SSH = mox.hyak.uw.edu

BBCP = mox1.hyak.uw.edu or mox2.hyak.uw.edu

File Transfers

Internal to Hyak systems

You can copy files at high speed without a password between the Hyak systems using commands like the ones below.

From ikt to mox

ikt1$ hyakbbcp myfile mox1.hyak.uw.edu:/gscratch/MYGROUP/
ikt1$ hyakbbcp -r mydirectory mox1.hyak.uw.edu:/gscratch/MYGROUP/

From mox to ikt

mox1$ hyakbbcp myfile ikt1.hyak.uw.edu:/gscratch/MYGROUP/
mox1$ hyakbbcp -r mydirectory ikt1.hyak.uw.edu:/gscratch/MYGROUP/



There's a scrubbed temporary filesystem available at /gscratch/scrubbed. Files are removed 30 days after they're created.


Home Directory

mmlsquota gscratch:home --block-size G

Allocation Directory

mmlsquota -j <my short group> gscratch --block-size G

Slurm Primer

Show Queue

All Jobs
Jobs in Allocation
squeue -p <my short group>
All Jobs in ckpt (was bf)
squeue -p ckpt
Jobs in ckpt from Allocation
squeue -A <my short group>-ckpt


Own Allocation

Like ikt you're limited to the number of nodes your group purchased for your allocation.

sbatch -p <my short group> -A <my short group> test-job.sh

Checkpoint Allocation (formerly bf queue)

If you want access to all the nodes in the cluster, you need to use the ckpt partition.

sbatch -p ckpt -A <my short group>-ckpt test-job.sh

Interactive Session

Build Allocation - usage limited by core count and time

srun -p build --pty /bin/bash

Own Allocation

srun -p <my short group name> --pty /bin/bash

Show Node and Memory Limits per Allocation


Show Job Info

scontrol show job <jobid>

Show Node Info

scontrol show node <node>

Sample Job Script

Change items that are bold and red. Check items that are bold and blue and change if necessary.

## Job Name
#SBATCH --job-name=test-job
## Resources
## Nodes
#SBATCH --nodes=2   
## Tasks per node (Slurm assumes you want to run 28 tasks per node unless explicitly told otherwise)
#SBATCH --ntasks-per-node=28
## Walltime (ten minutes)

#SBATCH --time=10:00
## Memory per node

#SBATCH --mem=2G
## Specify the working directory for this job
#SBATCH --workdir=/gscratch/MYGROUP/MYUSER/MYRUN

module load icc_<version>-impi_<VERSION>

Other documentation:

See articles on mox at below link (scroll down to links with prefix mox):