Preemption

Preemption means that jobs are interrupted by the system and continued later. It can be done in different ways, in case of hyak it means that the backfill jobs are killed and later restarted, in most cases on different nodes. It typically happens once every 3-4 hours, but also when the node's owner wants to use her node.

Unless the job is short (less than 1 hour), you have to ensure that your code can cope with preemtion.