Preemption

Preemption means that jobs are interrupted by the system and continued later. It can be done in different ways, in case of hyak it means that the backfill jobs are killed and later restarted, in most cases on different nodes. It typically happens once every 3-4 hours, but also when the node's owner wants to use her node.

Unless the job is short (less than 1 hour), you have to ensure that your code can cope with preemtion.

• frequently save intermittent results on the disk and do not recalculate those after a new start.
• you may output the start time when the job starts, this gives you an indication on how much processor time the job has had