community hpc

networking & slurm

for the curious: the topology, the scheduler, and why it's put together this way.

the network

Everything sits on an isolated subnet behind a single switch. The head node is the only machine exposed to the outside world; the compute nodes have no public route at all and talk to the head node and to each other over the private network. Public SSH is key-only, and requests to this website hit a separate Raspberry Pi so a busy cluster never takes the site down.

            internet
               |
          [ router ]
               |
        [ head node ]  <- ssh, slurmctld, nfs export
               |
        [ switch, private subnet ]
         /     |     \
    node01  node02  node03   <- slurmd, no public route

Names resolve through /etc/hosts on every node rather than a DNS server. At three nodes that's simpler and one less daemon to keep alive. Home directories live on the head node and are NFS-exported to the compute nodes, so the path to your files is identical everywhere — that's what lets a job land on any node and still find its data.

the scheduler

Slurm has two halves. slurmctld runs on the head node and owns the queue: it decides what runs where and when. slurmd runs on each compute node, starts the work it's told to, and reports back. When you run sbatch, your script goes to the controller, waits for resources matching what you asked for, and then gets handed to a node.

There's one partition, main, holding all 3 nodes. Backfill scheduling is on, which means a short job can jump ahead of a long one if it fits in the gap without delaying it. In practice: small, well-specified jobs start almost immediately.

useful slurm patterns

Multiple cores on one node — for anything threaded (OpenMP, numpy, most ML libraries):

#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
./my_program

Across nodes with MPI — for genuinely distributed work:

#SBATCH --nodes=3
#SBATCH --ntasks-per-node=4
srun ./my_mpi_program

Use srun, not mpirun — Slurm already knows the node list and wires it up for you.

Job arrays — the single most useful feature here. One submission, many independent runs:

#!/bin/bash
#SBATCH --job-name=sweep
#SBATCH --array=1-100
#SBATCH --cpus-per-task=1
#SBATCH --mem=512M
#SBATCH --time=00:10:00
#SBATCH --output=logs/sweep-%A_%a.out

python train.py --seed $SLURM_ARRAY_TASK_ID

That queues 100 tasks and drains them across the cluster as cores free up.

Interactive shell on a compute node — for debugging before you commit to a batch job:

srun --pty --cpus-per-task=2 --mem=1G --time=01:00:00 bash