Slurm

Introduction

Slurm is the most popular open-source workload manager for Linux clusters released under GNU GPL v2. As a cluster manager, slurm is widely used all over the world, from academic circles to industrial areas. On one hand, it virtualizes the physically individual computing nodes as a supercomputer whereby the procedure and method of visiting resources can be greatly simplified. On other hand, it provides automatic job submission/queuing/scheduling/executing, as well as results collection.

Slurm is comprised of 3 daemons.

slurmctld is the controller daemon running on server.
slurmdbd is an optional but recommended daemon for database-based storage and statistics running on server. E.g., MySQL/mariadb can be installed.
slurmd is a client daemon running on each computing node.

Installation

In Linux, slurm can be easily installed. Taking Archlinux for instance, the installation can be completed via pacman, i.e., by running command pacman -S slurm-llnl on head/server and each computing node. Then package slurm-llnl as well as its dependency munge for authentication will be individually installed. Accordingly, a new user slurm will be added.

Additionally, following optional services are encouraged.

For host resolution within the cluster, a network information service (NIS) or a static /etc/hosts file is necessary.
Network time synchronization should be achieved in the cluster. To this end, a NTP or timesyncd service should be started.
Network file system (NFS) can be individually installed and configured, see NFS for reference.

Configuration

After the intallation, slurm should be first configured before being used. All the configuration files reside in folder /etc/slurm-llnl. The configuration can be completed by a couple of methods below.

Fill the blanks in the online configurator and then download the resultant file as configuration file, i.e., slurm.conf.
Customize the examples (e.g., cgroup.conf.example, slurm.conf.example, slurmdbd.conf.example) and remove the extension .example.

What's worth noting, all the nodes in the cluster should have the same key in /etc/munge/munge.key (in mode 0700 and owned by user munge). To this end, the file in head can be copied to each computing node.

Additionally, configless configuration is an awesome feature whereby each computing node can pull the configuration from head instead of from an individual local file (i.e., /etc/slurm-llnl/slurm.conf).

Head/server

/etc/slurm-llnl/slurm.conf

  # slurm.conf file generated by configurator.html.
  # Put this file on all nodes of your cluster.
  # See the slurm.conf man page for more information.
  #
  ClusterName=CLUSTER
  SlurmctldHost=head
  SlurmctldParameters=enable_configless,reconfig_on_restart
  ProctrackType=proctrack/cgroup
  RebootProgram=reboot
  ReturnToService=2
  SlurmctldPidFile=/var/run/slurmctld.pid
  SlurmctldPort=6817
  SlurmdPidFile=/var/run/slurmd.pid
  SlurmdPort=6818
  SlurmdSpoolDir=/var/spool/slurmd
  SlurmUser=slurm
  StateSaveLocation=/var/spool/slurmctld
  TaskPlugin=task/affinity,task/cgroup

  # TIMERS
  InactiveLimit=0
  KillWait=30
  MinJobAge=300
  SlurmctldTimeout=120
  SlurmdTimeout=300
  Waittime=0

  # SCHEDULING
  SchedulerType=sched/backfill
  SelectType=select/linear

  # LOGGING AND ACCOUNTING
  AccountingStorageEnforce=nosteps
  AccountingStorageHost=localhost
  AccountingStorageType=accounting_storage/slurmdbd
  AccountingStoragePass=/var/run/munge/munge.socket.2
  AccountingStoreFlags=job_script
  JobCompHost=localhost
  JobCompPass=SLURM
  JobCompType=jobcomp/mysql
  JobCompUser=slurm
  JobAcctGatherFrequency=300
  SlurmctldDebug=info
  SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
  SlurmdDebug=info
  SlurmdLogFile=/var/log/slurm-llnl/slurmd.log

  # COMPUTE NODES (following information of each node can be obtained by running `slurmd -C' on each computing node)
  NodeName=node[1-10] CPUs=128 RealMemory=1000000 Sockets=2 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN
  NodeName=node[11-16] CPUs=256 RealMemory=1000000 Sockets=2 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN
  PartitionName=QUEUE Nodes=ALL Default=YES MaxTime=INFINITE State=UP

/etc/slurm-llnl/cgroup.conf

  ###
  #
  # Slurm cgroup support configuration file
  #
  # See man slurm.conf and man cgroup.conf for further
  # information on cgroup configuration parameters
  #
  ConstrainCores=yes
  ConstrainDevices=yes
  ConstrainRAMSpace=yes
  ConstrainSwapSpace=yes

/etc/slurm-llnl/slurmdbd.conf (this file should be 0640, and owned by user slurm)

  #
  # See the slurmdbd.conf man page for more information.
  #
  # Authentication info
  AuthType=auth/munge
  AuthInfo=/var/run/munge/munge.socket.2
  #
  # slurmDBD info
  DbdAddr=localhost
  DbdHost=localhost
  SlurmUser=slurm
  DebugLevel=verbose
  LogFile=/var/log/slurm-llnl/slurmdbd.log
  PidFile=/var/run/slurmdbd/slurmdbd.pid
  #
  # Database info
  StorageType=accounting_storage/mysql
  StorageHost=localhost
  StoragePass=SLURM
  StorageUser=slurm
  StorageLoc=slurm_acct_db

/etc/systemd/timesyncd.conf
```
  [Time]
  NTP=0.pool.ntp.org
```

/etc/hosts

  # Static table lookup for hostnames.
  # See hosts(5) for details.
  127.0.0.1        localhost
  ::1              localhost
  xx.xx.xx.xx      head
  xx.xx.xx.xx      node1
  xx.xx.xx.xx      node2
  xx.xx.xx.xx      node3
  xx.xx.xx.xx      node4
  xx.xx.xx.xx      node5

Finally, enable and start munge.service, slurmctld.service, and slurmdbd.service.

Node/client

/etc/systemd/timesyncd.conf
```
  [Time]
  NTP=head
```

/etc/hosts

  # Static table lookup for hostnames.
  # See hosts(5) for details.
  127.0.0.1        localhost
  ::1              localhost
  x.x.x.x          head

Targeting configless mode, an additional parameter for head indication should be provided to the client service first.

Edit the service by running systemctl edit --full slurmd and append an option --conf-server head to the ExecStart.
Enable and start munge.service and slurmd.service.