Slurm

Table of Contents

Introduction

Slurm is the most popular open-source workload manager for Linux clusters released under GNU GPL v2. As a cluster manager, slurm is widely used all over the world, from academic circles to industrial areas. On one hand, it virtualizes the physically individual computing nodes as a supercomputer whereby the procedure and method of visiting resources can be greatly simplified. On other hand, it provides automatic job submission/queuing/scheduling/executing, as well as results collection.

Slurm is comprised of 3 daemons.

  • slurmctld is the controller daemon running on server.
  • slurmdbd is an optional but recommended daemon for database-based storage and statistics running on server. E.g., MySQL/mariadb can be installed.
  • slurmd is a client daemon running on each computing node.

Installation

In Linux, slurm can be easily installed. Taking Archlinux for instance, the installation can be completed via pacman, i.e., by running command pacman -S slurm-llnl on head/server and each computing node. Then package slurm-llnl as well as its dependency munge for authentication will be individually installed. Accordingly, a new user slurm will be added.

Additionally, following optional services are encouraged.

  • For host resolution within the cluster, a network information service (NIS) or a static /etc/hosts file is necessary.
  • Network time synchronization should be achieved in the cluster. To this end, a NTP or timesyncd service should be started.
  • Network file system (NFS) can be individually installed and configured, see NFS for reference.

Configuration

After the intallation, slurm should be first configured before being used. All the configuration files reside in folder /etc/slurm-llnl. The configuration can be completed by a couple of methods below.

  1. Fill the blanks in the online configurator and then download the resultant file as configuration file, i.e., slurm.conf.
  2. Customize the examples (e.g., cgroup.conf.example, slurm.conf.example, slurmdbd.conf.example) and remove the extension .example.

What's worth noting, all the nodes in the cluster should have the same key in /etc/munge/munge.key (in mode 0700 and owned by user munge). To this end, the file in head can be copied to each computing node.

Additionally, configless configuration is an awesome feature whereby each computing node can pull the configuration from head instead of from an individual local file (i.e., /etc/slurm-llnl/slurm.conf).

Head/server

  • /etc/slurm-llnl/slurm.conf 

    # slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ClusterName=CLUSTER
SlurmctldHost=head
SlurmctldParameters=enable_configless,reconfig_on_restart
ProctrackType=proctrack/cgroup
RebootProgram=reboot
ReturnToService=2
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld
TaskPlugin=task/affinity,task/cgroup

# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0

# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/linear

# LOGGING AND ACCOUNTING
AccountingStorageEnforce=nosteps
AccountingStorageHost=localhost
AccountingStorageType=accounting_storage/slurmdbd
AccountingStoragePass=/var/run/munge/munge.socket.2
AccountingStoreFlags=job_script
JobCompHost=localhost
JobCompPass=SLURM
JobCompType=jobcomp/mysql
JobCompUser=slurm
JobAcctGatherFrequency=300
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log

# COMPUTE NODES (following information of each node can be obtained by running `slurmd -C' on each computing node)
NodeName=node[1-10] CPUs=128 RealMemory=1000000 Sockets=2 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN
NodeName=node[11-16] CPUs=256 RealMemory=1000000 Sockets=2 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN
PartitionName=QUEUE Nodes=ALL Default=YES MaxTime=INFINITE State=UP

  • /etc/slurm-llnl/cgroup.conf 

    ###
#
# Slurm cgroup support configuration file
#
# See man slurm.conf and man cgroup.conf for further
# information on cgroup configuration parameters
#
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes

  • /etc/slurm-llnl/slurmdbd.conf (this file should be 0640, and owned by user slurm) 

    #
# See the slurmdbd.conf man page for more information.
#
# Authentication info
AuthType=auth/munge
AuthInfo=/var/run/munge/munge.socket.2
#
# slurmDBD info
DbdAddr=localhost
DbdHost=localhost
SlurmUser=slurm
DebugLevel=verbose
LogFile=/var/log/slurm-llnl/slurmdbd.log
PidFile=/var/run/slurmdbd/slurmdbd.pid
#
# Database info
StorageType=accounting_storage/mysql
StorageHost=localhost
StoragePass=SLURM
StorageUser=slurm
StorageLoc=slurm_acct_db

  • /etc/systemd/timesyncd.conf 

    [Time]
NTP=0.pool.ntp.org

  • /etc/hosts 

    # Static table lookup for hostnames.
# See hosts(5) for details.
127.0.0.1        localhost
::1              localhost
xx.xx.xx.xx      head
xx.xx.xx.xx      node1
xx.xx.xx.xx      node2
xx.xx.xx.xx      node3
xx.xx.xx.xx      node4
xx.xx.xx.xx      node5

Finally, enable and start munge.service, slurmctld.service, and slurmdbd.service.

Node/client

  • /etc/systemd/timesyncd.conf 

    [Time]
NTP=head

  • /etc/hosts 

    # Static table lookup for hostnames.
# See hosts(5) for details.
127.0.0.1        localhost
::1              localhost
x.x.x.x          head

Targeting configless mode, an additional parameter for head indication should be provided to the client service first.

  1. Edit the service by running systemctl edit --full slurmd and append an option --conf-server head to the ExecStart.
  2. Enable and start munge.service and slurmd.service.