    configuration information, the nodes to be managed, information about
    how those nodes are grouped into partitions, and various scheduling
    parameters associated with those partitions. This file should be
    consistent across all nodes in the cluster.
    .LP
    The file location can be modified at system build time using the
    DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF
    environment variable. The Slurm daemons also allow you to override
    both the built\-in and environment\-provided location using the "\-f"
    option on the command line.
    .LP
    The contents of the file are case insensitive except for the names of nodes
    and partitions. Any text following a "#" in the configuration file is treated
    as a comment through the end of that line.
    Changes to the configuration file take effect upon restart of
    Slurm daemons, daemon receipt of the SIGHUP signal, or execution
    of the command "scontrol reconfigure" unless otherwise noted.
    .LP
    If a line begins with the word "Include" followed by whitespace
    and then a file name, that file will be included inline with the current
    configuration file. For large or complex systems, multiple configuration files
    may prove easier to manage and enable reuse of some files (See INCLUDE
    MODIFIERS for more details).
    .LP
    Note on file permissions:
    .LP
    The \fIslurm.conf\fR file must be readable by all users of Slurm, since it
    is used by many of the Slurm commands.  Other files that are defined
    in the \fIslurm.conf\fR file, such as log files and job accounting files,
    may need to be created/owned by the user "SlurmUser" to be successfully
    accessed.  Use the "chown" and "chmod" commands to set the ownership
    and permissions appropriately.
    See the section \fBFILE AND DIRECTORY PERMISSIONS\fR for information
    about the various files and directories used by Slurm.

    .SH "PARAMETERS"
    .LP
    The overall configuration parameters available include:

    .TP
    \fBAccountingStorageBackupHost\fR
    The name of the backup machine hosting the accounting storage database.
    If used with the accounting_storage/slurmdbd plugin, this is where the backup
    slurmdbd would be running.
    Only used with systems using SlurmDBD, ignored otherwise.

    .TP
    \fBAccountingStorageEnforce\fR
    This controls what level of association\-based enforcement to impose
    on job submissions.  Valid options are any combination of
    \fIassociations\fR, \fIlimits\fR, \fInojobs\fR, \fInosteps\fR, \fIqos\fR, \fIsafe\fR, and \fIwckeys\fR, or
    \fIall\fR for all things (expect nojobs and nosteps, they must be requested as well).
    like wise if nosteps is set Slurm will not account for any steps ran limits
    will still be enforced.

    If safe is enforced, a job will only be launched against an association or qos
    that has a GrpTRESMins limit set if the job will be able to run to completion.
    Without this option set, jobs will be launched as long as their usage
    hasn't reached the cpu-minutes limit which can lead to jobs being
    launched but then killed when the limit is reached.

    With qos and/or wckeys enforced jobs will not be scheduled unless a valid qos
    and/or workload characterization key is specified.

    When \fBAccountingStorageEnforce\fR is changed, a restart of the slurmctld
    daemon is required (not just a "scontrol reconfig").

    .TP
    \fBAccountingStorageHost\fR
    The name of the machine hosting the accounting storage database.
    Only used with systems using SlurmDBD, ignored otherwise.
    Also see \fBDefaultStorageHost\fR.

    .TP
    \fBAccountingStorageLoc\fR
    The fully qualified file name where accounting records are written
    when the \fBAccountingStorageType\fR is "accounting_storage/filetxt".
    Also see \fBDefaultStorageLoc\fR.

    .TP
    \fBAccountingStoragePass\fR
    The password used to gain access to the database to store the
    accounting data.  Only used for database type storage plugins, ignored
    otherwise.  In the case of Slurm DBD (Database Daemon) with MUNGE
    authentication this can be configured to use a MUNGE daemon
    specifically configured to provide authentication between clusters
    while the default MUNGE daemon provides authentication within a
    cluster.  In that case, \fBAccountingStoragePass\fR should specify the
    named port to be used for communications with the alternate MUNGE
    daemon (e.g.  "/var/run/munge/global.socket.2"). The default value is
    NULL.  Also see \fBDefaultStoragePass\fR.

    .TP
    \fBAccountingStoragePort\fR
    The listening port of the accounting storage database server.
    Only used for database type storage plugins, ignored otherwise.
    The default value is SLURMDBD_PORT as established at system
    build time. If no value is explicitly specified, it will be set to 6819.
    This value must be equal to the \fBDbdPort\fR parameter in the
    slurmdbd.conf file.
    Also see \fBDefaultStoragePort\fR.

    .TP
    \fBAccountingStorageTRES\fR
    Given a configuration of
    "AccountingStorageTRES=gres/gpu,gres/gpu:tesla,gres/gpu:volta"
    Then "gres/gpu:tesla" and "gres/gpu:volta" will track only jobs that explicitly
    request those two GPU types, while "gres/gpu" will track allocated GPUs of any
    type ("tesla", "volta" or any other GPU type).

    Given a configuration of
    "AccountingStorageTRES=gres/gpu:tesla,gres/gpu:volta"
    Then "gres/gpu:tesla" and "gres/gpu:volta" will track jobs that explicitly
    request those GPU types.
    If a job requests GPUs, but does not explicitly specify the GPU type, then
    it's resource allocation will be accounted for as either "gres/gpu:tesla" or
    "gres/gpu:volta", although the accounting may not match the actual GPU type
    allocated to the job and the GPUs allocated to the job could be heterogeneous.
    In an environment containing various GPU types, use of a job_submit plugin
    may be desired in order to force jobs to explicitly specify some GPU type.

    .TP
    \fBAccountingStorageType\fR
    The accounting storage mechanism type.  Acceptable values at
    present include "accounting_storage/filetxt", "accounting_storage/none"
    and "accounting_storage/slurmdbd".  The
    "accounting_storage/filetxt" value indicates that accounting records
    will be written to the file specified by the
    \fBAccountingStorageLoc\fR parameter.
    The "accounting_storage/slurmdbd" value indicates that accounting records
    will be written to the Slurm DBD, which manages an underlying MySQL
    database. See "man slurmdbd" for more information.  The
    default value is "accounting_storage/none" and indicates that account
    records are not maintained.
    Note: The filetxt plugin records only a limited subset of accounting
    information and will prevent some sacct options from proper operation.
    Also see \fBDefaultStorageType\fR.

    .TP
    \fBAccountingStorageUser\fR
    The user account for accessing the accounting storage database.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBDefaultStorageUser\fR.

    .TP
    \fBAccountingStoreJobComment\fR
    If set to "YES" then include the job's comment field in the job
    complete message sent to the Accounting Storage database.  The default
    is "YES".
    Note the AdminComment and SystemComment are always recorded in the database.

    .TP
    \fBAcctGatherNodeFreq\fR
    The AcctGather plugins sampling interval for node accounting.
    For AcctGather plugin values of none, this parameter is ignored.
    For all other values this parameter is the number
    real consumption. In case of node sharing between jobs the reported consumed
    energy per job (through sstat or sacct) will not reflect the real energy
    consumed by the jobs.

    Configurable values at present are:
    .RS
    .TP 20
    \fBacct_gather_energy/none\fR
    No energy consumption data is collected.
    .TP
    \fBacct_gather_energy/ipmi\fR
    Energy consumption data is collected from the Baseboard Management Controller
    (BMC) using the Intelligent Platform Management Interface (IPMI).
    .TP
    \fBacct_gather_energy/rapl\fR
    Energy consumption data is collected from hardware sensors using the Running
    Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the
    execution of the command "sudo modprobe msr".
    .RE

    .TP
    \fBAcctGatherInfinibandType\fR
    Identifies the plugin to be used for infiniband network traffic accounting.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    network traffic data for jobs and nodes.
    The collection of network traffic data takes place on the node level,
    hence only in case of exclusive job allocation the collected values will
    reflect the job's real traffic. In case of node sharing between jobs the reported
    network traffic per job (through sstat or sacct) will not reflect the real
    network traffic by the jobs.

    Configurable values at present are:
    .RS
    .TP 20
    \fBacct_gather_infiniband/none\fR
    No infiniband network data are collected.
    .TP
    \fBacct_gather_infiniband/ofed\fR
    Infiniband network traffic data are collected from the hardware monitoring
    counters of Infiniband devices through the OFED library.
    In order to account for per job network traffic, add the "ic/ofed" TRES to
    \fIAccountingStorageTRES\fR.
    .RE

    .TP
    \fBAcctGatherFilesystemType\fR
    Identifies the plugin to be used for filesystem traffic accounting.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    filesystem traffic data for jobs and nodes.
    The collection of filesystem traffic data takes place on the node level,
    hence only in case of exclusive job allocation the collected values will
    reflect the job's real traffic. In case of node sharing between jobs the reported
    \fIAccountingStorageTRES\fR.
    .RE

    .TP
    \fBAcctGatherProfileType\fR
    Identifies the plugin to be used for detailed job profiling.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    detailed data such as I/O counts, memory usage, or energy consumption for jobs
    and nodes. There are interfaces in this plugin to collect data as step start
    and completion, task start and completion, and at the account gather
    frequency. The data collected at the node level is related to jobs only in
    case of exclusive job allocation.

    Configurable values at present are:
    .RS
    .TP 20
    \fBacct_gather_profile/none\fR
    No profile data is collected.
    .TP
    \fBacct_gather_profile/hdf5\fR
    This enables the HDF5 plugin. The directory where the profile files
    are stored and which values are collected are configured in the
    acct_gather.conf file.
    .TP
    \fBacct_gather_profile/influxdb\fR
    This enables the influxdb plugin. The influxdb instance host, port, database,
    retention policy and which values are collected are configured in the
    acct_gather.conf file.
    .RE

    .TP
    \fBAllowSpecResourcesUsage\fR
    If set to 1, Slurm allows individual jobs to override node's configured
    CoreSpecCount value. For a job to take advantage of this feature,
    a command line option of \-\-core\-spec must be specified.  The default
    value for this option is 1 for Cray systems and 0 for other system types.

    .TP
    \fBAuthAltTypes\fR
    Command separated list of alternative authentication plugins that the slurmctld
    will permit for communication.

    .TP
    \fBAuthInfo\fR
    Additional information to be used for authentication of communications
    between the Slurm daemons (slurmctld and slurmd) and the Slurm
    clients.  The interpretation of this option is specific to the
    configured \fBAuthType\fR.
    Multiple options may be specified in a comma delimited list.
    If not specified, the default authentication information will be used.
    .RS
    .TP 14
    Credential lifetime, in seconds (e.g. "ttl=300").
    The default value is dependent upon the MUNGE installation, but is typically
    300 seconds.
    .RE

    .TP
    \fBAuthType\fR
    The authentication method for communications between Slurm
    components.
    Acceptable values at present include "auth/munge" and "auth/none".
    The default value is "auth/munge".
    "auth/none" includes the UID in each communication, but it is not verified.
    This may be fine for testing purposes, but
    \fBdo not use "auth/none" if you desire any security\fR.
    "auth/munge" indicates that MUNGE is to be used.
    (See "https://dun.github.io/munge/" for more information).
    All Slurm daemons and commands must be terminated prior to changing
    the value of \fBAuthType\fR and later restarted.

    .TP
    \fBBackupAddr\fR
    Defunct option, see \fBSlurmctldHost\fR.

    .TP
    \fBBackupController\fR
    Defunct option, see \fBSlurmctldHost\fR.

    The backup controller recovers state information from the
    \fBStateSaveLocation\fR directory, which must be readable and writable from both
    the primary and backup controllers.
    While not essential, it is recommended that you specify a backup controller.
    See  the \fBRELOCATING CONTROLLERS\fR section if you change this.

    .TP
    \fBBatchStartTimeout\fR
    The maximum time (in seconds) that a batch job is permitted for
    launching before being considered missing and releasing the
    allocation. The default value is 10 (seconds). Larger values may be
    required if more time is required to execute the \fBProlog\fR, load
    user environment variables (for Moab spawned jobs), or if the slurmd
    daemon gets paged from memory.
    .br
    .br
    \fBNote\fR: The test for a job being successfully launched is only performed when
    the Slurm daemon on the compute node registers state with the slurmctld daemon
    on the head node, which happens fairly rarely.
    Therefore a job will not necessarily be terminated if its start time exceeds
    \fBBatchStartTimeout\fR.
    This configuration parameter is also applied to launch tasks and avoid aborting
    \fBsrun\fR commands due to long running \fBProlog\fR scripts.

    .TP
    to take effect. Supported values presently include:
    .RS
    .TP
    \fBcheckpoint/none\fR
    no checkpoint support (default)
    .TP
    \fBcheckpoint/ompi\fR
    OpenMPI (version 1.3 or higher)
    .RE

    .TP
    \fBCliFilterPlugins\fR
    A comma delimited list of command line interface option filter/modification
    plugins. The specified plugins will be executed in the order listed.
    These are intended to be site\-specific plugins which can be used to set
    default job parameters and/or logging events.
    No cli_filter plugins are used by default.

    .TP
    \fBClusterName\fR
    The name by which this Slurm managed cluster is known in the
    accounting database.  This is needed distinguish accounting records
    when multiple clusters report to the same database. Because of limitations
    in some databases, any upper case letters in the name will be silently mapped
    to lower case. In order to avoid confusion, it is recommended that the name
    be lower case.

    .TP
    \fBCommunicationParameters\fR
    Comma separated options identifying communication options.
    .RS
    .TP 15
    \fBCheckGhalQuiesce\fR
    Used specifically on a Cray using an Aries Ghal interconnect.  This will check
    to see if the system is quiescing when sending a message, and if so, we wait
    until it is done before sending.
    .TP
    \fBNoAddrCache\fR By default, Slurm will cache a node's network address after
    successfully establishing the node's network address. This option disables the
    cache and Slurm will look up the node's network address each time a connection
    is made. This is useful, for example, in a cloud environment where the node
    addresses come and go out of DNS.
    .TP
    \fBNoCtldInAddrAny\fR
    Used to directly bind to the address of what the node resolves to running
    the slurmctld instead of binding messages to any address on the node,
    which is the default.
    .TP
    \fBNoInAddrAny\fR
    Used to directly bind to the address of what the node resolves to instead
    of binding messages to any address on the node which is the default.
    This option is for all daemons/clients except for the slurmctld.
    In that case, setting \fBKillWait\fR to a small value may be beneficial.
    The default value of \fBCompleteWait\fR is zero seconds.
    The value may not exceed 65533.

    .TP
    \fBControlAddr\fR
    Defunct option, see \fBSlurmctldHost\fR.

    .TP
    \fBControlMachine\fR
    Defunct option, see \fBSlurmctldHost\fR.

    .TP
    \fBCoreSpecPlugin\fR
    Identifies the plugins to be used for enforcement of core specialization.
    The slurmd daemon must be restarted for a change in CoreSpecPlugin
    to take effect.
    Acceptable values at present include:
    .RS
    .TP 20
    \fBcore_spec/cray_aries\fR
    used only for Cray systems
    .TP
    \fBcore_spec/none\fR
    used for all other system types
    .RE

    .TP
    \fBCpuFreqDef\fR
    Default CPU frequency value or frequency governor to use when running a
    job step if it has not been explicitly set with the \-\-cpu\-freq option.
    Acceptable values at present include a numeric value (frequency in kilohertz)
    or one of the following governors:
    .RS
    .TP 14
    \fBConservative\fR
    attempts to use the Conservative CPU governor
    .TP
    \fBOnDemand\fR
    attempts to use the OnDemand CPU governor
    .TP
    \fBPerformance\fR
    attempts to use the Performance CPU governor
    .TP
    \fBPowerSave\fR
    attempts to use the PowerSave CPU governor
    .RE
    There is no default value. If unset, no attempt to set the governor is
    made if the \-\-cpu\-freq option has not been set.

    .TP
    \fBCpuFreqGovernors\fR
    \fBPowerSave\fR
    attempts to use the PowerSave CPU governor
    .TP
    \fBUserSpace\fR
    attempts to use the UserSpace CPU governor (a default value)
    .RE
    The default is OnDemand, Performance and UserSpace.
    .TP
    \fBCredType\fR
    The cryptographic signature tool to be used in the creation of
    job step credentials.
    The slurmctld daemon must be restarted for a change in \fBCredType\fR
    to take effect.
    Acceptable values at present include "cred/munge".
    The default value is "cred/munge" and is the recommended.

    .TP
    \fBDebugFlags\fR
    Defines specific subsystems which should provide more detailed event logging.
    Multiple subsystems can be specified with comma separators.
    Most DebugFlags will result in verbose logging for the identified subsystems
    and could impact performance.
    Valid subsystems available today (with more to come) include:
    .RS
    .TP 17
    \fBAccrue\fR
    Accrue counters accounting details
    .TP
    \fBAgent\fR
    RPC agents (outgoing RPCs from Slurm daemons)
    .TP
    \fBBackfill\fR
    Backfill scheduler details
    .TP
    \fBBackfillMap\fR
    Backfill scheduler to log a very verbose map of reserved resources through
    time. Combine with \fBBackfill\fR for a verbose and complete view of the
    backfill scheduler's work.
    .TP
    \fBBurstBuffer\fR
    Burst Buffer plugin
    .TP
    \fBCPU_Bind\fR
    CPU binding details for jobs and steps
    .TP
    \fBCpuFrequency\fR
    Cpu frequency details for jobs and steps using the \-\-cpu\-freq option.
    .TP
    \fBElasticsearch\fR
    Elasticsearch debug info
    .TP
    \fBEnergy\fR
    \fBHeteroJobs\fR
    Heterogeneous job details
    .TP
    \fBGang\fR
    Gang scheduling details
    .TP
    \fBJobContainer\fR
    Job container plugin details
    .TP
    \fBLicense\fR
    License management details
    .TP
    \fBNodeFeatures\fR
    Node Features plugin debug info
    .TP
    \fBNO_CONF_HASH\fR
    Do not log when the slurm.conf files differs between Slurm daemons
    .TP
    \fBPower\fR
    Power management plugin
    .TP
    \fBPowerSave\fR
    Power save (suspend/resume programs) details
    .TP
    \fBPriority\fR
    Job prioritization
    .TP
    \fBProfile\fR
    AcctGatherProfile plugins details
    .TP
    \fBProtocol\fR
    Communication protocol details
    .TP
    \fBReservation\fR
    Advanced reservations
    .TP
    \fBRoute\fR
    Message forwarding and message aggregation debug info
    .TP
    \fBSelectType\fR
    Resource selection plugin
    .TP
    \fBSteps\fR
    Slurmctld resource allocation for job steps
    .TP
    \fBSwitch\fR
    Switch plugin
    .TP
    \fBTimeCray\fR
    Timing of Cray APIs
    .TP
    \fBTRESNode\fR
    .TP
    \fBDefMemPerCPU\fR
    Default real memory size available per allocated CPU in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBDefMemPerCPU\fR would generally be used if individual processors
    are allocated to jobs (\fBSelectType=select/cons_res\fR or
    \fBSelectType=select/cons_tres\fR).
    The default value is 0 (unlimited).
    Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
    mutually exclusive.

    .TP
    \fBDefMemPerGPU\fR
    Default real memory size available per allocated GPU in megabytes.
    The default value is 0 (unlimited).
    Also see \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
    mutually exclusive.

    .TP
    \fBDefMemPerNode\fR
    Default real memory size available per allocated node in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBDefMemPerNode\fR would generally be used if whole nodes
    are allocated to jobs (\fBSelectType=select/linear\fR) and
    resources are over\-subscribed (\fBOverSubscribe=yes\fR or
    \fBOverSubscribe=force\fR).
    The default value is 0 (unlimited).
    Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
    mutually exclusive.

    .TP
    \fBDefaultStorageHost\fR
    The default name of the machine hosting the accounting storage and
    job completion databases.
    Only used for database type storage plugins and when the
    \fBAccountingStorageHost\fR and \fBJobCompHost\fR have not been
    defined.

    .TP
    \fBDefaultStorageLoc\fR
    The fully qualified file name where accounting records and/or job
    completion records are written when the \fBDefaultStorageType\fR is
    "filetxt".
    Also see \fBAccountingStorageLoc\fR and \fBJobCompLoc\fR.

    .TP
    \fBDefaultStoragePass\fR
    The password used to gain access to the database to store the
    accounting and job completion data.
    The value "filetxt" indicates that records will be written to a file.
    The value "mysql" indicates that accounting records will be written to a MySQL
    or MariaDB database.
    The default value is "none", which means that records are not maintained.
    Also see \fBAccountingStorageType\fR and \fBJobCompType\fR.

    .TP
    \fBDefaultStorageUser\fR
    The user account for accessing the accounting storage and/or job
    completion database.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBAccountingStorageUser\fR and \fBJobCompUser\fR.

    .TP
    \fBDisableRootJobs\fR
    If set to "YES" then user root will be prevented from running any jobs.
    The default value is "NO", meaning user root will be able to execute jobs.
    \fBDisableRootJobs\fR may also be set by partition.

    .TP
    \fBEioTimeout\fR
    The number of seconds srun waits for slurmstepd to close the TCP/IP
    connection used to relay data between the user application and srun
    when the user application terminates. The default value is 60 seconds.
    May not exceed 65533.

    .TP
    \fBEnforcePartLimits\fR
    If set to "ALL" then jobs which exceed a partition's size and/or
    time limits will be rejected at submission time. If job is submitted to
    multiple partitions, the job must satisfy the limits on all the requested
    partitions. If set to "NO" then the job will be accepted and remain queued
    until the partition limits are altered(Time and Node Limits).
    If set to "ANY" a job must satisfy any of the requested partitions
    to be submitted. The default value is "NO".
    NOTE: If set, then a job's QOS can not be used to exceed partition limits.
    NOTE: The partition limits being considered are it's configured MaxMemPerCPU,
    MaxMemPerNode, MinNodes, MaxNodes, MaxTime, AllocNodes, AllowAccounts,
    AllowGroups, AllowQOS, and QOS usage threshold.

    .TP
    \fBEpilog\fR
    Fully qualified pathname of a script to execute as user root on every
    node when a user's job completes (e.g. "/usr/local/slurm/epilog"). A
    glob pattern (See \fBglob\fR (7)) may also be used to run more than
    one epilog script (e.g. "/etc/slurm/epilog.d/*"). The Epilog script
    or scripts may be used to purge files, disable user login, etc.
    By default there is no epilog.
    See \fBProlog and Epilog Scripts\fR for more information.

    .TP
    \fBEpilogMsgTime\fR
    The program executes as SlurmUser, which gives it permission to drain
    nodes and requeue the job if a failure occurs (See scontrol(1)).
    Exactly what the program does and how it accomplishes this is completely at
    the discretion of the system administrator.
    Information about the job being initiated, it's allocated nodes, etc. are
    passed to the program using environment variables.
    See \fBProlog and Epilog Scripts\fR for more information.

    .TP
    \fBExtSensorsFreq\fR
    The external sensors plugin sampling interval.
    If \fBExtSensorsType=ext_sensors/none\fR, this parameter is ignored.
    For all other values of \fBExtSensorsType\fR, this parameter is the number
    of seconds between external sensors samples for hardware components (nodes,
    switches, etc.) The default value is zero. This value disables external
    sensors sampling. Note: This parameter does not affect external sensors
    data collection for jobs/steps.

    .TP
    \fBExtSensorsType\fR
    Identifies the plugin to be used for external sensors data collection.
    Slurmctld calls this plugin to collect external sensors data for jobs/steps
    and hardware components. In case of node sharing between jobs the reported
    values per job/step (through sstat or sacct) may not be accurate.  See also
    "man ext_sensors.conf".

    Configurable values at present are:
    .RS
    .TP 20
    \fBext_sensors/none\fR
    No external sensors data is collected.
    .TP
    \fBext_sensors/rrd\fR
    External sensors data is collected from the RRD database.
    .RE

    .TP
    \fBFairShareDampeningFactor\fR
    Dampen the effect of exceeding a user or group's fair share of allocated
    resources. Higher values will provides greater ability to differentiate
    between exceeding the fair share at high levels (e.g. a value of 1 results
    in almost no difference between overconsumption by a factor of 10 and 100,
    while a value of 5 will result in a significant difference in priority).
    The default value is 1.

    .TP
    \fBFederationParameters\fR
    Used to define federation options. Multiple options may be comma separated.

    .RS
    .TP
    \fBfed_display\fR
    Also see \fBMaxJobId\fR

    .TP
    \fBGetEnvTimeout\fR
    Used for Moab scheduled jobs only. Controls how long job should wait
    in seconds for loading the user's environment before attempting to
    load it from a cache file. Applies when the srun or sbatch
    \fI\-\-get\-user\-env\fR option is used. If set to 0 then always load
    the user's environment from the cache file.
    The default value is 2 seconds.

    .TP
    \fBGresTypes\fR
    A comma delimited list of generic resources to be managed (e.g.
    \fIGresTypes=gpu,mps\fR).
    These resources may have an associated GRES plugin of the same name providing
    additional functionality.
    No generic resources are managed by default.
    Ensure this parameter is consistent across all nodes in the cluster for
    proper operation.
    The slurmctld daemon must be restarted for changes to this parameter to become
    effective.

    .TP
    \fBGroupUpdateForce\fR
    If set to a non\-zero value, then information about which users are members
    of groups allowed to use a partition will be updated periodically, even when
    there have been no changes to the /etc/group file.
    If set to zero, group member information will be updated only after the
    /etc/group file is updated.
    The default value is 1.
    Also see the \fBGroupUpdateTime\fR parameter.

    .TP
    \fBGroupUpdateTime\fR
    Controls how frequently information about which users are members of
    groups allowed to use a partition will be updated, and how long user
    group membership lists will be cached.
    The time interval is given in seconds with a default value of 600 seconds.
    A value of zero will prevent periodic updating of group membership information.
    Also see the \fBGroupUpdateForce\fR parameter.

    .TP
    \fBGpuFreqDef\fR=[<\fItype\fR]=\fIvalue\fR>[,<\fItype\fR=\fIvalue\fR>]
    Default GPU frequency to use when running a job step if it
    has not been explicitly set using the \-\-gpu\-freq option.
    This option can be used to independently configure the GPU and its memory
    frequencies. Defaults to "high,memory=high".
    After the job is completed, the frequencies of all affected GPUs will be reset
    to the highest possible values.
    In some cases, system power caps may override the requested values.
    The field \fItype\fR can be "memory".
    \fBmedium\fR
    attempts to set a frequency in the middle of the available range.
    .TP
    \fBhigh\fR
    the highest available frequency.
    .TP
    \fBhighm1\fR
    (high minus one) will select the next highest available frequency.
    .RE

    .TP
    \fBHealthCheckInterval\fR
    The interval in seconds between executions of \fBHealthCheckProgram\fR.
    The default value is zero, which disables execution.

    .TP
    \fBHealthCheckNodeState\fR
    Identify what node states should execute the \fBHealthCheckProgram\fR.
    Multiple state values may be specified with a comma separator.
    The default value is ANY to execute on nodes in any state.
    .RS
    .TP 12
    \fBALLOC\fR
    Run on nodes in the ALLOC state (all CPUs allocated).
    .TP
    \fBANY\fR
    Run on nodes in any state.
    .TP
    \fBCYCLE\fR
    Rather than running the health check program on all nodes at the same time,
    cycle through running on all compute nodes through the course of the
    \fBHealthCheckInterval\fR. May be combined with the various node state
    options.
    .TP
    \fBIDLE\fR
    Run on nodes in the IDLE state.
    .TP
    \fBMIXED\fR
    Run on nodes in the MIXED state (some CPUs idle and other CPUs allocated).
    .RE

    .TP
    \fBHealthCheckProgram\fR
    Fully qualified pathname of a script to execute as user root periodically
    on all compute nodes that are \fBnot\fR in the NOT_RESPONDING state. This
    program may be used to verify the node is fully operational and DRAIN the node
    or send email if a problem is detected.
    Any action to be taken must be explicitly performed by the program
    (e.g. execute
    "scontrol update NodeName=foo State=drain Reason=tmp_file_system_full"
    to drain a node).
    The execution interval is controlled using the \fBHealthCheckInterval\fR
    command abnormally terminates, this will terminate its job allocation.
    This option has no effect upon batch jobs.
    When setting a value, take into consideration that a debugger using \fBsrun\fR
    to launch an application may leave the \fBsrun\fR command in a stopped state
    for extended periods of time.
    This limit is ignored for jobs running in partitions with the
    \fBRootOnly\fR flag set (the scheduler running as root will be
    responsible for the job).
    The default value is unlimited (zero) and may not exceed 65533 seconds.

    .TP
    \fBJobAcctGatherType\fR
    The job accounting mechanism type.
    Acceptable values at present include "jobacct_gather/linux" (for Linux
    systems) and is the recommended one, "jobacct_gather/cgroup" and
    "jobacct_gather/none" (no accounting data collected).
    The default value is "jobacct_gather/none".
    "jobacct_gather/cgroup" is a plugin for the Linux operating system
    that uses cgroups to collect accounting statistics. The plugin collects the
    following statistics: From the cgroup memory subsystem: memory.usage_in_bytes
    (reported as 'pages') and rss from memory.stat (reported as 'rss'). From the
    cgroup cpuacct subsystem: user cpu time and system cpu time. No value
    is provided by cgroups for virtual memory size ('vsize').
    In order to use the \fBsstat\fR tool "jobacct_gather/linux",
    or "jobacct_gather/cgroup" must be configured.
    .br
    \fBNOTE:\fR Changing this configuration parameter changes the contents of
    the messages between Slurm daemons. Any previously running job steps are
    managed by a slurmstepd daemon that will persist through the lifetime of
    that job step and not change it's communication protocol. Only change this
    configuration parameter when there are no running job steps.

    .TP
    \fBJobAcctGatherFrequency\fR
    The job accounting and profiling sampling intervals.
    The supported format is follows:
    .RS
    .TP 12
    \fBJobAcctGatherFrequency=\fR\fI<datatype>\fR\fB=\fR\fI<interval>\fR
    where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling
    interval for the jobacct_gather plugin or a
    sampling interval for a profiling type by the
    acct_gather_profile plugin. Multiple,
    comma-separated \fI<datatype>\fR=\fI<interval>\fR intervals
    may be specified. Supported datatypes are as follows:
    .RS
    .TP
    \fBtask=\fI<interval>\fR
    where \fI<interval>\fR is the task sampling interval in seconds
    for the jobacct_gather plugins and for task
    profiling by the acct_gather_profile plugin.
    .TP
    .RE
    .RE
    The default value for task sampling interval
    is 30 seconds. The default value for all other intervals is 0.
    An interval of 0 disables sampling of the specified type.
    If the task sampling interval is 0, accounting
    information is collected only at job termination (reducing Slurm
    interference with the job).
    .br
    .br
    Smaller (non\-zero) values have a greater impact upon job performance,
    but a value of 30 seconds is not likely to be noticeable for
    applications having less than 10,000 tasks.
    .br
    .br
    Users can independently override each interval on a per job basis using the
    \fB\-\-acctg\-freq\fR option when submitting the job.
    .RE

    .TP
    \fBJobAcctGatherParams\fR
    Arbitrary parameters for the job account gather plugin
    Acceptable values at present include:
    .RS
    .TP 20
    \fBNoShared\fR
    Exclude shared memory from accounting.
    .TP
    \fBUsePss\fR
    Use PSS value instead of RSS to calculate real usage of memory.
    The PSS value will be saved as RSS.
    .TP
    \fBOverMemoryKill\fR
    Kill jobs or steps that are being detected to use more memory than requested
    every time accounting information is gathered by the JobAcctGather plugin.
    This parameter will not kill a job directly, but only the step.
    See \fBMemLimitEnforce\fR for that purpose. This parameter should be used
    with caution as if jobs exceeds its memory allocation it may affect other
    processes and/or machine health.
    NOTE: It is recommended to limit memory by enabling task/cgroup
    in TaskPlugin and making use of ConstrainRAMSpace=yes cgroup.conf instead
    of using this JobAcctGather mechanism for memory enforcement, since the former
    has a lower resolution (JobAcctGatherFreq) and OOMs could happen at some point.
    .RE

    .TP
    \fBJobCheckpointDir\fR
    Specifies the default directory for storing or reading job checkpoint
    information. The data stored here is only a few thousand bytes per job
    and includes information needed to resubmit the job request, not job's
    memory image. The directory must be readable and writable by
    \fBSlurmUser\fR, but not writable by regular users. The job memory images
    database, or an url with format http://yourelasticserver:port when
    \fBJobCompType\fR is "jobcomp/elasticsearch".
    NOTE: when you specify a URL for Elasticsearch, Slurm will remove any trailing
    slashes "/" from the configured URL and append "/slurm/jobcomp", which are the
    Elasticsearch index name (slurm) and mapping (jobcomp).
    NOTE: More information is available at the Slurm web site
    ( https://slurm.schedmd.com/elasticsearch.html ).
    Also see \fBDefaultStorageLoc\fR.

    .TP
    \fBJobCompPass\fR
    The password used to gain access to the database to store the job
    completion data.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBDefaultStoragePass\fR.

    .TP
    \fBJobCompPort\fR
    The listening port of the job completion database server.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBDefaultStoragePort\fR.

    .TP
    \fBJobCompType\fR
    The job completion logging mechanism type.
    Acceptable values at present include "jobcomp/none", "jobcomp/elasticsearch",
    "jobcomp/filetxt", "jobcomp/mysql" and "jobcomp/script".
    The default value is "jobcomp/none", which means that upon job completion
    the record of the job is purged from the system.  If using the accounting
    infrastructure this plugin may not be of interest since the information
    here is redundant.
    The value "jobcomp/elasticsearch" indicates that a record of the job
    should be written to an Elasticsearch server specified by the
    \fBJobCompLoc\fR parameter.
    NOTE: More information is available at the Slurm web site
    ( https://slurm.schedmd.com/elasticsearch.html ).
    The value "jobcomp/filetxt" indicates that a record of the job should be
    written to a text file specified by the \fBJobCompLoc\fR parameter.
    The value "jobcomp/mysql" indicates that a record of the job should be
    written to a MySQL or MariaDB database specified by the \fBJobCompLoc\fR
    parameter.
    The value "jobcomp/script" indicates that a script specified by the
    \fBJobCompLoc\fR parameter is to be executed with environment variables
    indicating the job information.
    .TP
    \fBJobCompUser\fR
    The user account for accessing the job completion database.
    Only used for database type storage plugins, ignored otherwise.
    Also see \fBDefaultStorageUser\fR.

    .TP
    \fBJobContainerType\fR

    .TP
    \fBJobFileAppend\fR
    This option controls what to do if a job's output or error file
    exist when the job is started.
    If \fBJobFileAppend\fR is set to a value of 1, then append to
    the existing file.
    By default, any existing file is truncated.

    .TP
    \fBJobRequeue\fR
    This option controls the default ability for batch jobs to be requeued.
    Jobs may be requeued explicitly by a system administrator, after node
    failure, or upon preemption by a higher priority job.
    If \fBJobRequeue\fR is set to a value of 1, then batch job may be requeued
    unless explicitly disabled by the user.
    If \fBJobRequeue\fR is set to a value of 0, then batch job will not be requeued
    unless explicitly enabled by the user.
    Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR
    option to change the default behavior for individual jobs.
    The default value is 1.

    .TP
    \fBJobSubmitPlugins\fR
    A comma delimited list of job submission plugins to be used.
    The specified plugins will be executed in the order listed.
    These are intended to be site\-specific plugins which can be used to set
    default job parameters and/or logging events.
    Sample plugins available in the distribution include "all_partitions",
    "defaults", "logging", "lua", and "partition".
    For examples of use, see the Slurm code in "src/plugins/job_submit" and
    "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs.
    Slurm can be configured to use multiple job_submit plugins if desired,
    however the lua plugin will only execute one lua script named "job_submit.lua"
    located in the default script directory (typically the subdirectory "etc" of
    the installation directory).
    No job submission plugins are used by default.

    .TP
    \fBKeepAliveTime\fR
    Specifies how long sockets communications used between the srun command and its
    slurmstepd process are kept alive after disconnect.
    Longer values can be used to improve reliability of communications in the
    event of network failures.
    The default value leaves the system default value.
    The value may not exceed 65533.

    .TP
    \fBKillOnBadExit\fR
    If set to 1, a step will be terminated immediately if any task is
    crashed or aborted, as indicated by a non-zero exit code.
    With the default value of 0, if one of the processes is crashed or aborted
    \fBNodeFeaturesPlugins\fR
    Identifies the plugins to be used for support of node features which can
    change through time. For example, a node which might be booted with various
    BIOS setting. This is supported through the use of a node's active_features
    and available_features information.
    Acceptable values at present include:
    .RS
    .TP 20
    \fBnode_features/knl_cray\fR
    used only for Intel Knights Landing processors (KNL) on Cray systems
    .TP 20
    \fBnode_features/knl_generic\fR
    used for Intel Knights Landing processors (KNL) on a generic Linux system
    .RE

    .TP
    \fBLaunchParameters\fR
    Identifies options to the job launch plugin.
    Acceptable values include:
    .RS
    .TP 24
    \fBbatch_step_set_cpu_freq\fR
    Set the cpu frequency for the batch step from given \-\-cpu\-freq, or
    slurm.conf CpuFreqDef, option.  By default only steps started with srun will
    utilize the cpu freq setting options.

    NOTE: If you are using srun to launch your steps inside a batch script
    (advised) this option will create a situation where you may have multiple
    agents setting the cpu_freq as the batch step usually runs on the same
    resources one or more steps the sruns in the script will create.
    .TP 24
    \fBcray_net_exclusive\fR
    Allow jobs on a Cray Native cluster exclusive access to network resources.
    This should only be set on clusters providing exclusive access to each
    node to a single job at once, and not using parallel steps within the job,
    otherwise resources on the node can be oversubscribed.
    .TP 24
    \fBenable_nss_slurm\fR
    Permits passwd and group resolution for a job to be serviced by slurmstepd rather
    than requiring a lookup from a network based service. See
    https://slurm.schedmd.com/nss_slurm.html for more information.
    .TP 24
    \fBlustre_no_flush\fR
    If set on a Cray Native cluster, then do not flush the Lustre cache on job step
    completion. This setting will only take effect after reconfiguring, and will
    only take effect for newly launched jobs.
    .TP 24
    \fBmem_sort\fR
    Sort NUMA memory at step start. User can override this default with
    SLURM_MEM_BIND environment variable or \-\-mem\-bind=nosort command line option.
    .TP
    \fBdisable_send_gids\fR
    .RE

    .TP
    \fBLaunchType\fR
    Identifies the mechanism to be used to launch application tasks.
    Acceptable values include:
    .RS
    .TP
    \fBlaunch/slurm\fR
    The default value.
    .RE

    .TP
    \fBLicenses\fR
    Specification of licenses (or other resources available on all
    nodes of the cluster) which can be allocated to jobs.
    License names can optionally be followed by a colon
    and count with a default count of one.
    Multiple license names should be comma separated (e.g.
    "Licenses=foo:4,bar").
    Note that Slurm prevents jobs from being scheduled if their
    required license specification is not available.
    Slurm does not prevent jobs from using licenses that are
    not explicitly listed in the job submission specification.

    .TP
    \fBLogTimeFormat\fR
    Format of the timestamp in slurmctld and slurmd log files. Accepted
    values are "iso8601", "iso8601_ms", "rfc5424", "rfc5424_ms", "clock",
    "short" and "thread_id". The values ending in "_ms" differ from the ones without
    in that fractional seconds with millisecond precision are printed. The
    default value is "iso8601_ms". The "rfc5424" formats are the same as
    the "iso8601" formats except that the timezone value is also
    shown. The "clock" format shows a timestamp in microseconds retrieved
    with the C standard clock() function. The "short" format is a short
    date and time format. The "thread_id" format shows the timestamp
    in the C standard ctime() function form without the year but
    including the microseconds, the daemon's process ID and the current thread name
    and ID.

    .TP
    \fBMailDomain\fR
    Domain name to qualify usernames if email address is not explicitly given
    with the "--mail-user" option. If unset, the local MTA will need to qualify
    local address itself.

    .TP
    \fBMailProg\fR
    Fully qualified pathname to the program used to send email per user request.
    The default value is "/bin/mail" (or "/usr/bin/mail" if "/bin/mail" does not
    exist but "/usr/bin/mail" does exist).

    to ensure the slurmctld daemon does not exhaust its memory or other
    resources. Once this limit is reached, requests to submit additional
    jobs will fail. The default value is 10000 jobs.
    NOTE: Each task of a job array counts as one job even though they will not
    occupy separate job records until modified or initiated.
    Performance can suffer with more than a few hundred thousand jobs.
    Setting per MaxSubmitJobs per user is generally valuable to prevent a single
    user from filling the system with jobs.
    This is accomplished using Slurm's database and configuring enforcement of
    resource limits.
    This value may not be reset via "scontrol reconfig".
    It only takes effect upon restart of the slurmctld daemon.

    .TP
    \fBMaxJobId\fR
    The maximum job id to be used for jobs submitted to Slurm without a specific
    requested value. Job ids are unsigned 32bit integers with the first 26 bits
    reserved for local job ids and the remaining 6 bits reserved for a cluster id
    to identify a federated job's origin. The maximun allowed local job id is
    67,108,863 (0x3FFFFFF). The default value is 67,043,328 (0x03ff0000).
    \fBMaxJobId\fR only applies to the local job id and not the federated job id.
    Job id values generated will be incremented by 1 for each subsequent job. Once
    \fBMaxJobId\fR is reached, the next job will be assigned \fBFirstJobId\fR.
    Federated jobs will always have a job ID of 67,108,865 or higher.
    Also see \fBFirstJobId\fR.

    .TP
    \fBMaxMemPerCPU\fR
    Maximum real memory size available per allocated CPU in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBMaxMemPerCPU\fR would generally be used if individual processors
    are allocated to jobs (\fBSelectType=select/cons_res\fR or
    \fBSelectType=select/cons_tres\fR).
    The default value is 0 (unlimited).
    Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerNode\fR.
    \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.

    NOTE: If a job specifies a memory per CPU limit that exceeds this system limit,
    that job's count of CPUs per task will automatically be increased. This may
    result in the job failing due to CPU count limits.

    .TP
    \fBMaxMemPerNode\fR
    Maximum real memory size available per allocated node in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBMaxMemPerNode\fR would generally be used if whole nodes
    are allocated to jobs (\fBSelectType=select/linear\fR) and
    resources are over\-subscribed (\fBOverSubscribe=yes\fR or
    \fBOverSubscribe=force\fR).
    The default value is 0 (unlimited).
    Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
    \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.
    \fBMCSParameters\fR
    MCS = Multi-Category Security
    MCS Plugin Parameters.
    The supported parameters are specific to the \fBMCSPlugin\fR.
    Changes to this value take effect when the Slurm daemons are reconfigured.
    More information about MCS is available here
    fR(8) manual.

    .TP
    \fBPowerParameters\fR
    System power management parameters.
    The supported parameters are specific to the \fBPowerPlugin\fR.
    Changes to this value take effect when the Slurm daemons are reconfigured.
    .TP
    \fBcapmc_path=\fR
    Specifies the absolute path of the capmc command.
    The default value is "/opt/cray/capmc/default/bin/capmc".
    Supported by the power/cray_aries plugin.
    .TP
    \fBcap_watts=#\fR
    Specifies the total power limit to be established across all compute nodes
    managed by Slurm.
    A value of 0 sets every compute node to have an unlimited cap.
    The default value is 0.
    Supported by the power/cray_aries plugin.
    .TP
    \fBdecrease_rate=#\fR
    Specifies the maximum rate of change in the power cap for a node where the
    actual power usage is below the power cap by an amount greater than
    \fBlower_threshold\fR (see below).
    Value represents a percentage of the difference between a node's minimum and
    maximum power consumption.
    The default value is 50 percent.
    Supported by the power/cray_aries plugin.
    .TP
    \fBget_timeout=#\fR
    Amount of time allowed to get power state information in milliseconds.
    The default value is 5,000 milliseconds or 5 seconds.
    Supported by the power/cray_aries plugin and represents the time allowed for the
    capmc command to respond to various "get" options.
    .TP
    \fBincrease_rate=#\fR
    Specifies the maximum rate of change in the power cap for a node where the
    actual power usage is within \fBupper_threshold\fR (see below) of the power cap.
    Value represents a percentage of the difference between a node's minimum and
    maximum power consumption.
    The default value is 20 percent.
    Supported by the power/cray_aries plugin.
    .TP
    \fBjob_level\fR
    All nodes associated with every job will have the same power cap, to the extent
    possible.
    Also see the \-\-power=level option on the job submission commands.
    .TP
    \fBjob_no_level\fR
    Disable the user's ability to set every node associated with a job to the same
    power cap.
    Each node will have it's power cap set independently.
    This disables the \-\-power=level option on the job submission commands.
    .TP
    \fBlower_threshold=#\fR
    Specify a lower power consumption threshold.
    If a node's current power consumption is below this percentage of its current
    cap, then its power cap will be reduced.
    The default value is 90 percent.
    capmc command to respond to various "set" options.
    .TP
    \fBset_watts=#\fR
    Specifies the power limit to be set on every compute nodes managed by Slurm.
    Every node gets this same power cap and there is no variation through time
    based upon actual power usage on the node.
    Supported by the power/cray_aries plugin.
    .TP
    \fBupper_threshold=#\fR
    Specify an upper power consumption threshold.
    If a node's current power consumption is above this percentage of its current
    cap, then its power cap will be increased to the extent possible.
    The default value is 95 percent.
    Supported by the power/cray_aries plugin.
    .RE

    .TP
    \fBPowerPlugin\fR
    Identifies the plugin used for system power management.
    Currently supported plugins include:
    \fBcray_aries\fR and
    \fBnone\fR.
    Changes to this value require restarting Slurm daemons to take effect.
    More information about system power management is available here
    fR(1) will run the user's default shell when
    a command to execute is not specified on the \fBsalloc\fR command line.
    If \fBSallocDefaultCommand\fR is specified, \fBsalloc\fR will instead
    run the configured command. The command is passed to '/bin/sh \-c', so
    shell metacharacters are allowed, and commands with multiple arguments
    should be quoted. For instance:

    .nf
        SallocDefaultCommand = "$SHELL"
    .fi

    would run the shell in the user's $SHELL environment variable.
    and

    .nf
        SallocDefaultCommand = "srun \-n1 \-N1 \-\-mem\-per\-cpu=0 \-\-pty \-\-preserve\-env \-\-mpi=none $SHELL"
    .fi

    would run spawn the user's default shell on the allocated resources, but not
    consume any of the CPU or memory resources, configure it as a pseudo\-terminal,
    and preserve all of the job's environment variables (i.e. and not over\-write
    them with the job step's allocation information).

    For systems with generic resources (GRES) defined, the \fBSallocDefaultCommand\fR
    value should explicitly specify a zero count for the configured GRES.
    Failure to do so will result in the launched shell consuming those GRES and
    preventing subsequent srun commands from using them.
    For example, on Cray systems add "\-\-gres=craynetwork:0" as shown below:
    .nf
        SallocDefaultCommand = "srun \-n1 \-N1 \-\-mem\-per\-cpu=0 \-\-gres=craynetwork:0 \-\-pty \-\-preserve\-env \-\-mpi=none $SHELL"
    .fi

    For systems with TaskPlugin set, adding an option of "\-\-cpu\-bind=no" is
    recommended if the default shell should have access to all of the CPUs
    allocated to the job on that node, otherwise the shell may be limited to a
    single cpu or core.

    .TP
    \fBSbcastParameters\fR
    Controls sbcast command behavior. Multiple options can be specified in a comma
    separated list.
    Supported values include:
    .RS
    .TP 15
    \fBDestDir=\fR
    Destination directory for file being broadcast to allocated compute nodes.
    Default value is current working directory.
    .TP
    \fBCompression=\fR
    rejected.
    .TP
    \fBassoc_limit_stop\fR
    If set and a job cannot start due to association limits, then do not attempt
    to initiate any lower priority jobs in that partition. Setting this can
    decrease system throughput and utilization, but avoid potentially starving larger
    jobs by preventing them from launching indefinitely.
    .TP
    \fBbatch_sched_delay=#\fR
    How long, in seconds, the scheduling of batch jobs can be delayed.
    This can be useful in a high\-throughput environment in which batch jobs are
    submitted at a very high rate (i.e. using the sbatch command) and one wishes
    to reduce the overhead of attempting to schedule each job at submit time.
    The default value is 3 seconds.
    .TP
    \fBbb_array_stage_cnt=#\fR
    Number of tasks from a job array that should be available for burst buffer
    resource allocation. Higher values will increase the system overhead as each
    task from the job array will be moved to it's own job record in memory, so
    relatively small values are generally recommended.
    The default value is 10.
    .TP
    \fBbf_busy_nodes\fR
    When selecting resources for pending jobs to reserve for future execution
    (i.e. the job can not be started immediately), then preferentially select
    nodes that are in use.
    This will tend to leave currently idle resources available for backfilling
    longer running jobs, but may result in allocations having less than optimal
    network topology.
    This option is currently only supported by the select/cons_res and
    select/cons_tres plugins (or select/cray_aries with SelectTypeParameters set to
    "OTHER_CONS_RES" or "OTHER_CONS_TRES", which layers the select/cray_aries plugin
    over the select/cons_res or select/cons_tres plugin respectively).
    .TP
    \fBbf_continue\fR
    The backfill scheduler periodically releases locks in order to permit other
    operations to proceed rather than blocking all activity for what could be an
    extended period of time.
    Setting this option will cause the backfill scheduler to continue processing
    pending jobs from its original job list after releasing locks even if job
    or node state changes.
    This can result in lower priority jobs being backfill scheduled instead
    of newly arrived higher priority jobs, but will permit more queued jobs to be
    considered for backfill scheduling.
    .TP
    \fBbf_hetjob_immediate\fR
    Instruct the backfill scheduler to attempt to start a heterogeneous job as
    soon as all of its components are determined able to do so. Otherwise, the
    backfill scheduler will delay heterogeneous jobs initiation attempts until
    after the rest of the queue has been processed. This delay may result in lower
    priority jobs being allocated resources, which could delay the initiation of
    the heterogeneous job due to account and/or QOS limits being reached. This
    the \fBPriorityTier\fR layer and for the \fBPriority\fR resulting from the
    priority/multifactor plugin calculations. When enabled, if any heterogeneous job
    requested an advanced reservation, then all of that job's components will be
    treated as if they had requested an advanced reservation (and get
    preferential treatment in scheduling).

    Note that this operation does not update the \fBPriority\fR values of the
    heterogeneous job components, only their order within the list, so the output of
    the sprio command will not be effected.

    Heterogeneous jobs have special scheduling properties: they are only scheduled
    by the backfill scheduling plugin, each of their components is considered
    separately when reserving resources (and might have different \fBPriorityTier\fR
    or different \fBPriority\fR values), and no heterogeneous job component is
    actually allocated resources until all if its components can be initiated.
    This may imply potential scheduling deadlock scenarios because components
    from different heterogeneous jobs can start reserving resources in an
    interleaved fashion (not consecutively), but none of the jobs can reserve
    resources for all components and start. Enabling this option can help to
    mitigate this problem. By default, this option is disabled.
    .TP
    \fBbf_ignore_newly_avail_nodes\fR
    If set, then only resources available at the beginning of a backfill cycle
    will be considered for use. Otherwise resources made available during that
    backfill cycle (during a yield with bf_continue set) may be used for lower
    priority jobs, delaying the initiation of higher priority jobs.
    Disabled by default.
    .TP
    \fBbf_interval=#\fR
    The number of seconds between backfill iterations.
    Higher values result in less overhead and better responsiveness.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Default: 30, Min: 1, Max: 10800 (3h).

    .TP
    \fBbf_job_part_count_reserve=#\fR
    The backfill scheduling logic will reserve resources for the specified count
    of highest priority jobs in each partition.
    For example, bf_job_part_count_reserve=10 will cause the backfill scheduler to
    reserve resources for the ten highest priority jobs in each partition.
    Any lower priority job that can be started using currently available resources
    and not adversely impact the expected start time of these higher priority jobs
    will be started by the backfill scheduler
    The default value is zero, which will reserve resources for any pending job
    and delay initiation of lower priority jobs.
    Also see bf_min_age_reserve and bf_min_prio_reserve.
    Default: 0, Min: 0, Max: 100000.

    .TP
    \fBbf_max_job_array_resv=#\fR
    The maximum number of tasks from a job array for which the backfill scheduler
    will reserve resources in the future.
    \fBbf_max_job_assoc=#\fR
    The maximum number of jobs per user association to attempt starting with the
    backfill scheduler.
    This setting is similar to \fBbf_max_job_user\fR but is handy if a user
    has multiple associations equating to basically different users.
    One can set this limit to prevent users from flooding the backfill
    queue with jobs that cannot start and that prevent jobs from other users
    to start.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Also see the \fBbf_max_job_user\fR \fBbf_max_job_part\fR, \fBbf_max_job_test\fR
    and \fBbf_max_job_user_part=#\fR options.
    Set \fBbf_max_job_test\fR to a value much higher than \fBbf_max_job_assoc\fR.
    Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
    .TP
    \fBbf_max_job_part=#\fR
    The maximum number of jobs per partition to attempt starting with the backfill
    scheduler. This can be especially helpful for systems with large numbers of
    partitions and jobs.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Also see the \fBpartition_job_depth\fR and \fBbf_max_job_test\fR options.
    Set \fBbf_max_job_test\fR to a value much higher than \fBbf_max_job_part\fR.
    Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
    .TP
    \fBbf_max_job_start=#\fR
    The maximum number of jobs which can be initiated in a single iteration
    of the backfill scheduler.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Default: 0 (no limit), Min: 0, Max: 10000.
    .TP
    \fBbf_max_job_test=#\fR
    The maximum number of jobs to attempt backfill scheduling for
    (i.e. the queue depth).
    Higher values result in more overhead and less responsiveness.
    Until an attempt is made to backfill schedule a job, its expected
    initiation time value will not be set.
    In the case of large clusters, configuring a relatively small value may be
    desirable.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Default: 100, Min: 1, Max: 1,000,000.
    .TP
    \fBbf_max_job_user=#\fR
    The maximum number of jobs per user to attempt starting with the backfill
    scheduler for ALL partitions.
    One can set this limit to prevent users from flooding the backfill
    queue with jobs that cannot start and that prevent jobs from other users
    to start.  This is similar to the MAXIJOB limit in Maui.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Also see the \fBbf_max_job_part\fR, \fBbf_max_job_test\fR and
    \fBbf_max_job_user_part=#\fR options.
    Set \fBbf_max_job_test\fR to a value much higher than \fBbf_max_job_user\fR.
    Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
    .TP
    Default: bf_interval value (def. 30 sec), Min: 1, Max: 3600 (1h).
    NOTE: If bf_interval is short and bf_max_time is large, this may cause locks
    to be acquired too frequently and starve out other serviced RPCs. It's
    advisable if using this parameter to set max_rpc_cnt high enough that
    scheduling isn't always disabled, and low enough that the interactive
    workload can get through in a reasonable period of time. max_rpc_cnt needs to
    be below 256 (the default RPC thread limit). Running around the middle (150)
    may give you good results.
    NOTE: When increasing the amount of time spent in the backfill scheduling cycle,
    Slurm can be prevented from responding to client requests in a timely manner.
    To address this you can use \fBmax_rpc_cnt\fR to specify a number of queued RPCs
    before the scheduler stops to respond to these requests.
    .TP
    \fBbf_min_age_reserve=#\fR
    The backfill and main scheduling logic will not reserve resources for pending
    jobs until they have been pending and runnable for at least the specified
    number of seconds.
    In addition, jobs waiting for less than the specified number of seconds will
    not prevent a newly submitted job from starting immediately, even if the newly
    submitted job has a lower priority.
    This can be valuable if jobs lack time limits or all time limits have the same
    value.
    The default value is zero, which will reserve resources for any pending job
    and delay initiation of lower priority jobs.
    Also see bf_job_part_count_reserve and bf_min_prio_reserve.
    Default: 0, Min: 0, Max: 2592000 (30 days).
    .TP
    \fBbf_min_prio_reserve=#\fR
    The backfill and main scheduling logic will not reserve resources for pending
    jobs unless they have a priority equal to or higher than the specified value.
    In addition, jobs with a lower priority will not prevent a newly submitted job
    from starting immediately, even if the newly submitted job has a lower priority.
    This can be valuable if one wished to maximum system utilization without regard
    for job priority below a certain threshold.
    The default value is zero, which will reserve resources for any pending job
    and delay initiation of lower priority jobs.
    Also see bf_job_part_count_reserve and bf_min_age_reserve.
    Default: 0, Min: 0, Max: 2^63.
    .TP
    \fBbf_resolution=#\fR
    The number of seconds in the resolution of data maintained about when jobs
    begin and end.
    Higher values result in less overhead and better responsiveness.
    This option applies only to \fBSchedulerType=sched/backfill\fR.
    Default: 60, Min: 1, Max: 3600 (1 hour).
    .TP
    \fBbf_window=#\fR
    The number of minutes into the future to look when considering jobs to schedule.
    Higher values result in more overhead and less responsiveness.
    A value at least as long as the highest allowed time limit is generally
    advisable to prevent job starvation.
    In order to limit the amount of data managed by the backfill scheduler,
    The value is specified in units of seconds. For example, a value of 60 will
    cause the backfill scheduler on the first iteration to identify the job ending
    soonest and determine if the pending job can be started after that job plus
    all other jobs expected to end within 30 seconds (default initial value) of the
    first job. On the next iteration, the pending job will be evaluated for
    starting after the next job expected to end plus all jobs ending within
    90 seconds of that time (30 second default, plus the 60 second option value).
    The third iteration will have a 150 second window and the fourth 210 seconds.
    Without this option, the time windows will double on each iteration and thus
    be 30, 60, 120, 240 seconds, etc. The use of bf_window_linear is not recommended
    with more than a few hundred simultaneously executing jobs.
    .TP
    \fBbf_yield_interval=#\fR
    The backfill scheduler will periodically relinquish locks in order for other
    pending operations to take place.
    This specifies the times when the locks are relinquished in microseconds.
    Smaller values may be helpful for high throughput computing when used in
    conjunction with the \fBbf_continue\fR option.
    Also see the \fBbf_yield_sleep\fR option.
    Default: 2,000,000 (2 sec), Min: 1, Max: 10,000,000 (10 sec).
    .TP
    \fBbf_yield_sleep=#\fR
    The backfill scheduler will periodically relinquish locks in order for other
    pending operations to take place.
    This specifies the length of time for which the locks are relinquished in
    microseconds.
    Also see the \fBbf_yield_interval\fR option.
    Default: 500,000 (0.5 sec), Min: 1, Max: 10,000,000 (10 sec).
    .TP
    \fBbuild_queue_timeout=#\fR
    Defines the maximum time that can be devoted to building a queue of jobs to
    be tested for scheduling.
    If the system has a huge number of jobs with dependencies, just building the
    job queue can take so much time as to adversely impact overall system
    performance and this parameter can be adjusted as needed.
    The default value is 2,000,000 microseconds (2 seconds).
    .TP
    \fBdefault_queue_depth=#\fR
    The default number of jobs to attempt scheduling (i.e. the queue depth) when a
    running job completes or other routine actions occur, however the frequency
    with which the scheduler is run may be limited by using the \fBdefer\fR or
    \fBsched_min_interval\fR parameters described below.
    The full queue will be tested on a less frequent basis as defined by the
    \fBsched_interval\fR option described below. The default value is 100.
    See the \fBpartition_job_depth\fR option to limit depth by partition.
    .TP
    \fBdefer\fR
    Setting this option will avoid attempting to schedule each job
    individually at job submit time, but defer it until a later time when
    scheduling multiple jobs simultaneously may be possible.
    This option may improve system responsiveness when large numbers of jobs
    (many hundreds) are submitted at the same time, but it will delay the
    Users can override the default by using a suffix of "M" for megabytes.
    .TP
    \fBdisable_job_shrink\fR
    Deny user requests to shrink the side of running jobs. (However, running jobs
    may still shrink due to node failure if the \-\-no-kill option was set.)
    .TP
    \fBdisable_hetero_steps\fR
    Disable job steps that span heterogeneous job allocations.
    The default value on Cray systems.
    .TP
    \fBenable_hetero_steps\fR
    Enable job steps that span heterogeneous job allocations.
    The default value except for Cray systems.
    .TP
    \fBenable_user_top\fR
    Enable use of the "scontrol top" command by non-privileged users.
    .TP
    \fBIgnore_NUMA\fR
    Some processors (e.g. AMD Opteron 6000 series) contain multiple NUMA nodes per
    socket. This is a configuration which does not map into the hardware entities
    that Slurm optimizes resource allocation for (PU/thread, core, socket,
    baseboard, node and network switch). In order to optimize resource allocations
    on such hardware, Slurm will consider each NUMA node within the socket as a
    separate socket by default. Use the Ignore_NUMA option to report the correct
    socket count, but \fBnot\fR optimize resource allocations on the NUMA nodes.
    .TP
    \fBinventory_interval=#\fR
    On a Cray system using Slurm on top of ALPS this limits the number of times
    a Basil Inventory call is made.  Normally this call happens every scheduling
    consideration to attempt to close a node state change window with respects to
    what ALPS has.  This call is rather slow, so making it less frequently improves
    performance dramatically, but in the situation where a node changes state the
    window is as large as this setting.  In an HTC environment this setting is a
    must and we advise around 10 seconds.
    .TP
    \fBkill_invalid_depend\fR
    If a job has an invalid dependency and it can never run terminate it
    and set its state to be JOB_CANCELLED. By default the job stays pending
    with reason DependencyNeverSatisfied.
    .TP
    \fBmax_array_tasks\fR
    Specify the maximum number of tasks that be included in a job array.
    The default limit is MaxArraySize, but this option can be used to set a lower
    limit. For example, max_array_tasks=1000 and MaxArraySize=100001 would permit
    a maximum task ID of 100000, but limit the number of tasks in any single job
    array to 1000.
    .TP
    \fBmax_depend_depth=#\fR
    Maximum number of jobs to test for a circular job dependency. Stop testing
    after this number of job dependencies have been tested. The default value is
    10 jobs.
    .TP
    Default: 0 (option disabled), Min: 0, Max: 1000.
    The default value is zero, which disables this option.
    If a value is set, then a value of 10 or higher is recommended. It may require
    some tuning for each system, but needs to be high enough that scheduling isn't
    always disabled, and low enough that requests can get through in a reasonable
    period of time.
    .TP
    \fBmax_sched_time=#\fR
    How long, in seconds, that the main scheduling loop will execute for before
    exiting.
    If a value is configured, be aware that all other Slurm operations will be
    deferred during this time period.
    Make certain the value is lower than \fBMessageTimeout\fR.
    If a value is not explicitly configured, the default value is half of
    \fBMessageTimeout\fR with a minimum default value of 1 second and a maximum
    default value of 2 seconds.
    For example if MessageTimeout=10, the time limit will be 2 seconds
    (i.e. MIN(10/2, 2) = 2).
    .TP
    \fBmax_script_size=#\fR
    Specify the maximum size of a batch script, in bytes.
    The default value is 4 megabytes.
    Larger values may adversely impact system performance.
    .TP
    \fBmax_switch_wait=#\fR
    Maximum number of seconds that a job can delay execution waiting for the
    specified desired switch count. The default value is 300 seconds.
    .TP
    \fBno_backup_scheduling\fR
    If used, the backup controller will not schedule jobs when it takes over. The
    backup controller will allow jobs to be submitted, modified and cancelled but
    won't schedule new jobs. This is useful in Cray environments when the backup
    controller resides on an external Cray node.  A restart is required to alter
    this option. This is explicitly set on a Cray/ALPS system.
    .TP
    \fBno_env_cache\fR
    If used, any job started on node that fails to load the env from a node will
    fail instead of using the cached env.  This will also implicitly imply the
    requeue_setup_env_fail option as well.
    .TP
    \fBnohold_on_prolog_fail\fR
    By default, if the Prolog exits with a non-zero value the job is requeued in
    a held state. By specifying this parameter the job will be requeued but not
    held so that the scheduler can dispatch it to another host.
    .TP
    \fBpack_serial_at_end\fR
    If used with the select/cons_res or select/cons_tres plugin,
    then put serial jobs at the end of
    the available nodes rather than using a best fit algorithm.
    This may reduce resource fragmentation for some workloads.
    .TP
    \fBpartition_job_depth=#\fR
    Specify how many attempts should be made in reording preemptable jobs to
    minimize the count of jobs preempted.
    The default value is 1. High values may adversely impact performance.
    The logic to support this option is only available in the select/cons_res and
    select/cons_tres plugins.
    .TP
    \fBpreempt_strict_order\fR
    If set, then execute extra logic in an attempt to preempt only the lowest
    priority jobs.
    It may be desirable to set this configuration parameter when there are multiple
    priorities of preemptable jobs.
    The logic to support this option is only available in the select/cons_res and
    select/cons_tres plugins.
    .TP
    \fBpreempt_youngest_first\fR
    If set, then the preemption sorting algorithm will be changed to sort by the
    job start times to favor preempting younger jobs over older. (Requires
    preempt/partition_prio or preempt/qos plugins.)
    .TP
    \fBreduce_completing_frag\fR
    This option is used to control how scheduling of resources is performed when
    jobs are in completing state, which influences potential fragmentation.
    If the option is not set then no jobs will be started in any partition when
    any job is in completing state.
    If the option is set then no jobs will be started in any individual partition
    that has a job in completing state.
    In addition, no jobs will be started in any partition with nodes that overlap
    with any nodes in the partition of the completing job.
    This option is to be used in conjunction with \fBCompleteWait\fR.
    NOTE: \fBCompleteWait\fR must be set for this to work.
    .TP
    \fBrequeue_setup_env_fail\fR
    By default if a job environment setup fails the job keeps running with
    a limited environment. By specifying this parameter the job will be
    requeued in held state and the execution node drained.
    .TP
    \fBsalloc_wait_nodes\fR
    If defined, the salloc command will wait until all allocated nodes are ready for
    use (i.e. booted) before the command returns. By default, salloc will return as
    soon as the resource allocation has been made.
    .TP
    \fBsbatch_wait_nodes\fR
    If defined, the sbatch script will wait until all allocated nodes are ready for
    use (i.e. booted) before the initiation. By default, the sbatch script will be
    initiated as soon as the first node in the job allocation is ready. The sbatch
    command can use the \-\-wait\-all\-nodes option to override this configuration
    parameter.
    .TP
    \fBsched_interval=#\fR
    How frequently, in seconds, the main scheduling loop will execute and test all
    pending jobs.
    The default value is 60 seconds.
    cycle and the beginning of the next scheduling cycle.
    A value of zero will disable throttling of the scheduling logic interval.
    The default value is 1,000,000 microseconds on Cray/ALPS systems and
    2 microseconds on other systems.
    .TP
    \fBspec_cores_first\fR
    Specialized cores will be selected from the first cores of the first sockets,
    cycling through the sockets on a round robin basis.
    By default, specialized cores will be selected from the last cores of the
    last sockets, cycling through the sockets on a round robin basis.
    .TP
    \fBstep_retry_count=#\fR
    When a step completes and there are steps ending resource allocation, then
    retry step allocations for at least this number of pending steps.
    Also see \fBstep_retry_time\fR.
    The default value is 8 steps.
    .TP
    \fBstep_retry_time=#\fR
    When a step completes and there are steps ending resource allocation, then
    retry step allocations for all steps which have been pending for at least this
    number of seconds.
    Also see \fBstep_retry_count\fR.
    The default value is 60 seconds.
    .TP
    \fBwhole_hetjob\fR
    Requests to cancel, hold or release any component of a heterogeneous job will
    be applied to all components of the job.

    NOTE: this option was previously named whole_pack and this is still supported
    for retrocompatibility.
    .RE

    .TP
    \fBSchedulerTimeSlice\fR
    Number of seconds in each time slice when gang scheduling is enabled
    (\fBPreemptMode=SUSPEND,GANG\fR).
    The value must be between 5 seconds and 65533 seconds.
    The default value is 30 seconds.

    .TP
    \fBSchedulerType\fR
    Identifies the type of scheduler to be used.
    Note the \fBslurmctld\fR daemon must be restarted for a change in
    scheduler type to become effective (reconfiguring a running daemon has
    no effect for this parameter).
    The \fBscontrol\fR command can be used to manually change job priorities
    if desired.
    Acceptable values include:
    .RS
    .TP
    \fBsched/backfill\fR
    For a backfill scheduling module to augment the default FIFO scheduling.
    (e.g. the time limit) or down/drained nodes.
    In that case, lower priority jobs can be initiated and not impact the higher
    priority job.
    .TP
    \fBsched/hold\fR
    To hold all newly arriving jobs if a file "/etc/slurm.hold"
    exists otherwise use the built\-in FIFO scheduler
    .RE

    .TP
    \fBSelectType\fR
    Identifies the type of resource selection algorithm to be used.
    Changing this value can only be done by restarting the slurmctld daemon
    and will result in the loss of all job information (running and pending)
    since the job state save format used by each plugin is different.
    Acceptable values include
    .RS
    .TP
    \fBselect/cons_res\fR
    The resources (cores and memory) within a node are individually allocated as
    consumable resources.
    Note that whole nodes can be allocated to jobs for selected
    partitions by using the \fIOverSubscribe=Exclusive\fR option.
    See the partition \fBOverSubscribe\fR parameter for more information.
    .TP
    \fBselect/cray_aries\fR
    for a Cray system.
    The default value is "select/cray_aries" for all Cray systems.
    .TP
    \fBselect/linear\fR
    for allocation of entire nodes assuming a one\-dimensional array of nodes in
    which sequentially ordered nodes are preferable.
    For a heterogeneous cluster (e.g. different CPU counts on the various nodes),
    resource allocations will favor nodes with high CPU counts as needed based upon
    the job's node and CPU specification if \fBTopologyPlugin=topology/none\fR is
    configured. Use of other topology plugins with select/linear and heterogeneous
    nodes is not recommended and may result in valid job allocation requests being
    rejected.
    This is the default value.
    .TP
    \fBselect/cons_tres\fR
    The resources (cores, memory, GPUs and all other trackable resources) within
    a node are individually allocated as consumable resources.
    Note that whole nodes can be allocated to jobs for selected
    partitions by using the \fIOverSubscribe=Exclusive\fR option.
    See the partition \fBOverSubscribe\fR parameter for more information.
    .RE

    .TP
    \fBSelectTypeParameters\fR
    The permitted values of \fBSelectTypeParameters\fR depend upon the
    configured value of \fBSelectType\fR.
    .TP
    \fBOTHER_CONS_RES\fR
    Layer the select/cons_res plugin under the select/cray_aries plugin, the default is
    to layer on select/linear.  This also allows all the options available for
    \fBSelectType=select/cons_res\fR.
    .TP
    \fBOTHER_CONS_TRES\fR
    Layer the select/cons_tres plugin under the select/cray_aries plugin, the default is
    to layer on select/linear.  This also allows all the options available for
    \fBSelectType=select/cons_tres\fR.
    .RE

    The following options are supported by the \fBSelectType=select/cons_res\fR
    and \fBSelectType=select/cons_tres\fR plugins:
    .RS
    .TP
    \fBCR_CPU\fR
    CPUs are consumable resources.
    Configure the number of \fBCPUs\fR on each node, which may be equal to the
    count of cores or hyper\-threads on the node depending upon the desired minimum
    resource allocation.
    The node's \fBBoards\fR, \fBSockets\fR, \fBCoresPerSocket\fR and
    \fBThreadsPerCore\fR may optionally be configured and result in job
    allocations which have improved locality; however doing so will prevent
    more than one job being from being allocated on each core.
    .TP
    \fBCR_CPU_Memory\fR
    CPUs and memory are consumable resources.
    Configure the number of \fBCPUs\fR on each node, which may be equal to the
    count of cores or hyper\-threads on the node depending upon the desired minimum
    resource allocation.
    The node's \fBBoards\fR, \fBSockets\fR, \fBCoresPerSocket\fR and
    \fBThreadsPerCore\fR may optionally be configured and result in job
    allocations which have improved locality; however doing so will prevent
    more than one job being from being allocated on each core.
    Setting a value for \fBDefMemPerCPU\fR is strongly recommended.
    .TP
    \fBCR_Core\fR
    Cores are consumable resources.
    On nodes with hyper\-threads, each thread is counted as a CPU to
    satisfy a job's resource requirement, but multiple jobs are not
    allocated threads on the same core.
    The count of CPUs allocated to a job may be rounded up to account for every
    CPU on an allocated core.
    .TP
    \fBCR_Core_Memory\fR
    Cores and memory are consumable resources.
    On nodes with hyper\-threads, each thread is counted as a CPU to
    satisfy a job's resource requirement, but multiple jobs are not
    allocated threads on the same core.
    The count of CPUs allocated to a job may be rounded up to account for every
    CPU on an allocated core.
    "\-m" parameter with srun/salloc/sbatch.
    Without this option, cores will be allocated cyclicly across the sockets.
    .TP
    \fBCR_LLN\fR
    Schedule resources to jobs on the least loaded nodes (based upon the number
    of idle CPUs). This is generally only recommended for an environment with
    serial jobs as idle resources will tend to be highly fragmented, resulting
    in parallel jobs being distributed across many nodes.
    Note that node \fBWeight\fR takes precedence over how many idle resources are
    on each node.
    Also see the partition configuration parameter \fBLLN\fR
    use the least loaded nodes in selected partitions.
    .TP
    \fBCR_Pack_Nodes\fR
    If a job allocation contains more resources than will be used for launching
    tasks (e.g. if whole nodes are allocated to a job), then rather than
    distributing a job's tasks evenly across it's allocated nodes, pack them as
    tightly as possible on these nodes.
    For example, consider a job allocation containing two \fBentire\fR nodes with
    eight CPUs each.
    If the job starts ten tasks across those two nodes without this option, it will
    start five tasks on each of the two nodes.
    With this option, eight tasks will be started on the first node and two tasks
    on the second node.
    .TP
    \fBCR_Socket\fR
    Sockets are consumable resources.
    On nodes with multiple cores, each core or thread is counted as a CPU
    to satisfy a job's resource requirement, but multiple jobs are not
    allocated resources on the same socket.
    .TP
    \fBCR_Socket_Memory\fR
    Memory and sockets are consumable resources.
    On nodes with multiple cores, each core or thread is counted as a CPU
    to satisfy a job's resource requirement, but multiple jobs are not
    allocated resources on the same socket.
    Setting a value for \fBDefMemPerCPU\fR is strongly recommended.
    .TP
    \fBCR_Memory\fR
    Memory is a consumable resource.
    NOTE: This implies \fIOverSubscribe=YES\fR or \fIOverSubscribe=FORCE\fR for
    all partitions.
    Setting a value for \fBDefMemPerCPU\fR is strongly recommended.
    .RE
    .RE

    .TP
    \fBSlurmUser\fR
    The name of the user that the \fBslurmctld\fR daemon executes as.
    For security purposes, a user other than "root" is recommended.
    This user must exist on all nodes of the cluster for authentication
    of communications between Slurm components.
    .TP
    \fBshutdown_on_reboot\fR
    If set, the Slurmd will shut itself down when a reboot request is received.
    .RE

    .TP
    \fBSlurmdUser\fR
    The name of the user that the \fBslurmd\fR daemon executes as.
    This user must exist on all nodes of the cluster for authentication
    of communications between Slurm components.
    The default value is "root".

    .TP
    \fBSlurmctldAddr\fR
    An optional address to be used for communications to the currently active
    slurmctld daemon, normally used with Virtual IP addressing of the currently
    active server.
    If this parameter is not specified then each primary and backup server will
    have its own unique address used for communications as specified in the
    \fBSlurmctldHost\fR parameter.
    If this parameter is specified then the \fBSlurmctldHost\fR parameter will
    still be used for communications to specific slurmctld primary or backup
    servers, for example to cause all of them to read the current configuration
    files or shutdown.
    Also see the \fBSlurmctldPrimaryOffProg\fR and \fBSlurmctldPrimaryOnProg\fR
    configuration parameters to configure programs to manipulate virtual IP
    address manipulation.

    .TP
    \fBSlurmctldDebug\fR
    The level of detail to provide \fBslurmctld\fR daemon's logs.
    The default value is \fBinfo\fR.
    If the \fBslurmctld\fR daemon is initiated with \-v or \-\-verbose options,
    that debug level will be preserve or restored upon reconfiguration.

    .RS
    .TP 10
    \fBquiet\fR
    Log nothing
    .TP
    \fBfatal\fR
    Log only fatal errors
    .TP
    \fBerror\fR
    Log only errors
    .TP
    \fBinfo\fR
    Log errors and general informational messages
    .TP
    \fBverbose\fR
    Log errors and verbose informational messages
    .TP
    .RE

    .TP
    \fBSlurmctldHost\fR
    The short, or long, hostname of the machine where Slurm control daemon is
    executed (i.e. the name returned by the command "hostname \-s").
    This hostname is optionally followed by the address, either the IP address or
    a name by which the address can be identifed, enclosed in parentheses (e.g.
    SlurmctldHost=master1(12.34.56.78)). This value must be specified at least once.
    If specified more than once, the first hostname named will be where the daemon
    runs.
    If the first specified host fails, the daemon will execute on the second host.
    If both the first and second specified host fails, the daemon will execute on
    the third host.

    .TP
    \fBSlurmctldLogFile\fR
    Fully qualified pathname of a file into which the \fBslurmctld\fR daemon's
    logs are written.
    The default value is none (performs logging via syslog).
    .br
    See the section \fBLOGGING\fR if a pathname is specified.

    .TP
    \fBSlurmctldParameters\fR
    Multiple options may be comma-separated.

    .RS
    .TP
    \fBallow_user_triggers\fR
    Permit setting triggers from non-root/slurm_user users. SlurmUser must also
    be set to root to permit these triggers to work. See the \fBstrigger\fR man
    page for additional details.
    .TP
    \fBcloud_dns\fR
    By default, Slurm expects that the network address for a cloud node won't
    be known until the creation of the node and that Slurm will be notified of the
    node's address (e.g. \fBscontrol update nodename=<name> nodeaddr=<addr>\fR).
    Since Slurm communications rely on the node configuration found in the
    slurm.conf, Slurm will tell the client command, after waiting for all nodes to
    boot, each node's ip address. However, in environments where the nodes are in
    DNS, this step can be avoided by configuring this option.
    .TP
    \fBidle_on_node_suspend\fR Mark nodes as idle, regardless of current state,
    when suspending nodes with \fISuspendProgram\fB so that nodes will be eligible
    to be resumed at a later time.
    .TP
    \fBpreempt_send_user_signal\fR Send the user signal (e.g. --signal=<sig_num>)
    at preemption time even if the signal time hasn't been reached. In the case of
    a gracetime preemption the user signal will be sent if the user signal has been
    specified and not sent, otherwise a SIGTERM will be sent to the tasks.
    .TP
    daemon begins and terminated when it ends.
    Only the plugin's init and fini functions are called.

    .TP
    \fBSlurmctldPort\fR
    The port number that the Slurm controller, \fBslurmctld\fR, listens
    to for work. The default value is SLURMCTLD_PORT as established at system
    build time. If none is explicitly specified, it will be set to 6817.
    \fBSlurmctldPort\fR may also be configured to support a range of port
    numbers in order to accept larger bursts of incoming messages by specifying
    two numbers separated by a dash (e.g. \fBSlurmctldPort=6817\-6818\fR).
    NOTE: Either \fBslurmctld\fR and \fBslurmd\fR daemons must not
    execute on the same nodes or the values of \fBSlurmctldPort\fR and
    \fBSlurmdPort\fR must be different.

    \fBNote\fR: On Cray systems, Realm-Specific IP Addressing (RSIP) will
    automatically try to interact with anything opened on ports 8192\-60000.
    Configure SlurmctldPort to use a port outside of the configured SrunPortRange
    and RSIP's port range.

    .TP
    \fBSlurmctldPrimaryOffProg\fR
    This program is executed when a slurmctld daemon running as the primary server
    becomes a backup server. By default no program is executed.
    See also the related "SlurmctldPrimaryOnProg" parameter.

    .TP
    \fBSlurmctldPrimaryOnProg\fR
    This program is executed when a slurmctld daemon running as a backup server
    becomes the primary server. By default no program is executed.
    When using virtual IP addresses to manage High Available Slurm services,
    this program can be used to add the IP address to an interface (and optionally
    try to kill the unresponsive  slurmctld daemon and flush the ARP caches on
    nodes on the local ethernet fabric).
    See also the related "SlurmctldPrimaryOffProg" parameter.
    .TP
    \fBSlurmctldSyslogDebug\fR
    The slurmctld daemon will log events to the syslog file at the specified
    level of detail. If not set, the slurmctld daemon will log to syslog at
    level \fBfatal\fR, unless there is no \fBSlurmctldLogFile\fR and it is running
    in the background, in which case it will log to syslog at the level specified
    by \fBSlurmctldDebug\fR (at \fBfatal\fR in the case that \fBSlurmctldDebug\fR
    is set to \fBquiet\fR) or it is run in the foreground, when it will be set to
    quiet.

    .RS
    .TP 10
    \fBquiet\fR
    Log nothing
    .TP
    \fBfatal\fR
    Log only fatal errors
    Log errors and verbose informational messages and more debugging messages
    .TP
    \fBdebug3\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug4\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug5\fR
    Log errors and verbose informational messages and even more debugging messages
    .RE

    .TP
    \fBSlurmctldTimeout\fR
    The interval, in seconds, that the backup controller waits for the
    primary controller to respond before assuming control.
    The default value is 120 seconds.
    May not exceed 65533.

    .TP
    \fBSlurmdDebug\fR
    The level of detail to provide \fBslurmd\fR daemon's logs.
    The default value is \fBinfo\fR.
    .RS
    .TP 10
    \fBquiet\fR
    Log nothing
    .TP
    \fBfatal\fR
    Log only fatal errors
    .TP
    \fBerror\fR
    Log only errors
    .TP
    \fBinfo\fR
    Log errors and general informational messages
    .TP
    \fBverbose\fR
    Log errors and verbose informational messages
    .TP
    \fBdebug\fR
    Log errors and verbose informational messages and debugging messages
    .TP
    \fBdebug2\fR
    Log errors and verbose informational messages and more debugging messages
    .TP
    \fBdebug3\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug4\fR
    Log errors and verbose informational messages and even more debugging messages
    .br
    See the section \fBLOGGING\fR if a pathname is specified.

    .TP
    \fBSlurmdPidFile\fR
    Fully qualified pathname of a file into which the  \fBslurmd\fR daemon may write
    its process id. This may be used for automated signal processing.
    Any "%h" within the name is replaced with the hostname on which the
    \fBslurmd\fR is running.
    Any "%n" within the name is replaced with the Slurm node name on which the
    \fBslurmd\fR is running.
    The default value is "/var/run/slurmd.pid".

    .TP
    \fBSlurmdPort\fR
    The port number that the Slurm compute node daemon, \fBslurmd\fR, listens
    to for work. The default value is SLURMD_PORT as established at system
    build time. If none is explicitly specified, its value will be 6818.
    NOTE: Either slurmctld and slurmd daemons must not execute
    on the same nodes or the values of \fBSlurmctldPort\fR and \fBSlurmdPort\fR
    must be different.

    \fBNote\fR: On Cray systems, Realm-Specific IP Addressing (RSIP) will
    automatically try to interact with anything opened on ports 8192\-60000.
    Configure SlurmdPort to use a port outside of the configured SrunPortRange
    and RSIP's port range.

    .TP
    \fBSlurmdSpoolDir\fR
    Fully qualified pathname of a directory into which the \fBslurmd\fR
    daemon's state information and batch job script information are written. This
    must be a common pathname for all nodes, but should represent a directory which
    is local to each node (reference a local file system). The default value
    is "/var/spool/slurmd".
    Any "%h" within the name is replaced with the hostname on which the
    \fBslurmd\fR is running.
    Any "%n" within the name is replaced with the Slurm node name on which the
    \fBslurmd\fR is running.

    .TP
    \fBSlurmdSyslogDebug\fR
    The slurmd daemon will log events to the syslog file at the specified
    level of detail. If not set, the slurmd daemon will log to syslog at
    level \fBfatal\fR, unless there is no \fBSlurmdLogFile\fR and it is running
    in the background, in which case it will log to syslog at the level specified
    by \fBSlurmdDebug\fR  (at \fBfatal\fR in the case that \fBSlurmdDebug\fR
    is set to \fBquiet\fR) or it is run in the foreground, when it will be set to
    quiet.

    .RS
    .TP 10
    \fBquiet\fR
    \fBdebug\fR
    Log errors and verbose informational messages and debugging messages
    .TP
    \fBdebug2\fR
    Log errors and verbose informational messages and more debugging messages
    .TP
    \fBdebug3\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug4\fR
    Log errors and verbose informational messages and even more debugging messages
    .TP
    \fBdebug5\fR
    Log errors and verbose informational messages and even more debugging messages
    .RE

    .TP
    \fBSlurmdTimeout\fR
    The interval, in seconds, that the Slurm controller waits for \fBslurmd\fR
    to respond before configuring that node's state to DOWN.
    A value of zero indicates the node will not be tested by \fBslurmctld\fR to
    confirm the state of \fBslurmd\fR, the node will not be automatically set to
    a DOWN state indicating a non\-responsive \fBslurmd\fR, and some other tool
    will take responsibility for monitoring the state of each compute node
    and its \fBslurmd\fR daemon.
    Slurm's hierarchical communication mechanism is used to ping the \fBslurmd\fR
    daemons in order to minimize system noise and overhead.
    The default value is 300 seconds.
    The value may not exceed 65533 seconds.

    .TP
    \fBSlurmSchedLogFile\fR
    Fully qualified pathname of the scheduling event logging file.
    The syntax of this parameter is the same as for \fBSlurmctldLogFile\fR.
    In order to configure scheduler logging, set both the \fBSlurmSchedLogFile\fR
    and \fBSlurmSchedLogLevel\fR parameters.

    .TP
    \fBSlurmSchedLogLevel\fR
    The initial level of scheduling event logging, similar to the
    \fBSlurmctldDebug\fR parameter used to control the initial level of
    \fBslurmctld\fR logging.
    Valid values for \fBSlurmSchedLogLevel\fR are "0" (scheduler logging
    disabled) and "1" (scheduler logging enabled).
    If this parameter is omitted, the value defaults to "0" (disabled).
    In order to configure scheduler logging, set both the \fBSlurmSchedLogFile\fR
    and \fBSlurmSchedLogLevel\fR parameters.
    The scheduler logging level can be changed dynamically using \fBscontrol\fR.

    .TP
    \fBSrunEpilog\fR
    Fully qualified pathname of an executable to be run by srun following
    from which srun ports will be selected. This is useful if sites want to
    allow only certain port range on their network.

    \fBNote\fR: On Cray systems, Realm-Specific IP Addressing (RSIP) will
    automatically try to interact with anything opened on ports 8192\-60000.
    Configure SrunPortRange to use a range of ports above those used by RSIP,
    ideally 1000 or more ports, for example "SrunPortRange=60001\-63000".

    \fBNote\fR: A sufficient number of ports must be configured based on the
    estimated number of srun on the submission nodes considering that srun opens
    3 listening ports plus 2 more for every 48 hosts. Example:
    .RS
    .TP 18
    \fBsrun \-N 48\fR will use 5 listening ports.

    .TP
    \fBsrun \-N 50\fR will use 7 listening ports.
    .TP

    \fBsrun \-N 200\fR will use 13 listening ports.
    .RE

    .TP
    \fBSrunProlog\fR
    Fully qualified pathname of an executable to be run by srun prior to
    the launch of a job step.  The command line arguments for the
    executable will be the command and arguments of the job step.  This
    configuration parameter may be overridden by srun's \fB\-\-prolog\fR
    parameter. Note that while the other "Prolog" executables (e.g.,
    TaskProlog) are run by slurmd on the compute nodes where the tasks are
    executed, the \fBSrunProlog\fR runs on the node where the "srun" is
    executing.

    .TP
    \fBStateSaveLocation\fR
    Fully qualified pathname of a directory into which the Slurm controller,
    \fBslurmctld\fR, saves its state (e.g. "/usr/local/slurm/checkpoint").
    Slurm state will saved here to recover from system failures.
    \fBSlurmUser\fR must be able to create files in this directory.
    If you have a \fBBackupController\fR configured, this location should be
    readable and writable by both systems.
    Since all running and pending job information is stored here, the use of
    a reliable file system (e.g. RAID) is recommended.
    The default value is "/var/spool".
    If any slurm daemons terminate abnormally, their core files will also be written
    into this directory.

    .TP
    \fBSuspendExcNodes\fR
    Specifies the nodes which are to not be placed in power save mode, even
    if the node remains idle for an extended period of time.
    Use Slurm's hostlist expression to identify nodes with an optional ":" separator
    \fBSuspendTimeout\fR, and \fBSuspendExcParts\fR.

    .TP
    \fBSuspendExcParts\fR
    Specifies the partitions whose nodes are to not be placed in power save
    mode, even if the node remains idle for an extended period of time.
    Multiple partitions can be identified and separated by commas.
    By default no nodes are excluded.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeProgram\fR,
    \fBResumeRate\fR, \fBSuspendProgram\fR, \fBSuspendRate\fR, \fBSuspendTime\fR
    \fBSuspendTimeout\fR, and \fBSuspendExcNodes\fR.

    .TP
    \fBSuspendProgram\fR
    \fBSuspendProgram\fR is the program that will be executed when a node
    remains idle for an extended period of time.
    This program is expected to place the node into some power save mode.
    This can be used to reduce the frequency and voltage of a node or
    completely power the node off.
    The program executes as \fBSlurmUser\fR.
    The argument to the program will be the names of nodes to
    be placed into power savings mode (using Slurm's hostlist
    expression format).
    By default, no program is run.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeProgram\fR,
    \fBResumeRate\fR, \fBSuspendRate\fR, \fBSuspendTime\fR, \fBSuspendTimeout\fR,
    \fBSuspendExcNodes\fR, and \fBSuspendExcParts\fR.

    .TP
    \fBSuspendRate\fR
    The rate at which nodes are placed into power save mode by \fBSuspendProgram\fR.
    The value is number of nodes per minute and it can be used to prevent
    a large drop in power consumption (e.g. after a large job completes).
    A value of zero results in no limits being imposed.
    The default value is 60 nodes per minute.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeProgram\fR,
    \fBResumeRate\fR, \fBSuspendProgram\fR, \fBSuspendTime\fR, \fBSuspendTimeout\fR,
    \fBSuspendExcNodes\fR, and \fBSuspendExcParts\fR.

    .TP
    \fBSuspendTime\fR
    Nodes which remain idle or down for this number of seconds will be placed into
    power save mode by \fBSuspendProgram\fR.
    For efficient system utilization, it is recommended that the value of
    \fBSuspendTime\fR be at least as large as the sum of \fBSuspendTimeout\fR
    plus \fBResumeTimeout\fR.
    A value of \-1 disables power save mode and is the default.
    Related configuration options include \fBResumeTimeout\fR, \fBResumeProgram\fR,
    \fBResumeRate\fR, \fBSuspendProgram\fR, \fBSuspendRate\fR, \fBSuspendTimeout\fR,
    \fBSuspendExcNodes\fR, and \fBSuspendExcParts\fR.

    .TP
    Identifies the type of switch or interconnect used for application
    communications.
    Acceptable values include
    "switch/cray_aries" for Cray systems,
    "switch/none" for switches not requiring special processing for job launch
    or termination (Ethernet, and InfiniBand) and
    The default value is "switch/none".
    All Slurm daemons, commands and running jobs must be restarted for a
    change in \fBSwitchType\fR to take effect.
    If running jobs exist at the time \fBslurmctld\fR is restarted with a new
    value of \fBSwitchType\fR, records of all jobs in any state may be lost.

    .TP
    \fBTaskEpilog\fR
    Fully qualified pathname of a program to be execute as the slurm job's
    owner after termination of each task.
    See \fBTaskProlog\fR for execution order details.

    .TP
    \fBTaskPlugin\fR
    Identifies the type of task launch plugin, typically used to provide
    resource management within a node (e.g. pinning tasks to specific
    processors). More than one task plugin can be specified in a comma separated
    list. The prefix of "task/" is optional. Acceptable values include:
    .RS
    .TP 15
    \fBtask/affinity\fR
    enables resource containment using CPUSETs.
    This enables the \-\-cpu\-bind and/or \-\-mem\-bind srun options.
    If you use "task/affinity" and encounter problems, it may be due to
    the variety of system calls used to implement task affinity on
    different operating systems.
    .TP
    \fBtask/cgroup\fR
    enables resource containment using Linux control cgroups.
    This enables the \-\-cpu\-bind and/or \-\-mem\-bind srun options.
    NOTE: see "man cgroup.conf" for configuration details.
    .TP
    \fBtask/none\fR
    for systems requiring no special handling of user tasks.
    Lacks support for the \-\-cpu\-bind and/or \-\-mem\-bind srun options.
    The default value is "task/none".
    .RE

    \fBNOTE:\fR It is recommended to stack \fBtask/affinity,task/cgroup\fR together
    when configuring TaskPlugin, and setting \fBTaskAffinity=no\fR and
    \fBConstrainCores=yes\fR in \fBcgroup.conf\fR. This setup uses the task/affinity
    plugin for setting the affinity of the tasks (which is better and different than
    task/cgroup) and uses the task/cgroup plugin to fence tasks into the specified
    resources, thus combining the best of both pieces.

    \fBNOTE:\fR For CRAY systems only: task/cgroup must be used with, and listed
    \fBNone\fR, \fBBoards\fR, \fBSockets\fR, \fBCores\fR and \fBThreads\fR are mutually
    exclusive and since they decrease scheduling flexibility are not generally
    recommended (select no more than one of them).
    \fBCpusets\fR and \fBSched\fR
    are mutually exclusive (select only one of them).
    All TaskPluginParam options are supported on FreeBSD except \fBCpusets\fR.
    The \fBSched\fR option uses cpuset_setaffinity() on FreeBSD, not sched_setaffinity().

    .RS
    .TP 10
    \fBBoards\fR
    Bind tasks to boards by default.
    Overrides automatic binding.
    .TP
    \fBCores\fR
    Bind tasks to cores by default.
    Overrides automatic binding.
    .TP
    \fBCpusets\fR
    Use cpusets to perform task affinity functions.
    By default, \fBSched\fR task binding is performed.
    .TP
    \fBNone\fR
    Perform no task binding by default.
    Overrides automatic binding.
    .TP
    \fBSched\fR
    Use \fIsched_setaffinity\fR (if available) to bind tasks to
    processors.
    .TP
    \fBSockets\fR
    Bind to sockets by default.
    Overrides automatic binding.
    .TP
    \fBThreads\fR
    Bind to threads by default.
    Overrides automatic binding.
    .TP
    \fBSlurmdOffSpec\fR
    If specialized cores or CPUs are identified for the node (i.e. the
    \fBCoreSpecCount\fR or \fBCpuSpecList\fR are configured for the node),
    then Slurm daemons running on the compute node (i.e. slurmd and slurmstepd)
    should run outside of those resources (i.e. specialized resources are
    completely unavailable to Slurm daemons and jobs spawned by Slurm).
    This option may not be used with the \fBtask/cray_aries\fR plugin.
    .TP
    \fBVerbose\fR
    Verbosely report binding before tasks run.
    Overrides user options.
    .TP
    \fBAutobind\fR
    Set a default binding in the event that "auto binding" doesn't find a match.
    Will set environment variables for the task being spawned.
    Everything after the equal sign to the end of the
    line will be used as the value for the environment variable.
    Exporting of functions is not currently supported.
    .TP
    \fBprint ...\fR
    Will cause that line (without the leading "print ")
    to be printed to the job's standard output.
    .TP
    \fBunset NAME\fR
    Will clear environment variables for the task being spawned.
    .TP
    The order of task prolog/epilog execution is as follows:
    .TP
    \fB1. pre_launch_priv()\fR
    Function in TaskPlugin
    .TP
    \fB1. pre_launch()\fR
    Function in TaskPlugin
    .TP
    \fB2. TaskProlog\fR
    System\-wide per task program defined in slurm.conf
    .TP
    \fB3. user prolog\fR
    Job step specific task program defined using
    \fBsrun\fR's \fB\-\-task\-prolog\fR option or \fBSLURM_TASK_PROLOG\fR
    environment variable
    .TP
    \fB4.\fR Execute the job step's task
    .TP
    \fB5. user epilog\fR
    Job step specific task program defined using
    \fBsrun\fR's \fB\-\-task\-epilog\fR option or \fBSLURM_TASK_EPILOG\fR
    environment variable
    .TP
    \fB6. TaskEpilog\fR
    System\-wide per task program defined in slurm.conf
    .TP
    \fB7. post_term()\fR
    Function in TaskPlugin
    .RE

    .TP
    \fBTCPTimeout\fR
    Time permitted for TCP connection to be established. Default value is 2 seconds.

    .TP
    \fBTmpFS\fR
    Fully qualified pathname of the file system available to user jobs for
    temporary storage. This parameter is used in establishing a node's \fBTmpDisk\fR
    space.
    The default value is "/tmp".
    jobs which can take advantage of it. If most job allocations are not optimized
    for network topology, they make fragment resources to the point that topology
    optimization for other jobs will be difficult to achieve.
    \fBNOTE\fR: Jobs may span across nodes without common parent switches with
    this enabled.
    .RE

    .TP
    \fBTopologyPlugin\fR
    Identifies the plugin to be used for determining the network topology
    and optimizing job allocations to minimize network contention.
    See \fBNETWORK TOPOLOGY\fR below for details.
    Additional plugins may be provided in the future which gather topology
    information directly from the network.
    Acceptable values include:
    .RS
    .TP 21
    \fBtopology/3d_torus\fR
    best\-fit logic over three\-dimensional topology
    .TP
    \fBtopology/node_rank\fR
    orders nodes based upon information a node_rank field in the node record
    as generated by a select plugin. Slurm performs a best\-fit algorithm over
    those ordered nodes
    .TP
    \fBtopology/none\fR
    default for other systems, best\-fit logic over one\-dimensional topology
    .TP
    \fBtopology/tree\fR
    used for a hierarchical network as described in a \fItopology.conf\fR file
    .RE

    .TP
    \fBTrackWCKey\fR
    Boolean yes or no.  Used to set display and track of the Workload
    Characterization Key.  Must be set to track correct wckey usage.
    NOTE: You must also set TrackWCKey in your slurmdbd.conf file to create
    historical usage reports.

    .TP
    \fBTreeWidth\fR
    \fBSlurmd\fR daemons use a virtual tree network for communications.
    \fBTreeWidth\fR specifies the width of the tree (i.e. the fanout).
    On architectures with a front end node running the slurmd daemon, the value
    must always be equal to or greater than the number of front end nodes which
    eliminates the need for message forwarding between the slurmd daemons.
    On other architectures the default value is 50, meaning each slurmd daemon can
    communicate with up to 50 other slurmd daemons and over 2500 nodes can be
    contacted with two message hops.
    The default value will work well for most clusters.
    Optimal system performance can typically be achieved if \fBTreeWidth\fR
    is set to the square root of the number of nodes in the cluster for
    \fBUnkillableStepTimeout\fR
    The length of time, in seconds, that Slurm will wait before deciding that
    processes in a job step are unkillable (after they have been signaled with
    SIGKILL) and execute \fBUnkillableStepProgram\fR as described above.
    The default timeout value is 60 seconds.
    If exceeded, the compute node will be drained to prevent future jobs from being
    scheduled on the node.

    .TP
    \fBUsePAM\fR
    If set to 1, PAM (Pluggable Authentication Modules for Linux) will be enabled.
    PAM is used to establish the upper bounds for resource limits. With PAM support
    enabled, local system administrators can dynamically configure system resource
    limits. Changing the upper bound of a resource limit will not alter the limits
    of running jobs, only jobs started after a change has been made will pick up
    the new limits.
    The default value is 0 (not to enable PAM support).
    Remember that PAM also needs to be configured to support Slurm as a service.
    For sites using PAM's directory based configuration option, a configuration
    file named \fBslurm\fR should be created. The module\-type, control\-flags, and
    module\-path names that should be included in the file are:
    .br
    auth        required      pam_localuser.so
    .br
    auth        required      pam_shells.so
    .br
    account     required      pam_unix.so
    .br
    account     required      pam_access.so
    .br
    session     required      pam_unix.so
    .br
    For sites configuring PAM with a general configuration file, the appropriate
    lines (see above), where \fBslurm\fR is the service\-name, should be added.

    \fBNOTE\fR: UsePAM option has nothing to do with the
    \fBcontribs/pam/pam_slurm\fR and/or \fBcontribs/pam_slurm_adopt\fR modules. So
    these two modules can work independently of the value set for UsePAM.

    .TP
    \fBVSizeFactor\fR
    Memory specifications in job requests apply to real memory size (also known
    as resident set size). It is possible to enforce virtual memory limits for
    both jobs and job steps by limiting their virtual memory to some percentage
    of their real memory allocation. The \fBVSizeFactor\fR parameter specifies
    the job's or job step's virtual memory limit as a percentage of its real
    memory limit. For example, if a job's real memory limit is 500MB and
    VSizeFactor is set to 101 then the job will be killed if its real memory
    exceeds 500MB or its virtual memory exceeds 505MB (101 percent of the
    real memory limit).
    The default value is 0, which disables enforcement of virtual memory limits.
    The value may not exceed 65533 percent.
    \fBhome_xauthority\fR
    If set, xauth data on the compute node will be placed in \fB~/.Xauthority\fR
    rather than in a temporary file under \fBTmpFS\fR.
    .RE

    .LP
    The configuration of nodes (or machines) to be managed by Slurm is
    also specified in \fB/etc/slurm.conf\fR.
    Changes in node configuration (e.g. adding nodes, changing their
    processor count, etc.) require restarting both the slurmctld daemon
    and the slurmd daemons.
    All slurmd daemons must know each node in the system to forward
    messages in support of hierarchical communications.
    Only the NodeName must be supplied in the configuration file.
    All other node configuration information is optional.
    It is advisable to establish baseline node configurations,
    especially if the cluster is heterogeneous.
    Nodes which register to the system with less than the configured resources
    (e.g. too little memory), will be placed in the "DOWN" state to
    avoid scheduling jobs on them.
    Establishing baseline configurations will also speed Slurm's
    scheduling process by permitting it to compare job requirements
    against these (relatively few) configuration parameters and
    possibly avoid having to check job requirements
    against every individual node's configuration.
    The resources checked at node registration time are: CPUs,
    RealMemory and TmpDisk.
    .LP
    Default values can be specified with a record in which
    \fBNodeName\fR is "DEFAULT".
    The default entry values will apply only to lines following it in the
    configuration file and the default values can be reset multiple times
    in the configuration file with multiple entries where "NodeName=DEFAULT".
    Each line where \fBNodeName\fR is "DEFAULT" will replace or add to previous
    default values and not a reinitialize the default values.
    The "NodeName=" specification must be placed on every line
    describing the configuration of nodes.
    A single node name can not appear as a NodeName value in more than one line
    (duplicate node name records will be ignored).
    In fact, it is generally possible and desirable to define the
    configurations of all nodes in only a few lines.
    This convention permits significant optimization in the scheduling
    of larger clusters.
    In order to support the concept of jobs requiring consecutive nodes
    on some architectures,
    node specifications should be place in this file in consecutive order.
    No single node name may be listed more than once in the configuration
    file.
    Use "DownNodes=" to record the state of nodes which are temporarily
    in a DOWN, DRAIN or FAILING state without altering permanent
    configuration information.
    A job step's tasks are allocated to nodes in order the nodes appear
    (e.g. "rack[0\-63]_blade[0\-41]").
    If one or more numeric expressions are included, one of them
    must be at the end of the name (e.g. "unit[0\-31]rack" is invalid),
    but arbitrary names can always be used in a comma separated list.
    .LP
    The node configuration specified the following information:

    .TP
    \fBNodeName\fR
    Name that Slurm uses to refer to a node.
    Typically this would be the string that "/bin/hostname \-s" returns.
    It may also be the fully qualified domain name as returned by "/bin/hostname \-f"
    (e.g. "foo1.bar.com"), or any valid domain name associated with the host
    through the host database (/etc/hosts) or DNS, depending on the resolver
    settings.  Note that if the short form of the hostname is not used, it
    may prevent use of hostlist expressions (the numeric portion in brackets
    must be at the end of the string).
    It may also be an arbitrary string if \fBNodeHostname\fR is specified.
    If the \fBNodeName\fR is "DEFAULT", the values specified
    with that record will apply to subsequent node specifications
    unless explicitly set to other values in that node record or
    replaced with a different set of default values.
    Each line where \fBNodeName\fR is "DEFAULT" will replace or add to previous
    default values and not a reinitialize the default values.
    For architectures in which the node order is significant,
    nodes will be considered consecutive in the order defined.
    For example, if the configuration for "NodeName=charlie" immediately
    follows the configuration for "NodeName=baker" they will be
    considered adjacent in the computer.

    .TP
    \fBNodeHostname\fR
    Typically this would be the string that "/bin/hostname \-s" returns.
    It may also be the fully qualified domain name as returned by "/bin/hostname \-f"
    (e.g. "foo1.bar.com"), or any valid domain name associated with the host
    through the host database (/etc/hosts) or DNS, depending on the resolver
    settings.  Note that if the short form of the hostname is not used, it
    may prevent use of hostlist expressions (the numeric portion in brackets
    must be at the end of the string).
    A node range expression can be used to specify a set of nodes.
    If an expression is used, the number of nodes identified by
    \fBNodeHostname\fR on a line in the configuration file must
    be identical to the number of nodes identified by \fBNodeName\fR.
    By default, the \fBNodeHostname\fR will be identical in value to
    \fBNodeName\fR.

    .TP
    \fBNodeAddr\fR
    Name that a node should be referred to in establishing
    a communications path.
    This name will be used as an
    argument to the gethostbyname() function for identification.

    .TP
    \fBCoreSpecCount\fR
    Number of cores reserved for system use.
    These cores will not be available for allocation to user jobs.
    Depending upon the \fBTaskPluginParam\fR option of \fBSlurmdOffSpec\fR,
    Slurm daemons (i.e. slurmd and slurmstepd) may either be confined to these
    resources (the default) or prevented from using these resources.
    Isolation of the Slurm daemons from user jobs may improve application performance.
    If this option and \fBCpuSpecList\fR are both designated for a
    node, an error is generated.  For information on the algorithm used by Slurm
    to select the cores refer to the core specialization documentation
    ( https://slurm.schedmd.com/core_spec.html ).

    .TP
    \fBCoresPerSocket\fR
    Number of cores in a single physical processor socket (e.g. "2").
    The CoresPerSocket value describes physical cores, not the
    logical number of processors per socket.
    \fBNOTE\fR: If you have multi\-core processors, you will likely
    need to specify this parameter in order to optimize scheduling.
    The default value is 1.

    .TP
    \fBCpuBind\fR
    If a job step request does not specify an option to control how tasks are bound
    to allocated CPUs (\-\-cpu-bind) and all nodes allocated to the job have the same
    \fBCpuBind\fR option the node \fBCpuBind\fR option will control how tasks are
    bound to allocated resources. Supported values for \fBCpuBind\fR are "none",
    "board", "socket", "ldom" (NUMA), "core" and "thread".

    .TP
    \fBCPUs\fR
    Number of logical processors on the node (e.g. "2").
    CPUs and Boards are mutually exclusive. It can be set to the total
    number of sockets, cores or threads. This can be useful when you
    want to schedule only the cores on a hyper-threaded node.
    If \fBCPUs\fR is omitted, it will be set equal to the product of
    \fBSockets\fR, \fBCoresPerSocket\fR, and \fBThreadsPerCore\fR.
    The default value is 1.

    .TP
    \fBCpuSpecList\fR
    A comma delimited list of Slurm abstract CPU IDs reserved for system use.
    The list will be expanded to include all other CPUs, if any, on the same cores.
    These cores will not be available for allocation to user jobs.
    Depending upon the \fBTaskPluginParam\fR option of \fBSlurmdOffSpec\fR,
    Slurm daemons (i.e. slurmd and slurmstepd) may either be confined to these
    resources (the default) or prevented from using these resources.
    Isolation of the Slurm daemons from user jobs may improve application performance.
    If this option and \fBCoreSpecCount\fR are both designated for a node,
    an error is generated.
    .TP
    \fBGres\fR
    A comma delimited list of generic resources specifications for a node.
    The format is: "<name>[:<type>][:no_consume]:<number>[K|M|G]".
    The first field is the resource name, which matches the GresType configuration
    parameter name.
    The optional type field might be used to identify a model of that generic
    resource.
    It is forbidden to specify both an untyped GRES and a typed GRES with the same
    wait(2))
    Available in \fBEpilogSlurmctld\fR only.
    .TP
    \fBSLURM_JOB_EXIT_CODE2\fR
    The exit code of the job script (or salloc). The value has the format