    .SH "DESCRIPTION"
    \fBscontrol\fR is used to view or modify Slurm configuration including: job,
    job step, node, partition, reservation, and overall system configuration. Most
    of the commands can only be executed by user root or an Administrator. If an
    attempt to view or modify configuration information is made by an unauthorized
    user, an error message will be printed and the requested action will not occur.
    If no command is entered on the execute line, \fBscontrol\fR will operate in an
    interactive mode and prompt for input. It will continue prompting for input and
    executing commands until explicitly terminated. If a command is entered on the
    execute line, \fBscontrol\fR will execute that command and terminate. All
    commands and options are case\-insensitive, although node names, partition
    names, and reservation names are case\-sensitive (node names "LX" and "lx" are
    distinct). All commands and options can be abbreviated to the extent that the
    specification is unique. A modified Slurm configuration can be written to
    a file using the \fIscontrol write config\fR command. The resulting file
    will be named using the convention "slurm.conf.<datetime>" and located in the
    same directory as the original "slurm.conf" file. The directory containing
    the original slurm.conf must be writable for this to occur.

    .SH "OPTIONS"
    .TP
    \fB\-a\fR, \fB\-\-all\fR
    When the \fIshow\fR command is used, then display all partitions, their jobs
    and jobs steps. This causes information to be displayed about partitions
    that are configured as hidden and partitions that are unavailable to user's
    group.
    .TP
    \fB\-d\fR, \fB\-\-details\fR
    Causes the \fIshow\fR command to provide additional details where available.
    .TP
    \fB\-\-federation\fR
    Report jobs from from federation if a member of one.
    .TP
    \fB\-F\fR, \fB\-\-future\fR
    Report nodes in FUTURE state.
    .TP
    \fB\-h\fR, \fB\-\-help\fR
    Print a help message describing the usage of scontrol.
    .TP
    \fB\-\-hide\fR
    Do not display information about hidden partitions, their jobs and job steps.
    By default, neither partitions that are configured as hidden nor those partitions
    unavailable to user's group will be displayed (i.e. this is the default behavior).
    .TP
    \fB\-\-local\fR
    Show only information local to this cluster. Ignore other clusters in the
    federated if a member of one. Overrides \-\-federation.
    .TP
    \fB\-M\fR, \fB\-\-clusters\fR=<\fIstring\fR>
    The cluster to issue commands to. Only one cluster name may be specified.
    Note that the SlurmDBD must be up for this option to work properly.
    .TP
    \fB\-v\fR, \fB\-\-verbose\fR
    Print detailed event logging. Multiple \fB\-v\fR's will further increase
    the verbosity of logging. By default only errors will be displayed.

    .TP
    \fB\-V\fR , \fB\-\-version\fR
    Print version information and exit.
    .TP
    \fBCOMMANDS\fR

    .TP
    \fBabort\fP
    Instruct the Slurm controller to terminate immediately and generate a core file.
    See "man slurmctld" for information about where the core file will be
    written.

    .TP
    \fBcancel_reboot \fI<NodeList>\fP
    Cancel pending reboots on nodes.

    .TP
    \fBcheckpoint\fP \fICKPT_OP\fP \fIID\fP
    Perform a checkpoint activity on the job step(s) with the specified identification.
    \fIID\fP can be used to identify a specific job (e.g. "<job_id>",
    which applies to all of its existing steps)
    or a specific job step (e.g. "<job_id>.<step_id>").
    Acceptable values for \fICKPT_OP\fP include:
    .RS
    .TP 12
    \fIable\fP
    Test if presently not disabled, report start time if checkpoint in progress
    .TP
    \fIcreate\fP
    Create a checkpoint and continue the job or job step
    .TP
    \fIdisable\fP
    Disable future checkpoints
    .TP
    \fIenable\fP
    Enable future checkpoints
    .TP
    \fIerror\fP
    Report the result for the last checkpoint request, error code and message
    .TP
    \fIrestart\fP
    Restart execution of the previously checkpointed job or job step
    .TP
    \fIrequeue\fP
    Create a checkpoint and requeue the batch job, combines vacate
    and restart operations
    .TP
    This value takes precedent over any \-\-checkpoint\-dir value specified
    at job submission time.
    .TP
    \fIStickToNodes\fP
    If set, resume job on the same nodes are previously used.
    Valid with the \fIrestart\fP option only.
    .RE

    .TP
    \fBcluster\fR \fICLUSTER_NAME\fP
    The cluster to issue commands to. Only one cluster name may be specified.

    .TP
    \fBcreate\fP \fISPECIFICATION\fP
    Create a new partition or reservation.  See the full list of parameters
    below.  Include the tag "res" to create a reservation without specifying
    a reservation name.

    .TP
    \fBcompleting\fP
    Display all jobs in a COMPLETING state along with associated nodes in either a
    COMPLETING or DOWN state.

    .TP
    \fBdelete\fP \fISPECIFICATION\fP
    Delete the entry with the specified \fISPECIFICATION\fP.
    The two \fISPECIFICATION\fP choices are \fIPartitionName=<name>\fP and
    \fIReservation=<name>\fP. Reservations and partitions should have
    no associated jobs at the time of their deletion (modify the jobs first).
    If the specified partition is in use, the request is denied.

    .TP
    \fBerrnumstr\fP \fIERRNO\fP
    Given a Slurm error number, return a descriptive string.

    .TP
    \fBfsdampeningfactor\fP \fIFACTOR\fP
    Set the FairShareDampeningFactor in slurmctld.

    .TP
    \fBhelp\fP
    Display a description of scontrol options and commands.

    .TP
    \fBhold\fP \fIjob_list\fP
    Prevent a pending job from being started (sets it's priority to 0).
    Use the \fIrelease\fP command to permit the job to be scheduled.
    The job_list argument is a comma separated list of job IDs OR
    "jobname=" with the job's name, which will attempt to hold all jobs having
    that name.
    Note that when a job is held by a system administrator using the \fBhold\fP
    command, only a system administrator may release the job for execution (also
    supplied process id, \fIproc_id\fP, on the current node.  This will work only
    with processes on node on which scontrol is run, and only for those processes
    spawned by Slurm and their descendants.

    .TP
    \fBlistpids\fP [\fIjob_id\fP[.\fIstep_id\fP]] [\fINodeName\fP]
    Print a listing of the process IDs in a job step (if JOBID.STEPID is provided),
    or all of the job steps in a job (if \fIjob_id\fP is provided), or all of the job
    steps in all of the jobs on the local node (if \fIjob_id\fP is not provided
    or \fIjob_id\fP is "*").  This will work only with processes on the node on
    which scontrol is run, and only for those processes spawned by Slurm and
    their descendants. Note that some Slurm configurations
    (\fIProctrackType\fP value of \fIpgid\fP)
    are unable to identify all processes associated with a job or job step.

    Note that the NodeName option is only really useful when you have multiple
    slurmd daemons running on the same host machine.  Multiple slurmd daemons on
    one host are, in general, only used by Slurm developers.

    .TP
    \fBping\fP
    Ping the primary and secondary slurmctld daemon and report if
    they are responding.

    .TP
    \fBreboot\fP [ASAP] [nextstate=<RESUME|DOWN>] [reason=<reason>] <\fIALL|\
    NodeList\fP>
    Reboot the nodes in the system when they become idle using the
    \fBRebootProgram\fP as configured in Slurm's slurm.conf file.
    Each node will have the "REBOOT" flag added to its node state.
    After a node reboots and the slurmd daemon starts up again, the
    HealthCheckProgram will run once. Then, the slurmd daemon will register
    itself with the slurmctld daemon and the "REBOOT" flag will be cleared.
    If the node reason is "Reboot ASAP", Slurm will clear the node's "DRAIN"
    state flag as well.
    The "ASAP" option adds the "DRAIN" flag to each node's state, preventing
    additional jobs from running on the node so it can be
    rebooted and returned to service "As Soon As Possible" (i.e. ASAP).
    "ASAP" will also set the node reason to "Reboot ASAP" if the "reason" option
    isn't specified.
    If the "nextstate" option is specified
    as "DOWN", then the node will remain in a down state after rebooting. If
    "nextstate" is specified as "RESUME", then the nodes will resume as normal
    when the node registers and the node reason will be cleared.
    Resuming nodes will be considered as available in
    backfill future scheduling and won't be replaced by idle nodes in a reservation.
    When using the "nextstate" and "reason" options together the reason will be
    appended with "reboot issued" when the reboot is issued and "reboot complete"
    when the node registers with a "nextstate" of "DOWN".
    The "reason" option sets each node's reason to a user-defined message.
    You must specify either a list of nodes or that ALL nodes are to be rebooted.
    NOTE: By default, this command does not prevent additional jobs from being
    Prolog, SlurmctldLogFile, SlurmdLogFile, etc.).
    The Slurm controller (slurmctld) forwards the request all other daemons
    (slurmd daemon on each compute node). Running jobs continue execution.
    Most configuration parameters can be changed by just running this command,
    however, Slurm daemons should be shutdown and restarted if any of these
    parameters are to be changed: AuthType, BackupAddr, BackupController,
    ControlAddr, ControlMach, PluginDir, StateSaveLocation, SlurmctldPort
    or SlurmdPort. The slurmctld daemon and all slurmd daemons must be
    restarted if nodes are added to or removed from the cluster.

    .TP
    \fBrelease\fP \fIjob_list\fP
    Release a previously held job to begin execution.
    The job_list argument is a comma separated list of job IDs OR
    "jobname=" with the job's name, which will attempt to hold all jobs having
    that name.
    Also see \fBhold\fR.

    .TP
    \fBrequeue\f  [option] \fIjob_list\fP
    Requeue a running, suspended or finished Slurm batch job into pending state.
    The job_list argument is a comma separated list of job IDs.
    The command accepts the following option:
    .RS
    .TP
    \fIIncomplete\fP
    Operate only on jobs (or tasks of a job array) which have not completed.
    Specifically only jobs in the following states will be requeued:
    CONFIGURING, RUNNING, STOPPED or SUSPENDED.
    .RE

    .TP
    \fBrequeuehold\fP [option] \fIjob_list\fP
    Requeue a running, suspended or finished Slurm batch job into pending state,
    moreover the job is put in held state (priority zero).
    The job_list argument is a comma separated list of job IDs.
    A held job can be released using scontrol to reset its priority (e.g.
    "scontrol release <job_id>"). The command accepts the following options:
    .RS
    .TP
    \fIIncomplete\fP
    Operate only on jobs (or tasks of a job array) which have not completed.
    Specifically only jobs in the following states will be requeued:
    CONFIGURING, RUNNING, STOPPED or SUSPENDED.
    .TP
    \fIState=SpecialExit\fP
    The "SpecialExit" keyword specifies that the job
    has to be put in a special state \fBJOB_SPECIAL_EXIT\fP.
    The "scontrol show job" command will display the JobState as
    \fBSPECIAL_EXIT\fP, while the "squeue" command as \fBSE\fP.
    .RE

    following way. Before suspending set all nodes to draining or set all
    partitions to down so that no new jobs can be scheduled. Then suspend
    jobs. Once maintenance is done resume jobs then resume nodes and/or set all
    partitions back to up.
    Use with caution.

    .TP
    \fBschedloglevel\fP \fILEVEL\fP
    Enable or disable scheduler logging.
    \fILEVEL\fP may be "0", "1", "disable" or "enable". "0" has the same
    effect as "disable". "1" has the same effect as "enable".
    This value is temporary and will be overwritten when the slurmctld
    daemon reads the slurm.conf configuration file (e.g. when the daemon
    is restarted or \fBscontrol reconfigure\fR is executed) if the
    SlurmSchedLogLevel parameter is present.

    .TP
    \fBsetdebug\fP \fILEVEL\fP
    Change the debug level of the slurmctld daemon.
    \fILEVEL\fP may be an integer value between zero and nine (using the
    same values as \fISlurmctldDebug\fP in the \fIslurm.conf\fP file) or
    the name of the most detailed message type to be printed:
    "quiet", "fatal", "error", "info", "verbose", "debug", "debug2", "debug3",
    "debug4", or "debug5".
    This value is temporary and will be overwritten whenever the slurmctld
    daemon reads the slurm.conf configuration file (e.g. when the daemon
    is restarted or \fBscontrol reconfigure\fR is executed).

    .TP
    \fBsetdebugflags\fP [+|\-]\fIFLAG\fP
    Add or remove DebugFlags of the slurmctld daemon.
    See "man slurm.conf" for a list of supported DebugFlags.
    NOTE: Changing the value of some DebugFlags will have no effect without
    restarting the slurmctld daemon, which would set DebugFlags based upon the
    contents of the slurm.conf configuration file.

    .TP
    \fBshow\fP \fIENTITY\fP \fIID\fP
    or
    .TP
    \fBshow\fP \fIENTITY\fP=\fIID\fP
    Display the state of the specified entity with the specified identification.
    \fIENTITY\fP may be \fIaliases\fP, \fIassoc_mgr\fP, \fIbbstat\fP,
    \fIburstbuffer\fP, \fIconfig\fP, \fIdaemons\fP, \fIdwstat\fP,
    \fIfederation\fP, \fIfrontend\fP, \fIjob\fP, \fInode\fP,
    \fIpartition\fP, \fIpowercap\fP, \fIreservation\fP, \fIslurmd\fP, \fIstep\fP,
    \fItopology\fP, \fIhostlist\fP, \fIhostlistsorted\fP or \fIhostnames\fP
    \fIID\fP can be used to identify a specific element of the identified
    entity: job ID, node name, partition name, reservation name, or job step ID for
    \fIjob\fP, \fInode\fP, \fIpartition\fP, or \fIstep\fP respectively.
    For an \fIENTITY\fP of \fIbbstat\fP or \fIdwstat\fP (they are equivalent)
    optional arguments are the options of the local status command.
    flags=<users,assoc,qos>, used to filter the desired section to be displayed. If
    no flags are specified, all sections are displayed.
    \fIburstbuffer\fP displays the current status of the BurstBuffer plugin.
    \fIconfig\fP displays parameter names from the configuration files in mixed
    case (e.g. SlurmdPort=7003) while derived parameters names are in upper case
    only (e.g. SLURM_VERSION).
    \fIhostnames\fP takes an optional hostlist expression as input and
    writes a list of individual host names to standard output (one per
    line). If no hostlist expression is supplied, the contents of the
    SLURM_JOB_NODELIST environment variable is used. For example "tux[1\-3]"
    is mapped to "tux1","tux2" and "tux3" (one hostname per line).
    \fIhostlist\fP takes a list of host names and prints the hostlist
    expression for them (the inverse of \fIhostnames\fP).
    \fIhostlist\fP can also take the absolute pathname of a file
    (beginning with the character '/') containing a list of hostnames.
    Multiple node names may be specified using simple node range expressions
    (e.g. "lx[10\-20]"). All other \fIID\fP values must identify a single
    element. The job step ID is of the form "job_id.step_id", (e.g. "1234.1").
    \fIslurmd\fP reports the current status of the slurmd daemon executing
    on the same node from which the scontrol command is executed (the
    local host). It can be useful to diagnose problems.
    By default \fIhostlist\fP does not sort the node list or make it
    unique (e.g. tux2,tux1,tux2 = tux[2,1-2]).  If you wanted a sorted
    list use \fIhostlistsorted\fP (e.g. tux2,tux1,tux2 = tux[1-2,2]).
    By default, all elements of the entity type specified are printed.
    For an \fIENTITY\fP of \fIjob\fP, if the job does not specify
    socket-per-node, cores-per-socket or threads-per-core then it
    will display '*' in ReqS:C:T=*:*:* field. For an \fIENTITY\fP of
    \fIfederation\fP, the federation name that the controller is part of and the
    sibling clusters part of the federation will be listed.

    .TP
    \fBshutdown\fP \fIOPTION\fP
    Instruct Slurm daemons to save current state and terminate.
    By default, the Slurm controller (slurmctld) forwards the request all
    other daemons (slurmd daemon on each compute node).
    An \fIOPTION\fP of \fIslurmctld\fP or \fIcontroller\fP results in
    only the slurmctld daemon being shutdown and the slurmd daemons
    remaining active.

    .TP
    \fBsuspend\fP \fIjob_list\fP
    Suspend a running job.
    The job_list argument is a comma separated list of job IDs.
    Use the \fIresume\fP command to resume its execution.
    User processes must stop on receipt of SIGSTOP signal and resume
    upon receipt of SIGCONT for this operation to be effective.
    Not all architectures and configurations support job suspension.
    If a suspended job is requeued, it will be placed in a held state.
    The time a job is suspended will not count against a job's time limit.
    Only an operator, administrator, SlurmUser, or root can suspend jobs.

    Move the specified job IDs to the top of the queue of jobs belonging to the
    identical user ID, partition name, account, and QOS.
    The job_list argument is a comma separated ordered list of job IDs.
    Any job not matching \fBall\fP of those fields will not be effected.
    Only jobs submitted to a single partition will be effected.
    This operation changes the order of jobs by adjusting job nice values.
    The net effect on that user's throughput will be negligible to slightly negative.
    This operation is disabled by default for non-privileged (non-operator, admin,
    SlurmUser, or root) users. This operation may be enabled for non-privileged
    users by the system administrator by including the option "enable_user_top" in
    the SchedulerParameters configuration parameter.

    .TP
    \fBuhold\fP \fIjob_list\fP
    Prevent a pending job from being started (sets it's priority to 0).
    The job_list argument is a space separated list of job IDs or job names.
    Use the \fIrelease\fP command to permit the job to be scheduled.
    This command is designed for a system administrator to hold a job so that
    the job owner may release it rather than requiring the intervention of a
    system administrator (also see the \fBhold\fP command).

    .TP
    \fBupdate\fP \fISPECIFICATION\fP
    Update job, step, node, partition, powercapping or reservation configuration per
    the supplied specification. \fISPECIFICATION\fP is in the same format as the
    Slurm configuration file and the output of the \fIshow\fP command described
    above. It may be desirable to execute the \fIshow\fP command (described above)
    on the specific entity you want to update, then use cut\-and\-paste tools to
    enter updated configuration values to the \fIupdate\fP. Note that while most
    configuration values can be changed using this command, not all can be changed
    using this mechanism. In particular, the hardware configuration of a node or
    the physical addition or removal of nodes from the cluster may only be
    accomplished through editing the Slurm configuration file and executing
    the \fIreconfigure\fP command (described above).

    .TP
    \fBversion\fP
    Display the version number of scontrol being executed.

    .TP
    \fBwait_job\fP \fIjob_id\fP
    Wait until a job and all of its nodes are ready for use or the job has entered
    some termination state. This option is particularly useful in the Slurm Prolog
    or in the batch script itself if nodes are powered down and restarted
    automatically as needed.

    \fBNOTE:\fP Don't use scontrol wait_job in PrologCtld or Prolog with
    PrologFlags=Alloc as this will result in deadlock.

    .TP
    \fBwrite batch_script\fP \fIjob_id optional_filename\fP
    Write the batch script for a given job_id to a file or to stdout. The file will
    All commands listed below can be used in the interactive mode, but \fINOT\fP
    on the initial command line.

    .TP
    \fBall\fP
    Show all partitions, their jobs and jobs steps. This causes information to be
    displayed about partitions that are configured as hidden and partitions that
    are unavailable to user's group.

    .TP
    \fBdetails\fP
    Causes the \fIshow\fP command to provide additional details where available.
    Job information will include CPUs and NUMA memory allocated on each node.
    Note that on computers with hyperthreading enabled and Slurm configured to
    allocate cores, each listed CPU represents one physical core.
    Each hyperthread on that core can be allocated a separate task, so a job's
    CPU count and task count may differ.
    See the \fB\-\-cpu\-bind\fR and \fB\-\-mem\-bind\fR option descriptions in
    srun man pages for more information.
    The \fBdetails\fP option is currently only supported for the \fIshow job\fP
    command.

    .TP
    \fBexit\fP
    Terminate scontrol interactive session.

    .TP
    \fBhide\fP
    Do not display partition, job or jobs step information for partitions that are
    configured as hidden or partitions that are unavailable to the user's group.
    This is the default behavior.

    .TP
    \fBoneliner\fP
    Print information one line per record.

    .TP
    \fBquiet\fP
    Print no warning or informational messages, only fatal error messages.

    .TP
    \fBquit\fP
    Terminate the execution of scontrol.

    .TP
    \fBverbose\fP
    Print detailed event logging.
    This includes time\-stamps on data structures, record counts, etc.

    .TP
    \fB!!\fP
    Repeat the last command executed.
    .TP
    \fIAdminComment\fP=<spec>
    Arbitrary descriptive string. Can only be set by a Slurm administrator.
    .TP
    \fIArrayTaskThrottle\fP=<count>
    Specify the maximum number of tasks in a job array that can execute at
    the same time.
    Set the count to zero in order to eliminate any limit.
    The task throttle count for a job array is reported as part of its ArrayTaskId
    field, preceded with a percent sign.
    For example "ArrayTaskId=1\-10%2" indicates the maximum number of running tasks
    is limited to 2.
    .TP
    \fIBurstBuffer\fP=<spec>
    Burst buffer specification to be changed for this job's resource use.
    Value may be cleared with blank data value, "BurstBuffer=".
    Format is burst buffer plugin specific.
    .TP
    \fIClusters\fP=<spec>
    Specifies the clusters that the federated job can run on.
    .TP
    \fIClusterFeatures\fP=<spec>
    Specifies features that a federated cluster must have to have a sibling job
    submitted to it. Slurm will attempt to submit a sibling job to a cluster if it
    has at least one of the specified features.
    .TP
    \fIComment\fP=<spec>
    Arbitrary descriptive string.
    .TP
    \fIContiguous\fP=<yes|no>
    Set the job's requirement for contiguous (consecutive) nodes to be allocated.
    Possible values are "YES" and "NO".
    Only the Slurm administrator or root can change this parameter.
    .TP
    \fICoreSpec\fP=<count>
    Number of cores to reserve per node for system use.
    The job will be charged for these cores, but be unable to use them.
    Will be reported as "*" if not constrained.
    .TP
    \fICPUsPerTask\fP=<count>
    Change the CPUsPerTask job's value.
    .TP
    \fIDeadline\fP=<time_spec>
    It accepts times of the form \fIHH:MM:SS\fR to specify a deadline to a job at
    a specific time of day (seconds are optional).
    You may also specify \fImidnight\fR, \fInoon\fR, \fIfika\fR (3 PM) or
    \fIteatime\fR (4 PM) and you can have a time\-of\-day suffixed
    with \fIAM\fR or \fIPM\fR for a deadline in the morning or the evening.
    You can specify a deadline for the job with
    a date of the form \fIMMDDYY\fR or \fIMM/DD/YY\fR or \fIMM.DD.YY\fR,
    or a date and time as \fIYYYY\-MM\-DD[THH:MM[:SS]]\fR.  You can also
    give times like \fInow + count time\-units\fR, where the time\-units
    is satisfied.
    Cancel dependency with an empty dependency_list (e.g. "Dependency=").
    <\fIdependency_list\fR> is of the form
    <\fItype:job_id[:job_id][,type:job_id[:job_id]]\fR>.
    Many jobs can share the same dependency and these jobs may even belong to
    different  users.
    .PD
    .RS
    .TP
    \fBafter:job_id[:jobid...]\fR
    This job can begin execution after the specified jobs have begun
    execution.
    .TP
    \fBafterany:job_id[:jobid...]\fR
    This job can begin execution after the specified jobs have terminated.
    .TP
    \fBafternotok:job_id[:jobid...]\fR
    This job can begin execution after the specified jobs have terminated
    in some failed state (non-zero exit code, node failure, timed out, etc).
    .TP
    \fBafterok:job_id[:jobid...]\fR
    This job can begin execution after the specified jobs have successfully
    executed (ran to completion with an exit code of zero).
    .TP
    \fBsingleton\fR
    This job can begin execution after any previously launched jobs
    sharing the same job name and user have terminated.
    In other words, only one job by that name and owned by that user can be running
    or suspended at any point in time.
    .RE
    .TP
    \fIEligibleTime\fP=<time_spec>
    See \fIStartTime\fP.
    .TP
    \fIEndTime\fP
    The time the job is expected to terminate based on the job's time
    limit.  When the job ends sooner, this field will be updated with the
    actual end time.
    .TP
    \fIExcNodeList\fP=<nodes>
    Set the job's list of excluded node. Multiple node names may be
    specified using simple node range expressions (e.g. "lx[10\-20]").
    Value may be cleared with blank data value, "ExcNodeList=".
    .TP
    \fIFeatures\fP=<features>
    Set the job's required node features.
    The list of features may include multiple feature names separated
    by ampersand (AND) and/or vertical bar (OR) operators.
    For example: \fBFeatures="opteron&video"\fR or \fBFeatures="fast|faster"\fR.
    In the first example, only nodes having both the feature "opteron" AND
    the feature "video" will be used.
    There is no mechanism to specify that you want one node with feature
    "a2a", and "flat" plus 2 nodes with the feature "haswell".
    Constraints with node counts may only be combined with AND operators.
    Value may be cleared with blank data value, for example "Features=".

    .TP
    \fIGres\fP=<list>
    Specifies a comma delimited list of generic consumable resources.
    The format of each entry on the list is "name[:count[*cpu]]".
    The name is that of the consumable resource.
    The count is the number of those resources with a default value of 1.
    The specified resources will be allocated to the job on each node
    allocated unless "*cpu" is appended, in which case the resources
    will be allocated on a per cpu basis.
    The available generic consumable resources is configurable by the system
    administrator.
    A list of available generic consumable resources will be printed and the
    command will exit if the option argument is "help".
    Examples of use include "Gres=gpus:2*cpu,disk=40G" and "Gres=help".

    .TP
    \fIJobId\fP=<job_list>
    Identify the job(s) to be updated.
    The job_list may be a comma separated list of job IDs.
    Either \fIJobId\fP or \fIJobName\fP is required.
    .TP
    \fILicenses\fP=<name>
    Specification of licenses (or other resources available on all nodes
    of the cluster) as described in salloc/sbatch/srun man pages.
    .TP
    \fIMinCPUsNode\fP=<count>
    Set the job's minimum number of CPUs per node to the specified value.
    .TP
    \fIMinMemoryCPU\fP=<megabytes>
    Set the job's minimum real memory required per allocated CPU to the specified
    value. Either \fIMinMemoryCPU\fP or \fIMinMemoryNode\fP may be set, but not both.
    .TP
    \fIMinMemoryNode\fP=<megabytes>
    Set the job's minimum real memory required per node to the specified value.
    Either \fIMinMemoryCPU\fP or \fIMinMemoryNode\fP may be set, but not both.
    .TP
    \fIMinTmpDiskNode\fP=<megabytes>
    Set the job's minimum temporary disk space required per node to the specified value.
    Only the Slurm administrator or root can change this parameter.
    .TP
    \fTimeMin\fP=<timespec>
    Change TimeMin value which specifies the minimum time limit minutes of the job.
    .TP
    \fIJobName\fP=<name>
    Identify the name of jobs to be modified or set the job's name to the
    specified value.
    When used to identify jobs to be modified, all jobs belonging to all users
    are modified unless the \fIUserID\fP option is used to identify a specific user.
    allocated to the job. Multiple node names may be specified using
    simple node range expressions (e.g. "lx[10\-20]"). After a job's allocation
    is reduced, subsequent \fBsrun\fR commands must explicitly specify node and
    task counts which are valid for the new allocation.
    .TP
    \fINumCPUs\fP=<min_count>[\-<max_count>]
    Set the job's minimum and optionally maximum count of CPUs to be allocated.
    .TP
    \fINumNodes\fP=<min_count>[\-<max_count>]
    Set the job's minimum and optionally maximum count of nodes to be allocated.
    If the job is already running, use this to specify a node count less than
    currently allocated and resources previously allocated to the job will be
    relinquished. After a job's allocation is reduced, subsequent \fBsrun\fR
    commands must explicitly specify node and task counts which are valid for the
    new allocation. Also see the \fINodeList\fP parameter above. This is the same
    than ReqNodes.
    .TP
    \fINumTasks\fP=<count>
    Set the job's count of required tasks to the specified value. This is the same
    than ReqProcs.
    .TP
    \fIOverSubscribe\fP=<yes|no>
    Set the job's ability to share compute resources (i.e. individual CPUs)
    with other jobs. Possible values are "YES" and "NO".
    This option can only be changed for pending jobs.
    .TP
    \fIPartition\fP=<name>
    Set the job's partition to the specified value.
    .TP
    \fIPriority\fP=<number>
    Set the job's priority to the specified value.
    Note that a job priority of zero prevents the job from ever being scheduled.
    By setting a job's priority to zero it is held.
    Set the priority to a non\-zero value to permit it to run.
    Explicitly setting a job's priority clears any previously set nice value and
    removes the priority/multifactor plugin's ability to manage a job's priority.
    In order to restore the priority/multifactor plugin's ability to manage a
    job's priority, hold and then release the job.
    Only the Slurm administrator or root can increase job's priority.
    .TP
    \fIQOS\fP=<name>
    Set the job's QOS (Quality Of Service) to the specified value.
    Value may be cleared with blank data value, "QOS=".
    .TP
    \fBReboot\fP=<yes|no>
    Set the job's flag that specifies whether to force the allocated nodes to reboot
    before starting the job. This is only supported with some system configurations
    and therefore it could be silently ignored.
    .TP
    \fIReqCores\fP=<count>
    Change the job's requested Cores count.
    .TP
    \fIReqThreads\fP=<count>
    Change the job's requested threads count.
    .TP
    \fIRequeue\fP=<0|1>
    Stipulates whether a job should be requeued after a node failure: 0
    for no, 1 for yes.
    .TP
    \fIReservationName\fP=<name>
    Set the job's reservation to the specified value.
    Value may be cleared with blank data value, "ReservationName=".
    .TP
    \fIResetAccrueTime\fP
    Reset the job's accrue time value to 0 meaning it will loose any time
    previously accrued for priority.  Helpful if you have a large queue of jobs
    already in the queue and want to start limiting how many jobs can accrue time
    without waiting for the queue to flush out.
    .TP
    \fISiteFactor\fP=<account>
    Specify the job's admin priority factor in the range of +/\-2147483645.
    Only privileged users can modify the value.
    .TP
    \fIStdOut\fP=<filepath>
    Set the batch job's stdout file path.
    .TP
    \fIShared\fP=<yes|no>
    See \fIOverSubscribe\fP option above.
    .TP
    \fIStartTime\fP=<time_spec>
    Set the job's earliest initiation time.
    It accepts times of the form \fIHH:MM:SS\fR to run a job at
    a specific time of day (seconds are optional).
    (If that time is already past, the next day is assumed.)
    You may also specify \fImidnight\fR, \fInoon\fR, \fIfika\fR (3 PM) or
    \fIteatime\fR (4 PM) and you can have a time\-of\-day suffixed
    with \fIAM\fR or \fIPM\fR for running in the morning or the evening.
    You can also say what day the job will be run, by specifying
    a date of the form \fIMMDDYY\fR or \fIMM/DD/YY\fR or \fIMM.DD.YY\fR,
    or a date and time as \fIYYYY\-MM\-DD[THH:MM[:SS]]\fR.  You can also
    give times like \fInow + count time\-units\fR, where the time\-units
    can be \fIminutes\fR, \fIhours\fR, \fIdays\fR, or \fIweeks\fR
    and you can tell Slurm to run the job today with the keyword
    \fItoday\fR and to run the job tomorrow with the keyword
    \fItomorrow\fR.
    .RS
    .PP
    Notes on date/time specifications:
     \- although the 'seconds' field of the HH:MM:SS time specification is
    allowed by the code, note that the poll time of the Slurm scheduler
    is not precise enough to guarantee dispatch of the job on the exact
    second.  The job will be eligible to start on the next poll
    following the specified time. The exact poll interval depends on the
    Slurm scheduler (e.g., 60 seconds with the default sched/builtin).
    the \fBSchedulerParameters\fR configuration parameter with the
    \fBmax_switch_wait\fR parameter option.
    Also see \fIwait\-for\-switch\fP.

    .TP
    \fIwait\-for\-switch\fP=<seconds>
    Change max time to wait for a switch <seconds> secs.

    .TP
    \fITasksPerNode\fP=<count>
    Change the job's requested TasksPerNode.

    .TP
    \fIThreadSpec\fP=<count>
    Number of threads to reserve per node for system use.
    The job will be charged for these threads, but be unable to use them.
    Will be reported as "*" if not constrained.

    .TP
    \fITimeLimit\fP=<time>
    The job's time limit.
    Output format is [days\-]hours:minutes:seconds or "UNLIMITED".
    Input format (for \fBupdate\fR command) set is minutes, minutes:seconds,
    hours:minutes:seconds, days\-hours, days\-hours:minutes or
    days\-hours:minutes:seconds.
    Time resolution is one minute and second values are rounded up to
    the next minute.
    If changing the time limit of a job, either specify a new time limit value or
    precede the time and equal sign with a "+" or "\-" to increment or decrement the
    current time limit (e.g. "TimeLimit+=30"). In order to increment or decrement
    the current time limit, the \fIJobId\fP specification must precede the
    \fITimeLimit\fP specification.
    Note that incrementing or decrementing the time limit for a job array is only
    allowed before the job array has been split into more than one job record.
    Only the Slurm administrator or root can increase job's TimeLimit.

    .TP
    \fIUserID\fP=<UID or name>
    Used with the \fIJobName\fP option to identify jobs to be modified.
    Either a user name or numeric ID (UID), may be specified.

    .TP
    \fIWCKey\fP=<key>
    Set the job's workload characterization key to the specified value.

    .TP
    \fIWorkDir\fP=<directory_name>
    Set the job's working directory to the specified value. Note that this may
    only be set for jobs in the PENDING state, and that jobs may fail to launch
    if they rely on relative paths to the originally submitted WorkDir.

    .TP
    Jobs submitted using the sbatch command have BatchFlag set to 1.
    Jobs submitted using other commands have BatchFlag set to 0.
    .TP
    \fIExitCode\fP=<exit>:<sig>
    Exit status reported for the job by the wait() function.
    The first number is the exit code, typically as set by the exit() function.
    The second number of the signal that caused the process to terminate if
    it was terminated by a signal.
    .TP
    \fIGroupId\fP
    The group under which the job was submitted.
    .TP
    \fIJobState\fP
    The current state of the job.
    .TP
    \fINodeListIndices\fP
    The NodeIndices expose the internal indices into the node table
    associated with the node(s) allocated to the job.
    .TP
    .na
    \fINtasksPerN:B:S:C\fP=
    fR(1), \fBsinfo\fR(1), \fBsqueue\fR(1),
    \fBslurm_checkpoint\fR (3),
    \fBslurm_create_partition\fR (3),
    \fBslurm_delete_partition\fR (3),
    \fBslurm_load_ctl_conf\fR (3),
    \fBslurm_load_jobs\fR (3), \fBslurm_load_node\fR (3),
    \fBslurm_load_partitions\fR (3),
    \fBslurm_reconfigure\fR (3),  \fBslurm_requeue\fR (3),
    \fBslurm_resume\fR (3),
    \fBslurm_shutdown\fR (3), \fBslurm_suspend\fR (3),
    \fBslurm_takeover\fR (3),
    \fBslurm_update_job\fR (3), \fBslurm_update_node\fR (3),
    \fBslurm_update_partition\fR (3),
    \fBslurm.conf\fR(5), \fBslurmctld\fR(8)