    Option(s) define multiple jobs in a co-scheduled heterogeneous job.
    For more details about heterogeneous jobs see the document
    .br
    https://slurm.schedmd.com/heterogeneous_jobs.html

    .SH "DESCRIPTION"
    sbatch submits a batch script to Slurm.  The batch script may be given to
    sbatch through a file name on the command line, or if no file name is specified,
    sbatch will read in a script from standard input. The batch script may contain
    options preceded with "#SBATCH" before any executable commands in the script.
    sbatch will stop processing further #SBATCH directives once the first
    non-comment non-whitespace line has been reached in the script.

    sbatch exits immediately after the script is successfully transferred to the
    Slurm controller and assigned a Slurm job ID.  The batch script is not
    necessarily granted resources immediately, it may sit in the queue of pending
    jobs for some time before its required resources become available.

    By default both standard output and standard error are directed to a file of
    the name "slurm\-%j.out", where the "%j" is replaced with the job allocation
    number. The file will be generated on the first node of the job allocation.
    Other than the batch script itself, Slurm does no movement of user files.

    When the job allocation is finally granted for the batch script, Slurm
    runs a single copy of the batch script on the first node in the set of
    allocated nodes.

    The following document describes the influence of various options on the
    allocation of cpus to jobs and tasks.
    .br
    https://slurm.schedmd.com/cpu_management.html

    .SH "RETURN VALUE"
    sbatch will return 0 on success or error code on failure.

    .SH "SCRIPT PATH RESOLUTION"

    .nr step 1 1
    The batch script is resolved in the following order:
    .br
    .IP \n[step]. 3
    If script starts with ".", then path is constructed as:
    current working directory / script
    .IP \n+[step].
    If script starts with a "/", then path is considered absolute.
    .IP \n+[step].
    If script is in current working directory.
    .IP \n+[step].
    If script can be resolved through PATH. See \fBpath_resolution\fR(7).
    .P
    Current working directory is the calling process working directory unless the
    number. For example, "\-\-array=0\-15:4" is equivalent to "\-\-array=0,4,8,12".
    A maximum number of simultaneously running tasks from the job array may be
    specified using a "%" separator.
    For example "\-\-array=0\-15%4" will limit the number of simultaneously
    running tasks from this job array to 4.
    The minimum index value is 0.
    the maximum value is one less than the configuration parameter MaxArraySize.
    NOTE: currently, federated job arrays only run on the local cluster.

    .TP
    \fB\-A\fR, \fB\-\-account\fR=<\fIaccount\fR>
    Charge resources used by this job to specified account.
    The \fIaccount\fR is an arbitrary string. The account name may
    be changed after job submission using the \fBscontrol\fR
    command.

    .TP
    \fB\-\-acctg\-freq\fR
    Define the job accounting and profiling sampling intervals.
    This can be used to override the \fIJobAcctGatherFrequency\fR parameter in Slurm's
    configuration file, \fIslurm.conf\fR.
    The supported format is as follows:
    .RS
    .TP 12
    \fB\-\-acctg\-freq=\fR\fI<datatype>\fR\fB=\fR\fI<interval>\fR
    where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling
    interval for the jobacct_gather plugin or a
    sampling interval for a profiling type by the
    acct_gather_profile plugin. Multiple,
    comma-separated \fI<datatype>\fR=\fI<interval>\fR intervals
    may be specified. Supported datatypes are as follows:
    .RS
    .TP
    \fBtask=\fI<interval>\fR
    where \fI<interval>\fR is the task sampling interval in seconds
    for the jobacct_gather plugins and for task
    profiling by the acct_gather_profile plugin.
    NOTE: This frequency is used to monitor memory usage. If memory limits
    are enforced the highest frequency a user can request is what is configured in
    the slurm.conf file.  They can not turn it off (=0) either.
    .TP
    \fBenergy=\fI<interval>\fR
    where \fI<interval>\fR is the sampling interval in seconds
    for energy profiling using the acct_gather_energy plugin
    .TP
    \fBnetwork=\fI<interval>\fR
    where \fI<interval>\fR is the sampling interval in seconds
    for infiniband profiling using the acct_gather_infiniband
    plugin.
    .TP
    \fBfilesystem=\fI<interval>\fR
    where \fI<interval>\fR is the sampling interval in seconds
    Smaller (non\-zero) values have a greater impact upon job performance,
    but a value of 30 seconds is not likely to be noticeable for
    applications having less than 10,000 tasks.
    .RE

    .TP
    \fB\-B\fR \fB\-\-extra\-node\-info\fR=<\fIsockets\fR[:\fIcores\fR[:\fIthreads\fR]]>
    Restrict node selection to nodes with at least the specified number of
    sockets, cores per socket and/or threads per core.
    NOTE: These options do not specify the resource allocation size.
    Each value specified is considered a minimum.
    An asterisk (*) can be used as a placeholder indicating that all available
    resources of that type are to be utilized. Values can also be specified as
    min-max. The individual levels can also be specified in separate options if
    desired:
    .nf
        \fB\-\-sockets\-per\-node\fR=<\fIsockets\fR>
        \fB\-\-cores\-per\-socket\fR=<\fIcores\fR>
        \fB\-\-threads\-per\-core\fR=<\fIthreads\fR>
    .fi
    If task/affinity plugin is enabled, then specifying an allocation in this
    manner also results in subsequently launched tasks being bound to threads
    if the \fB\-B\fR option specifies a thread count, otherwise an option of
    \fIcores\fR if a core count is specified, otherwise an option of \fIsockets\fR.
    If SelectType is configured to select/cons_res, it must have a parameter of
    CR_Core, CR_Core_Memory, CR_Socket, or CR_Socket_Memory for this option
    to be honored.
    If not specified, the scontrol show job will display 'ReqS:C:T=*:*:*'. This
    option applies to job allocations.

    .TP
    \fB\-\-batch\fR=<\fIlist\fR>
    Nodes can have \fBfeatures\fR assigned to them by the Slurm administrator.
    Users can specify which of these \fBfeatures\fR are required by their batch
    script using this options.
    For example a job's allocation may include both Intel Haswell and KNL nodes
    with features "haswell" and "knl" respectively.
    On such a configuration the batch script would normally benefit by executing
    on a faster Haswell node.
    This would be specified using the option "\-\-batch=haswell".
    The specification can include AND and OR operators using the ampersand and
    vertical bar separators. For example:
    "\-\-batch=haswell|broadwell" or "\-\-batch=haswell|big_memory".
    The \-\-batch argument must be a subset of the job's
    \fB\-\-constraint\fR=<\fIlist\fR> argument (i.e. the job can not request only
    KNL nodes, but require the script to execute on a Haswell node).
    If the request can not be satisfied from the resources allocated to the job,
    the batch script will execute on the first node of the job allocation.

    .TP
    \fB\-\-bb\fR=<\fIspec\fR>
    Burst buffer specification. The form of the specification is system dependent.
    tell the controller to defer the allocation of the job until the specified time.

    Time may be of the form \fIHH:MM:SS\fR to run a job at
    a specific time of day (seconds are optional).
    (If that time is already past, the next day is assumed.)
    You may also specify \fImidnight\fR, \fInoon\fR, \fIfika\fR (3 PM) or
    \fIteatime\fR (4 PM) and you can have a time\-of\-day suffixed
    with \fIAM\fR or \fIPM\fR for running in the morning or the evening.
    You can also say what day the job will be run, by specifying
    a date of the form \fIMMDDYY\fR or \fIMM/DD/YY\fR
    \fIYYYY\-MM\-DD\fR. Combine date and time using the following
    format \fIYYYY\-MM\-DD[THH:MM[:SS]]\fR. You can also
    give times like \fInow + count time\-units\fR, where the time\-units
    can be \fIseconds\fR (default), \fIminutes\fR, \fIhours\fR,
    \fIdays\fR, or \fIweeks\fR and you can tell Slurm to run
    the job today with the keyword \fItoday\fR and to run the
    job tomorrow with the keyword \fItomorrow\fR.
    The value may be changed after job submission using the
    \fBscontrol\fR command.
    For example:
    .nf
       \-\-begin=16:00
       \-\-begin=now+1hour
       \-\-begin=now+60           (seconds by default)
       \-\-begin=2010\-01\-20T12:34:00
    .fi

    .RS
    .PP
    Notes on date/time specifications:
     \- Although the 'seconds' field of the HH:MM:SS time specification is
    allowed by the code, note that the poll time of the Slurm scheduler
    is not precise enough to guarantee dispatch of the job on the exact
    second.  The job will be eligible to start on the next poll
    following the specified time. The exact poll interval depends on the
    Slurm scheduler (e.g., 60 seconds with the default sched/builtin).
     \- If no time (HH:MM:SS) is specified, the default is (00:00:00).
     \- If a date is specified without a year (e.g., MM/DD) then the current
    year is assumed, unless the combination of MM/DD and HH:MM:SS has
    already passed for that year, in which case the next year is used.
    .RE

    .TP
    \fB\-\-checkpoint\fR=<\fItime\fR>
    Specifies the interval between creating checkpoints of the job step.
    By default, the job step will have no checkpoints created.
    Acceptable time formats include "minutes", "minutes:seconds",
    "hours:minutes:seconds", "days\-hours", "days\-hours:minutes" and
    "days\-hours:minutes:seconds".

    .TP
    \fB\-\-cluster\-constraint\fR=[!]<\fIlist\fR>
    Users can specify which of these \fBfeatures\fR are required by their job
    using the constraint option.
    Only nodes having features matching the job constraints will be used to
    satisfy the request.
    Multiple constraints may be specified with AND, OR, matching OR,
    resource counts, etc. (some operators are not supported on all system types).
    Supported \fbconstraint\fR options include:
    .PD 1
    .RS
    .TP
    \fBSingle Name\fR
    Only nodes which have the specified feature will be used.
    For example, \fB\-\-constraint="intel"\fR
    .TP
    \fBNode Count\fR
    A request can specify the number of nodes needed with some feature
    by appending an asterisk and count after the feature name.
    For example "\fB\-\-nodes=16 \-\-constraint=graphics*4 ..."\fR
    indicates that the job requires 16 nodes and that at least four of those
    nodes must have the feature "graphics."
    .TP
    \fBAND\fR
    If only nodes with all of specified features will be used.
    The ampersand is used for an AND operator.
    For example, \fB\-\-constraint="intel&gpu"\fR
    .TP
    \fBOR\fR
    If only nodes with at least one of specified features will be used.
    The vertical bar is used for an OR operator.
    For example, \fB\-\-constraint="intel|amd"\fR
    .TP
    \fBMatching OR\fR
    If only one of a set of possible options should be used for all allocated
    nodes, then use the OR operator and enclose the options within square brackets.
    For example: "\fB\-\-constraint=[rack1|rack2|rack3|rack4]"\fR might
    be used to specify that all nodes must be allocated on a single rack of
    the cluster, but any of those four racks can be used.
    .TP
    \fBMultiple Counts\fR
    Specific counts of multiple resources may be specified by using the AND
    operator and enclosing the options within square brackets.
    For example: "\fB\-\-constraint=[rack1*2&rack2*4]"\fR might
    be used to specify that two nodes must be allocated from nodes with the feature
    of "rack1" and four nodes must be allocated from nodes with the feature
    "rack2".

    \fBNOTE:\fR This construct does not support multiple Intel KNL NUMA or MCDRAM
    modes. For example, while "\fB\-\-constraint=[(knl&quad)*2&(knl&hemi)*4]"\fR is
    not supported, "\fB\-\-constraint=[haswell*2&(knl&hemi)*4]"\fR is supported.
    Specification of multiple KNL modes requires the use of a heterogeneous job.

    .TP
    .TP
    \fB\-\-cores\-per\-socket\fR=<\fIcores\fR>
    Restrict node selection to nodes with at least the specified number of
    cores per socket.  See additional information under \fB\-B\fR option
    above when task/affinity plugin is enabled.

    .TP
    \fB\-\-cpu\-freq\fR =<\fIp1\fR[\-\fIp2\fR[:\fIp3\fR]]>

    Request that job steps initiated by srun commands inside this sbatch script
    be run at some requested frequency if possible, on the CPUs selected
    for the step on the compute node(s).

    \fBp1\fR can be  [#### | low | medium | high | highm1] which will set the
    frequency scaling_speed to the corresponding value, and set the frequency
    scaling_governor to UserSpace. See below for definition of the values.

    \fBp1\fR can be [Conservative | OnDemand | Performance | PowerSave] which
    will set the scaling_governor to the corresponding value. The governor has to be
    in the list set by the slurm.conf option CpuFreqGovernors.

    When \fBp2\fR is present, p1 will be the minimum scaling frequency and
    p2 will be the maximum scaling frequency.

    \fBp2\fR can be  [#### | medium | high | highm1] p2 must be greater than p1.

    \fBp3\fR can be [Conservative | OnDemand | Performance | PowerSave | UserSpace]
    which will set the governor to the corresponding value.

    If \fBp3\fR is UserSpace, the frequency scaling_speed will be set by a power
    or energy aware scheduling strategy to a value between p1 and p2 that lets the
    job run within the site's power goal. The job may be delayed if p1 is higher
    than a frequency that allows the job to run within the goal.

    If the current frequency is < min, it will be set to min. Likewise,
    if the current frequency is > max, it will be set to max.

    Acceptable values at present include:
    .RS
    .TP 14
    \fB####\fR
    frequency in kilohertz
    .TP
    \fBLow\fR
    the lowest available frequency
    .TP
    \fBHigh\fR
    the highest available frequency
    .TP
    \fBHighM1\fR
    (high minus one) will select the next highest available frequency
    .TP
    .TP
    \fBUserSpace\fR
    attempts to use the UserSpace CPU governor
    .TP
    .RE

    The following informational environment variable is set in the job
    step when \fB\-\-cpu\-freq\fR option is requested.
    .nf
            SLURM_CPU_FREQ_REQ
    .fi

    This environment variable can also be used to supply the value for the
    CPU frequency request if it is set when the 'srun' command is issued.
    The \fB\-\-cpu\-freq\fR on the command line will override the
    environment variable value.  The form on the environment variable is
    the same as the command line.
    See the \fBENVIRONMENT VARIABLES\fR
    section for a description of the SLURM_CPU_FREQ_REQ variable.

    \fBNOTE\fR: This parameter is treated as a request, not a requirement.
    If the job step's node does not support setting the CPU frequency, or
    the requested value is outside the bounds of the legal frequencies, an
    error is logged, but the job step is allowed to continue.

    \fBNOTE\fR: Setting the frequency for just the CPUs of the job step
    implies that the tasks are confined to those CPUs.  If task
    confinement (i.e., TaskPlugin=task/affinity or
    TaskPlugin=task/cgroup with the "ConstrainCores" option) is not
    configured, this parameter is ignored.

    \fBNOTE\fR: When the step completes, the frequency and governor of each
    selected CPU is reset to the previous values.

    \fBNOTE\fR: When submitting jobs with  the \fB\-\-cpu\-freq\fR option
    with linuxproc as the ProctrackType can cause jobs to run too quickly before
    Accounting is able to poll for job information. As a result not all of
    accounting information will be present.
    .RE

    .TP
    \fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR>
    Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
    allocated GPU.
    Requires the \fB\-\-gpus\fR option.
    Not compatible with the \fB\-\-cpus\-per\-task\fR option.

    .TP
    \fB\-c\fR, \fB\-\-cpus\-per\-task\fR=<\fIncpus\fR>
    Advise the Slurm controller that ensuing job steps will require \fIncpus\fR
    number of processors per task.  Without this option, the controller will
    just try to allocate one processor per task.
    .br
    HH:MM[:SS] [AM|PM]
    .br
    MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
    .br
    MM/DD[/YY]\-HH:MM[:SS]
    .br
    YYYY\-MM\-DD[THH:MM[:SS]]]

    .TP
    \fB\-\-delay\-boot\fR=<\fIminutes\fR>
    Do not reboot nodes in order to satisfied this job's feature specification if
    the job has been eligible to run for less than this time period.
    If the job has waited for less than the specified period, it will use only
    nodes which already have the specified features.
    The argument is in units of minutes.
    A default value may be set by a system administrator using the \fBdelay_boot\fR
    option of the \fBSchedulerParameters\fR configuration parameter in the
    slurm.conf file, otherwise the default value is zero (no delay).

    .TP
    \fB\-d\fR, \fB\-\-dependency\fR=<\fIdependency_list\fR>
    Defer the start of this job until the specified dependencies have been
    satisfied completed.
    <\fIdependency_list\fR> is of the form
    <\fItype:job_id[:job_id][,type:job_id[:job_id]]\fR> or
    <\fItype:job_id[:job_id][?type:job_id[:job_id]]\fR>.
    All dependencies must be satisfied if the "," separator is used.
    Any dependency may be satisfied if the "?" separator is used.
    Many jobs can share the same dependency and these jobs may even belong to
    different  users. The  value may be changed after job submission using the
    scontrol command.
    Once a job dependency fails due to the termination state of a preceding job,
    the dependent job will never be run, even if the preceding job is requeued and
    has a different termination state in a subsequent execution.
    .PD
    .RS
    .TP
    \fBafter:job_id[:jobid...]\fR
    This job can begin execution after the specified jobs have begun
    execution.
    .TP
    \fBafterany:job_id[:jobid...]\fR
    This job can begin execution after the specified jobs have terminated.
    .TP
    \fBafterburstbuffer:job_id[:jobid...]\fR
    This job can begin execution after the specified jobs have terminated and
    any associated burst buffer stage out operations have completed.
    .TP
    \fBaftercorr:job_id[:jobid...]\fR
    A task of this job array can begin execution after the corresponding task ID
    in the specified job has completed successfully (ran to completion with an
    .TP
    \fBsingleton\fR
    This job can begin execution after any previously launched jobs
    sharing the same job name and user have terminated.
    In other words, only one job by that name and owned by that user can be running
    or suspended at any point in time.
    .RE

    .TP
    \fB\-D\fR, \fB\-\-chdir\fR=<\fIdirectory\fR>
    Set the working directory of the batch script to \fIdirectory\fR before
    it is executed. The path can be specified as full path or relative path
    to the directory where the command is executed.

    .TP
    \fB\-e\fR, \fB\-\-error\fR=<\fIfilename pattern\fR>
    Instruct Slurm to connect the batch script's standard error directly to the
    file name specified in the "\fIfilename pattern\fR".
    By default both standard output and standard error are directed to the same file.
    For job arrays, the default file name is "slurm-%A_%a.out", "%A" is replaced
    by the job ID and "%a" with the array index.
    For other jobs, the default file name is "slurm-%j.out", where the "%j" is
    replaced by the job ID.
    See the \fBfilename pattern\fR section below for filename specification options.

    .TP
    \fB\-\-exclusive[=user|mcs]\fR
    The job allocation can not share nodes with other running jobs (or just other
    users with the "=user" option or with the "=mcs" option).
    The default shared/exclusive behavior depends on system configuration and the
    partition's \fBOverSubscribe\fR option takes precedence over the job's option.

    .TP
    \fB\-\-export\fR=<\fIenvironment variables [ALL] | NONE\fR>
    Identify which environment variables from the submission environment are
    propagated to the launched application. By default, \fIall\fR are propagated.
    Multiple environment variable names should be comma separated.
    Environment variable names may be specified to propagate the current
    value (e.g. "\-\-export=EDITOR") or specific values
    may be exported (e.g. "\-\-export=EDITOR=/bin/emacs").
    In these two examples, the \fIpropagated\fR environment will only contain the
    variable \fIEDITOR\fR, along with \fISLURM_*\fR environment variables.
    However, Slurm will then implicitly attempt to load the user's environment on
    the node where the script is being executed, as if \fI\-\-get\-user\-env\fR was
    specified. This will happen whenever \fINONE\fR or environment variables are
    specified.
    If one desires to add to the submission environment instead of
    replacing it, have the argument include \fIALL\fR
    (e.g. "\-\-export=ALL,EDITOR=/bin/emacs"). Make sure \fIALL\fR is specified
    first, since sbatch applies the environment from left to right, overwriting
    as necessary.
    Environment variables propagated from the submission environment will always
    this option, a readable file descriptor will be assumed (STDIN and
    STDOUT are not supported as valid arguments).  Otherwise a filename is
    assumed.  Export environment variables defined in <\fIfilename\fR> or
    read from <\fIfd\fR> to the job's execution environment. The
    content is one or more environment variable definitions of the form
    NAME=value, each separated by a null character.  This allows the use
    of special characters in environment definitions.

    .TP
    \fB\-F\fR, \fB\-\-nodefile\fR=<\fInode file\fR>
    Much like \-\-nodelist, but the list is contained in a file of name
    \fInode file\fR.  The node names of the list may also span multiple lines
    in the file.    Duplicate node names in the file will be ignored.
    The order of the node names in the list is not important; the node names
    will be sorted by Slurm.

    .TP
    \fB\-\-get\-user\-env\fR[=\fItimeout\fR][\fImode\fR]
    This option will tell sbatch to retrieve the
    login environment variables for the user specified in the \fB\-\-uid\fR option.
    The environment variables are retrieved by running something of this sort
    "su \- <username> \-c /usr/bin/env" and parsing the output.
    Be aware that any environment variables already set in sbatch's environment
    will take precedence over any environment variables in the user's
    login environment. Clear any environment variables before calling sbatch
    that you do not want propagated to the spawned program.
    The optional \fItimeout\fR value is in seconds. Default value is 8 seconds.
    The optional \fImode\fR value control the "su" options.
    With a \fImode\fR value of "S", "su" is executed without the "\-" option.
    With a \fImode\fR value of "L", "su" is executed with the "\-" option,
    replicating the login environment.
    If \fImode\fR not specified, the mode established at Slurm build time
    is used.
    Example of use include "\-\-get\-user\-env", "\-\-get\-user\-env=10"
    "\-\-get\-user\-env=10L", and "\-\-get\-user\-env=S".
    This option was originally created for use by Moab.

    .TP
    \fB\-\-gid\fR=<\fIgroup\fR>
    If \fBsbatch\fR is run as root, and the \fB\-\-gid\fR option is used,
    submit the job with \fIgroup\fR's group access permissions.  \fIgroup\fR
    may be the group name or the numerical group ID.

    .TP
    \fB\-G\fR, \fB\-\-gpus\fR=[<\fitype\fR>:]<\fInumber\fR>
    Specify the total number of GPUs required for the job.
    An optional GPU type specification can be supplied.
    For example "\-\-gpus=volta:3".
    Multiple options can be requested in a comma separated list, for example:
    "\-\-gpus=volta:3,kepler:1".
    See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and
    \fB\-\-gpus\-per\-task\fR options.
    \fBmap_gpu:<list>\fR
    Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
     #!/bin/sh
    .br
    host3
    .br
    host4

    .LP
    To create a heterogeneous job with 3 components, each allocating a unique set of nodes:
    .IP
    sbatch -w node[2-3] : -w node4 : -w node[5-7] work.bash
    .br
    Submitted batch job 34987

    .SH "COPYING"
    Copyright (C) 2006\-2007 The Regents of the University of California.
    Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
    .br
    Copyright (C) 2008\-2010 Lawrence Livermore National Security.
    .br
    Copyright (C) 2010\-2017 SchedMD LLC.
    .LP
    This file is part of Slurm, a resource management program.
    For details, see <https://slurm.schedmd.com/>.
    .LP
    Slurm is free software; you can redistribute it and/or modify it under
    the terms of the GNU General Public License as published by the Free
    Software Foundation; either version 2 of the License, or (at your option)
    any later version.
    .LP
    Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
    WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
    FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
    details.

    .SH "SEE ALSO"
    .LP
    \fBsinfo\fR(1), \fBsattach\fR(1), \fBsalloc\fR(1), \fBsqueue\fR(1), \fBscancel\fR(1), \fBscontrol\fR(1),
    \fBslurm.conf\fR(5), \fBsched_setaffinity\fR (2), \fBnuma\fR (3)