    .LP
    sdiag

    .SH "DESCRIPTION"
    .LP
    sdiag shows information related to slurmctld execution about: threads, agents,
    jobs, and scheduling algorithms. The goal is to obtain data from slurmctld
    behaviour helping to adjust configuration parameters or queues policies. The
    main reason behind is to know Slurm behaviour under systems with a high throughput.
    .LP
    It has two execution modes. The default mode \fB\-\-all\fR shows several counters
    and statistics explained later, and there is another execution option
    \fB\-\-reset\fR for resetting those values.
    .LP
    Values are reset at midnight UTC time by default.
    .LP
    The first block of information is related to global slurmctld execution:
    .TP
    \fBServer thread count\fR
    The number of current active slurmctld threads. A high number would mean a high
    load processing events like job submissions, jobs dispatching, jobs completing,
    etc. If this is often close to MAX_SERVER_THREADS it could point to a potential
    bottleneck.

    .TP
    \fBAgent queue size\fR
    Slurm design has scalability in mind and sending messages to thousands of nodes
    is not a trivial task. The agent mechanism helps to control communication
    between slurmctld and the slurmd daemons for a best effort. This value denotes
    the count of enqueued outgoing RPC requests in an internal retry list.

    .TP
    \fBAgent count\fR
    Number of agent threads. Each of these agent threads can create in turn a group
    of up to 2 + AGENT_THREAD_COUNT active threads at a time.

    .TP
    \fBDBD Agent queue size\fR
    Slurm queues up the messages intended for the SlurmDBD and processes them in a
    separate thread. If the SlurmDBD, or database, is down then this number will
    increase. The max queue size is calculated as:

    MAX(10000, ((max_job_cnt * 2) + (node_record_count * 4)))

    If this number begins to grow more than half of the max queue size, the slurmdbd
    and the database should be investigated immediately.

    .TP
    \fBJobs submitted\fR
    Number of jobs submitted since last reset

    .TP

    .TP
    \fBJob states ts:\fR
    Lists the timestamp of when the following job state counts were gathered.

    .TP
    \fBJobs pending:\fR
    Number of jobs pending at the given time of the time stamp above.

    .TP
    \fBJobs running:\fR
    Number of jobs running at the given time of the time stamp above.

    .TP
    \fBJobs running ts:\fR
    Time stamp of when the running job count was taken.

    .LP
    The next block of information is related to main scheduling algorithm based
    on jobs priorities. A scheduling cycle implies to get the job_write_lock lock,
    then trying to get resources for jobs pending, starting from the most priority
    one and going in descendent order. Once a job can not get the resources the
    loop keeps going but just for jobs requesting other partitions. Jobs with
    dependencies or affected  by accounts limits are not processed.

    .TP
    \fBLast cycle\fR
    Time in microseconds for last scheduling cycle.

    .TP
    \fBMax cycle\fR
    Maximum time in microseconds for any scheduling cycle since last reset.

    .TP
    \fBTotal cycles\fR
    Total run time in microseconds for all scheduling cycles since last reset.
    Scheduling is performed periodically and (depending upon configuration)
    when a job is submitted or a job is completed.

    .TP
    \fBMean cycle\fR
    Mean time in microseconds for all scheduling cycles since last reset.

    .TP
    \fBMean depth cycle\fR
    Mean of cycle depth. Depth means number of jobs processed in a scheduling cycle.

    .TP
    \fBCycles per minute\fR
    Counter of scheduling executions per minute.

    .TP
    execution cycle and for taking all the locks for too long.

    .TP
    \fBTotal backfilled jobs (since last slurm start)\fR
    Number of jobs started thanks to backfilling since last slurm start.

    .TP
    \fBTotal backfilled jobs (since last stats cycle start)\fR
    Number of jobs started thanks to backfilling since last time stats where reset.
    By default these values are reset at midnight UTC time.

    .TP
    \fBTotal backfilled heterogeneous job components\fR
    Number of heterogeneous job components started thanks to backfilling since
    last Slurm start.

    .TP
    \fBTotal cycles\fR
    Number of backfill scheduling cycles since last reset

    .TP
    \fBLast cycle when\fR
    Time when last backfill scheduling cycle happened in the format
    "weekday Month MonthDay hour:minute.seconds year"

    .TP
    \fBLast cycle\fR
    Time in microseconds of last backfill scheduling cycle.
    It counts only execution time, removing sleep time inside a scheduling cycle
    when it executes for an extended period time.
    Note that locks are released during the sleep time so that other work can
    proceed.

    .TP
    \fBMax cycle\fR
    Time in microseconds of maximum backfill scheduling cycle execution since last reset.
    It counts only execution time, removing sleep time inside a scheduling cycle
    when it executes for an extended period time.
    Note that locks are released during the sleep time so that other work can
    proceed.

    .TP
    \fBMean cycle\fR
    Mean time in microseconds of backfilling scheduling cycles since last reset.

    .TP
    \fBLast depth cycle\fR
    Number of processed jobs during last backfilling scheduling cycle. It counts
    every job even if that job can not be started due to dependencies or limits.

    .TP
    \fBLast depth cycle (try sched)\fR
    .TP
    \fBDepth Mean (try sched)\fR
    The subset of Depth Mean that the backfill scheduler attempted to schedule.

    .TP
    \fBLast queue length\fR
    Number of jobs pending to be processed by backfilling algorithm.
    A job is counted once for each partition it is queued to use.
    A pending job array will normally be counted as one job (tasks of a job array
    which have already been started/requeued or individually modified will already
    have individual job records and are each counted as a separate job).

    .TP
    \fBQueue length Mean\fR
    Mean count of jobs pending to be processed by backfilling algorithm.
    A job once for each partition it requested.
    A pending job array will normally be counted as one job (tasks of a job array
    which have already been started/requeued or individually modified will already
    have individual job records and are each counted as a separate job).

    .TP
    \fBLatency for 1000 calls to gettimeofday()\fR
    Latency of 1000 calls to the gettimeofday() syscall in microseconds,
    as measured at controller startup.

    .LP
    The next blocks of information report the most frequently issued
    remote procedure calls (RPCs), calls made for the Slurmctld daemon to perform
    some action.
    The fourth block reports the RPCs issued by message type.
    You will need to look up those RPC codes in the Slurm source code by looking
    them up in the file src/common/slurm_protocol_defs.h.
    The report includes the number of times each RPC is invoked, the total time
    consumed by all of those RPCs plus the average time consumed by each RPC in
    microseconds.
    The fifth block reports the RPCs issued by user ID, the total number of RPCs
    they have issued, the total time consumed by all of those RPCs plus the average
    time consumed by each RPC in microseconds.
    RPCs statistics are collected for the life of the slurmctld process unless
    explicitly \fB\-\-reset\fR.

    .LP
    The sixth block of information, labeled Pending RPC Statistics, shows
    information about pending outgoing RPCs on the slurmctld agent queue.
    The first section of this block shows types of RPCs on the queue and the
    count of each. The second section shows up to the first 25 individual RPCs
    pending on the agent queue, including the type and the destination host list.
    This information is cached and only refreshed on 30 second intervals.

    .SH "OPTIONS"
    .LP

    Reset scheduler and RPC counters to 0. Only supported for Slurm operators and
    administrators.

    .TP
    \fB\-t\fR, \fB\-\-sort\-by\-time\fR
    Sort Remote Procedure Call (RPC) data by total run time.

    .TP
    \fB\-T\fR, \fB\-\-sort\-by\-time2\fR
    Sort Remote Procedure Call (RPC) data by average run time.

    .TP
    \fB\-\-usage\fR
    Print list of options and exit.

    .TP
    \fB\-V\fR, \fB\-\-version\fR
    Print current version number and exit.

    .SH "ENVIRONMENT VARIABLES"
    .PP
    Some \fBsdiag\fR options may be set via environment variables. These
    environment variables, along with their corresponding options, are listed below.
    (Note: commandline options will always override these settings)
    .TP 20
    \fBSLURM_CONF\fR
    The location of the Slurm configuration file.

    .SH "COPYING"
    Copyright (C) 2010-2011 Barcelona Supercomputing Center.
    .br
    Copyright (C) 2010\-2019 SchedMD LLC.
    .LP
    Slurm is free software; you can redistribute it and/or modify it under
    the terms of the GNU General Public License as published by the Free
    Software Foundation; either version 2 of the License, or (at your option)
    any later version.
    .LP
    Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
    WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
    FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
    details.

    .SH "SEE ALSO"
    .LP
    sinfo(1), squeue(1), scontrol(1), slurm.conf(5),