Knowledge Center         Contents    Previous  Next    Index  
Platform Computing Corp.

Runtime Resource Usage Limits

Contents

About Resource Usage Limits

Resource usage limits control how much resource can be consumed by running jobs. Jobs that use more than the specified amount of a resource are signalled or have their priority lowered.

Limits can be specified by the LSF administrator:

For example, by defining a high-priority short queue, you can allow short jobs to be scheduled earlier than long jobs. To prevent some users from submitting long jobs to this short queue, you can set CPU limit for the queue so that no jobs submitted from the queue can run for longer than that limit.

Limits specified at the queue level are hard limits, while those specified with job submission or in an application profile are soft limits. The hard limit acts as a ceiling for the soft limit. See setrlimit(2) man page for concepts of hard and soft limits.

note:  
This chapter describes queue-level and job-level resource usage limits. Priority of limits is different if limits are also configured in an application profile. See Chapter 23, "Working with Application Profiles" for information about resource usage limits in application profiles.

Resource usage limits and resource allocation limits

Resource usage limits are not the same as resource allocation limits, which are enforced during job scheduling and before jobs are dispatched. You set resource allocation limits to restrict the amount of a given resource that must be available during job scheduling for different classes of jobs to start, and which resource consumers the limits apply to.

See Chapter 24, "Resource Allocation Limits" for more information.

Summary of resource usage limits

Limit
Job syntax (bsub)
Syntax (lsb.queues and lsb.applications)
Fomat/Default Units
Core file size limit
-C core_limit
CORELIMIT=limit
integer KB
CPU time limit
-c cpu_limit
CPULIMIT=[default] maximum
[hours:]minutes[/host_name | /host_model]
Data segment size limit
-D data_limit
DATALIMIT=[default] maximum
integer KB
File size limit
-F file_limit
FILELIMIT=limit
integer KB
Memory limit
-M mem_limit
MEMLIMIT=[default] maximum
integer KB
Process limit
-p process_limit
PROCESSLIMIT=[default] maximum
integer
Run time limit
-W run_limit
RUNLIMIT=[default] maximum
[hours:]minutes[/host_name | /host_model]
Stack segment size limit
-S stack_limit
STACKLIMIT=limit
integer KB
Virtual memory limit
-v swap_limit
SWAPLIMIT=limit
integer KB
Thread limit
-T thread_limit
THREADLIMIT=[default] maximum
integer

Priority of resource usage limits

If no limit is specified at job submission, then the following apply to all jobs submitted to the queue:

If ...
Then ...
Both default and maximum limits are defined
The default is enforced
Only a maximum is defined
The maximum is enforced
No limit is specified in the queue or at job submission
No limits are enforced

Incorrect resource usage limits

Incorrect limits are ignored, and a warning message is displayed when the cluster is reconfigured or restarted. A warning message is also logged to the mbatchd log file when LSF is started.

If no limit is specified at job submission, then the following apply to all jobs submitted to the queue:

If ...
Then ...
The default limit is not correct
The default is ignored and the maximum limit is enforced
Both default and maximum limits are specified, and the maximum is not correct
The maximum is ignored and the resource has no maximum limit, only a default limit
Both default and maximum limits are not correct
The default and maximum are ignored and no limit is enforced

Resource usage limits specified at job submission must be less than the maximum specified in lsb.queues. The job submission is rejected if the user-specified limit is greater than the queue-level maximum, and the following message is issued:

Cannot exceed queue's hard limit(s). Job not submitted.

Enforce limits on chunk jobs

By default, resource usage limits are not enforced for chunk jobs because chunk jobs are typically too short to allow LSF to collect resource usage.

  1. To enforce resource limits for chunk jobs, define LSB_CHUNK_RUSAGE=Y in lsf.conf. Limits may not be enforced for chunk jobs that take less than a minute to run.

Scaling the units for resource usage limits

The default unit for the following resource usage limits is KB:

This default may be too small for some environments that make use of very large resource usage limits, for example, GB or TB.

LSF_UNIT_FOR_LIMITS in lsf.conf specifies larger units for the resource usage limits with default unit of KB.

The unit for the resource usage limit can be one of:

LSF_UNIT_FOR_LIMITS applies cluster-wide to limts at the job-level (bsub), queue-level (lsb.queues), and application level (lsb.applications).

The lmit unit specified by LSF_UNIT_FOR_LIMITS also applies to limts modified with bmod, and the display of resource usage limits in query commands (bacct, bapp, bhist, bhosts, bjobs, bqueues, lsload, and lshosts).

important:  
Before changing the units of your resource usage limits, you should completely drain the cluster of all workload. There should be no running, pending, or finished jobs in the system.

In a MultiCluster environment, you should configure the same unit for all clusters.

After changing LSF_UNIT_FOR_LIMITS, you must restart your cluster.

How limit unit changes affect jobs

When LSF_UNIT_FOR_LIMITS is specified, the defined unit is used for the following commands. In command output, the larger unit appears as T, G, P, or E, depending on the job rusage and the unit defined.

Command
Option/Output
Default unit
bsub/bmod
-C (core limit)
KB
-M (memory limit)
KB
-S (stack limit)
KB
-v (swap limit)
KB
bjobs
rusage
CORELIMIT, MEMLIMIT, STACKLIMIT, SWAPLIMIT
KB (may show MB depending on job rusage)
bqueues
CORELIMIT, MEMLIMIT, STACKLIMIT, SWAPLIMIT
KB (may show MB depending on job rusage)
loadSched, loadStop
MB
bacct
Summary rusage
KB (may show MB depending on job rusage)
bapp
CORELIMIT, MEMLIMIT, STACKLIMIT, SWAPLIMIT
KB
bhist
History of limit change by bmod
KB
MEM, SWAP
KB (may show MB depending on job rusage)
bhosts
loadSched, loadStop
MB
lsload
mem, swp
KB (may show MB depending on job rusage)
lshosts
maxmem, maxswp
KB (may show MB depending on job rusage)

Example

A job is submitted with bsub -M 100 and LSF_UNIT_FOR_LIMITS=MB; the memory limit for the job is 100 MB rather than the default 100 KB.

Specifying Resource Usage Limits

Queues can enforce resource usage limits on running jobs. LSF supports most of the limits that the underlying operating system supports. In addition, LSF also supports a few limits that the underlying operating system does not support.

Specify queue-level resource usage limits using parameters in lsb.queues.

Specifying queue-level resource usage limits

Limits configured in lsb.queues apply to all jobs submitted to the queue. Job-level resource usage limits specified at job submission override the queue definitions.

Maximum value only

Specify only a maximum value for the resource.

For example, to specify a maximum run limit, use one value for the RUNLIMIT parameter in lsb.queues:

RUNLIMIT = 10 

The maximum run limit for the queue is 10 minutes. Jobs cannot run for more than 10 minutes. Jobs in the RUN state for longer than 10 minutes are killed by LSF.

If only one run limit is specified, jobs that are submitted with bsub -W with a run limit that exceeds the maximum run limit is not allowed to run. Jobs submitted without bsub -W are allowed to run but are killed when they are in the RUN state for longer than the specified maximum run limit.

For example, in lsb.queues:

RUNLIMIT = 10 

The maximum run limit for the queue is 10 minutes. Jobs cannot run for more than 10 minutes.

Default and maximum values

If you specify two limits, the first one is the default limit for jobs in the queue and the second one is the maximum (hard) limit. Both the default and the maximum limits must be positive integers. The default limit must be less than the maximum limit. The default limit is ignored if it is greater than the maximum limit.

Use the default limit to avoid having to specify resource usage limits in the bsub command.

For example, to specify a default and a maximum run limit, use two values for the RUNLIMIT parameter in lsb.queues:

RUNLIMIT = 10 15 

You can specify both default and maximum values for the following resource usage limits in lsb.queues:

Host specification with two limits

If default and maximum limits are specified for CPU time limits or run time limits, only one host specification is permitted. For example, the following CPU limits are correct (and have an identical effect):

CPULIMIT = 400/hostA 600 
CPULIMIT = 400 600/hostA 

The following CPU limit is not correct:

CPULIMIT = 400/hostA 600/hostB 

The following run limits are correct (and have an identical effect):

RUNLIMIT = 10/hostA 15 
RUNLIMIT = 10 15/hostA 

The following run limit is not correct:

RUNLIMIT = 10/hostA 15/hostB 

Default run limits for backfill scheduling

Default run limits are used for backfill scheduling of parallel jobs.

For example, in lsb.queues, you enter: RUNLIMIT = 10 15

Automatically assigning a default run limit to all jobs in the queue means that backfill scheduling works efficiently.

For example, in lsb.queues, you enter:

RUNLIMIT = 10 15 

The first number is the default run limit applied to all jobs in the queue that are submitted without a job-specific run limit. The second number is the maximum run limit.

If you submit a job to the queue without the -W option, the default run limit is used:

bsub myjob 

The job myjob cannot run for more than 10 minutes as specified with the default run limit.

If you submit a job to the queue with the -W option, the maximum run limit is used:

bsub -W 12 myjob 

The job myjob is allowed to run on the queue because the specified run limit (12) is less than the maximum run limit for the queue (15).

bsub -W 20 myjob 

The job myjob is rejected from the queue because the specified run limit (20) is more than the maximum run limit for the queue (15).

Specify job-level resource usage limits

  1. To specify resource usage limits at the job level, use one of the following bsub options:

Supported Resource Usage Limits and Syntax

Core file size limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-C core_limit
CORELIMIT=limit
integer KB

Sets a per-process (soft) core file size limit for each process that belongs to this batch job.

By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the the limit (MB, GB, TB, PB, or EB).

On some systems, no core file is produced if the image for the process is larger than the core limit. On other systems only the first core_limit KB of the image are dumped. The default is no soft limit.

CPU time limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-c cpu_limit
CPULIMIT=[default] maximum
[hours:]minutes[/host_name | /host_model]

Sets the soft CPU time limit to cpu_limit for this batch job. The default is no limit. This option is useful for avoiding runaway jobs that use up too many resources. LSF keeps track of the CPU time used by all processes of the job.

When the job accumulates the specified amount of CPU time, a SIGXCPU signal is sent to all processes belonging to the job. If the job has no signal handler for SIGXCPU, the job is killed immediately. If the SIGXCPU signal is handled, blocked, or ignored by the application, then after the grace period expires, LSF sends SIGINT, SIGTERM, and SIGKILL to the job to kill it.

You can define whether the CPU limit is a per-process limit enforced by the OS or a per-job limit enforced by LSF with LSB_JOB_CPULIMIT in lsf.conf.

Jobs submitted to a chunk job queue are not chunked if the CPU limit is greater than 30 minutes.

Format

cpu_limit is in the form [hour:]minute, where minute can be greater than 59. 3.5 hours can either be specified as 3:30 or 210.

Normalized CPU time

The CPU time limit is normalized according to the CPU factor of the submission host and execution host. The CPU limit is scaled so that the job does approximately the same amount of processing for a given CPU limit, even if it is sent to a host with a faster or slower CPU.

For example, if a job is submitted from a host with a CPU factor of 2 and executed on a host with a CPU factor of 3, the CPU time limit is multiplied by 2/3 because the execution host can do the same amount of work as the submission host in 2/3 of the time.

If the optional host name or host model is not given, the CPU limit is scaled based on the DEFAULT_HOST_SPEC specified in the lsb.params file. (If DEFAULT_HOST_SPEC is not defined, the fastest batch host in the cluster is used as the default.) If host or host model is given, its CPU scaling factor is used to adjust the actual CPU time limit at the execution host.

The following example specifies that myjob can run for 10 minutes on a DEC3000 host, or the corresponding time on any other host:

bsub -c 10/DEC3000 myjob 

See CPU Time and Run Time Normalization for more information.

Data segment size limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-D data_limit
DATALIMIT=[default] maximum
integer KB

Sets a per-process (soft) data segment size limit in KB for each process that belongs to this batch job (see getrlimit(2)).

This option affects calls to sbrk() and brk() . An sbrk() or malloc() call to extend the data segment beyond the data limit returns an error.

note:  
Linux does not use sbrk() and brk() within its calloc() and malloc(). Instead, it uses (mmap()) to create memory. DATALIMIT cannot be enforced on Linux applications that call sbrk() and malloc().

On AIX, if the XPG_SUS_ENV=ON environment variable is set in the user's environment before the process is executed and a process attempts to set the limit lower than current usage, the operation fails with errno set to EINVAL. If the XPG_SUS_ENV environment variable is not set, the operation fails with errno set to EFAULT.

The default is no soft limit.

File size limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-F file_limit
FILELIMIT=limit
integer KB

Sets a per-process (soft) file size limit in KB for each process that belongs to this batch job. If a process of this job attempts to write to a file such that the file size would increase beyond the file limit, the kernel sends that process a SIGXFSZ signal. This condition normally terminates the process, but may be caught. The default is no soft limit.

Memory limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-M mem_limit
MEMLIMIT=[default] maximum
integer KB

Sets a per-process physical memory limit for all of the processes belonging to a job

By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the the limit (MB, GB, TB, PB, or EB).

If LSB_MEMLIMIT_ENFORCE=Y or LSB_JOB_MEMLIMIT=Y are set in lsf.conf, LSF kills the job when it exceeds the memory limit. Otherwise, LSF passes the memory limit to the operating system. Some operating systems apply the memory limit to each process, and some do not enforce the memory limit at all.

LSF memory limit enforcement

To enable LSF memory limit enforcement, set LSB_MEMLIMIT_ENFORCE in lsf.conf to y. LSF memory limit enforcement explicitly sends a signal to kill a running process once it has allocated memory past mem_limit.

You can also enable LSF memory limit enforcement by setting LSB_JOB_MEMLIMIT in lsf.conf to y. The difference between LSB_JOB_MEMLIMIT set to y and LSB_MEMLIMIT_ENFORCE set to y is that with LSB_JOB_MEMLIMIT, only the per-job memory limit enforced by LSF is enabled. The per-process memory limit enforced by the OS is disabled. With LSB_MEMLIMIT_ENFORCE set to y, both the per-job memory limit enforced by LSF and the per-process memory limit enforced by the OS are enabled.

LSB_JOB_MEMLIMIT disables per-process memory limit enforced by the OS and enables per-job memory limit enforced by LSF. When the total memory allocated to all processes in the job exceeds the memory limit, LSF sends the following signals to kill the job: SIGINT first, then SIGTERM, then SIGKILL.

On UNIX, the time interval between SIGINT, SIGKILL, SIGTERM can be configured with the parameter JOB_TERMINATE_INTERVAL in lsb.params.

OS memory limit enforcement

OS enforcement usually allows the process to eventually run to completion. LSF passes mem_limit to the OS, which uses it as a guide for the system scheduler and memory allocator. The system may allocate more memory to a process if there is a surplus. When memory is low, the system takes memory from and lowers the scheduling priority (re-nice) of a process that has exceeded its declared mem_limit.

OS memory limit enforcement is only available on systems that support RLIMIT_RSS for setrlimit().

The following operating systems do not support the memory limit at the OS level:

Process limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-p process_limit
PROCESSLIMIT=[default] maximum
integer

Sets the limit of the number of processes to process_limit for the whole job. The default is no limit. Exceeding the limit causes the job to terminate.

Limits the number of concurrent processes that can be part of a job.

If a default process limit is specified, jobs submitted to the queue without a job-level process limit are killed when the default process limit is reached.

If you specify only one limit, it is the maximum, or hard, process limit. If you specify two limits, the first one is the default, or soft, process limit, and the second one is the maximum process limit.

Run time limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-W run_limit
RUNLIMIT=[default] maximum
[hours:]minutes[
/host_name |
/host_model]

A run time limit is the maximum amount of time a job can run before it is terminated. It sets the run time limit of a job. The default is no limit. If the accumulated time the job has spent in the RUN state exceeds this limit, the job is sent a USR2 signal. If the job does not terminate within 10 minutes after being sent this signal, it is killed.

With deadline constraint scheduling configured, a run limit also specifies the amount of time a job is expected to take, and the minimum amount of time that must be available before a job can be started.

Jobs submitted to a chunk job queue are not chunked if the run limit is greater than 30 minutes.

Format

run_limit is in the form [hour:]minute, where minute can be greater than 59. 3.5 hours can either be specified as 3:30 or 210.

Normalized run time

The run time limit is normalized according to the CPU factor of the submission host and execution host. The run limit is scaled so that the job has approximately the same run time for a given run limit, even if it is sent to a host with a faster or slower CPU.

For example, if a job is submitted from a host with a CPU factor of 2 and executed on a host with a CPU factor of 3, the run limit is multiplied by 2/3 because the execution host can do the same amount of work as the submission host in 2/3 of the time.

If the optional host name or host model is not given, the run limit is scaled based on the DEFAULT_HOST_SPEC specified in the lsb.params file. (If DEFAULT_HOST_SPEC is not defined, the fastest batch host in the cluster is used as the default.) If host or host model is given, its CPU scaling factor is used to adjust the actual run limit at the execution host.

The following example specifies that myjob can run for 10 minutes on a DEC3000 host, or the corresponding time on any other host:

bsub -W 10/DEC3000 myjob 

If ABS_RUNLIMIT=Y is defined in lsb.params, the run time limit is not normalized by the host CPU factor. Absolute wall-clock run time is used for all jobs submitted with a run limit.

See CPU Time and Run Time Normalization for more information.

Platform MultiCluster

For MultiCluster jobs, if no other CPU time normalization host is defined and information about the submission host is not available, LSF uses the host with the largest CPU factor (the fastest host in the cluster). The ABS_RUNLIMIT parameter in lsb.params is is not supported in either MultiCluster model; run time limit is normalized by the CPU factor of the execution host.

Thread limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-T thread_limit
THREADLIMIT=[default] maximum
integer

Sets the limit of the number of concurrent threads to thread_limit for the whole job. The default is no limit.

Exceeding the limit causes the job to terminate. The system sends the following signals in sequence to all processes belongs to the job: SIGINT, SIGTERM, and SIGKILL.

If a default thread limit is specified, jobs submitted to the queue without a job-level thread limit are killed when the default thread limit is reached.

If you specify only one limit, it is the maximum, or hard, thread limit. If you specify two limits, the first one is the default, or soft, thread limit, and the second one is the maximum thread limit.

Stack segment size limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-S stack_limit
STACKLIMIT=limit
integer KB

Sets a per-process (soft) stack segment size limit for all of the processes belonging to a job.

By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the the limit (MB, GB, TB, PB, or EB).

An sbrk() call to extend the stack segment beyond the stack limit causes the process to be terminated. The default is no soft limit.

Virtual memory (swap) limit

Job syntax (bsub)
Queue syntax (lsb.queues)
Fomat/Default Units
-v swap_limit
SWAPLIMIT=limit
integer KB

Sets a total process virtual memory limit for the whole job. The default is no limit. Exceeding the limit causes the job to terminate.

By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the the limit (MB, GB, TB, PB, or EB).

This limit applies to the whole job, no matter how many processes the job may contain.

Examples

Queue-level limits

CPULIMIT = 20/hostA 15

The first number is the default CPU limit. The second number is the maximum CPU limit.

However, the default CPU limit is ignored because it is a higher value than the maximum CPU limit.

CPULIMIT = 10/hostA

In this example, the lack of a second number specifies that there is no default CPU limit. The specified number is considered as the default and maximum CPU limit.

RUNLIMIT = 10/hostA 15

The first number is the default run limit. The second number is the maximum run limit.

The first number specifies that the default run limit is to be used for jobs that are submitted without a specified run limit (without the -W option of bsub).

RUNLIMIT = 10/hostA

No default run limit is specified. The specified number is considered as the default and maximum run limit.

THREADLIMIT=6

No default thread limit is specified. The value 6 is the default and maximum thread limit.

THREADLIMIT=6 8

The first value (6) is the default thread limit. The second value (8) is the maximum thread limit.

Job-level limits

bsub -M 5000 myjob

Submits myjob with a memory limit of 5000 KB.

bsub -W 14 myjob

myjob is expected to run for 14 minutes. If the run limit specified with bsub -W exceeds the value for the queue, the job is rejected.

bsub -T 4 myjob

Submits myjob with a maximum number of concurrent threads of 4.

CPU Time and Run Time Normalization

To set the CPU time limit and run time limit for jobs in a platform-independent way, LSF scales the limits by the CPU factor of the hosts involved. When a job is dispatched to a host for execution, the limits are then normalized according to the CPU factor of the execution host.

Whenever a normalized CPU time or run time is given, the actual time on the execution host is the specified time multiplied by the CPU factor of the normalization host then divided by the CPU factor of the execution host.

If ABS_RUNLIMIT=Y is defined in lsb.params or in lsb.applications for the application associated with your job, the run time limit and run time estimate are not normalized by the host CPU factor. Absolute wall-clock run time is used for all jobs submitted with a run time limit or a run time estimate.

Normalization host

If no host or host model is given with the CPU time or run time, LSF uses the default CPU time normalization host defined at the queue level (DEFAULT_HOST_SPEC in lsb.queues) if it has been configured, otherwise uses the default CPU time normalization host defined at the cluster level (DEFAULT_HOST_SPEC in lsb.params) if it has been configured, otherwise uses the submission host.

Example

CPULIMIT=10/hostA 

If hostA has a CPU factor of 2, and hostB has a CPU factor of 1 (hostB is slower than hostA), this specifies an actual time limit of 10 minutes on hostA, or on any other host that has a CPU factor of 2. However, if hostB is the execution host, the actual time limit on hostB is 20 minutes (10 * 2 / 1).

Normalization hosts for default CPU and run time limits

The first valid CPU factor encountered is used for both CPU limit and run time limit. To be valid, a host specification must be a valid host name that is a member of the LSF cluster. The CPU factor is used even if the specified limit is not valid.

If the CPU and run limit have different host specifications, the CPU limit host specification is enforced.

If no host or host model is given with the CPU or run time limits, LSF determines the default normalization host according to the following priority:

  1. DEFAULT_HOST_SPEC is configured in lsb.queues
  2. DEFAULT_HOST_SPEC is configured in lsb.params
  3. If DEFAULT_HOST_SPEC is not configured in lsb.queues or lsb.params, host with the largest CPU factor is used.

CPU time display (bacct, bhist, bqueues)

Normalized CPU time is displayed in the output of bqueues. CPU time is not normalized in the output if bacct and bhist.

PAM resource limits

PAM limits are system resource limits defined in limits.conf.

When USE_PAM_CREDS is enabled in lsb.queues or lsb.applications, applies PAM limits to an application or queue when its job is dispatched to a Linux host using PAM. The job will fail if the execution host does not have PAM configured.

Configure a PAM file

When USE_PAM_CREDS is enabled in lsb.queues or lsb.applications, the limits specified in the PAM configuration file are applied to an application or queue when its job is dispatched to a Linux host using PAM. The job will fail if the execution host does not have PAM configured.

  1. Create a PAM configuration file on each execution host you want.
  2. /etc/pam.d/lsf/limits.conf

  3. In the first two lines, specify the authentication and authorization you need to successfully run PAM limits. For example:
  4. auth required pam_localuser.so

    account required pam_unix.so

  5. Specify any resource limits. For example:
  6. session required pam_limits.so

On hosts that have a PAM configuration file with resource limits specified and when USE_PAM_CREDS=y in lsb.queues or lsb.applications, applies resource limits on jobs running on the execution host.

For more information about configuring a PAM file, check Linux documentation.


Platform Computing Inc.
www.platform.com
Knowledge Center         Contents    Previous  Next    Index