Knowledge Center         Contents    Previous  Next    Index  
Platform Computing Corp.

Advance Reservation

Contents

Understanding Advance Reservations

Advance reservations ensure access to specific hosts during specified times. During the time that an advance reservation is active only users or groups associated with the reservation have access to start new jobs on the reserved hosts.

Only LSF administrators or root can create or delete advance reservations. Any LSF user can view existing advance reservations.

Each reservation consists of the number of job slots to reserve, a list of hosts for the reservation, a start time, an end time, and an owner. You can also specify a resource requirement string instead of or in addition to a list of hosts.

Active reservations

When a reservation becomes active, LSF attempts to run all jobs associated with the reservation. By default jobs running before the reservation became active continue to run when the reservation becomes active. When a job associated with the reservation is pending because not enough job slots are available, LSF suspends all jobs not associated with the reservation that are running on the required hosts.

During the time the reservation is active, only users or groups associated with the reservation have access to start new jobs on the reserved hosts. The reservation is active only within the time frame specified, and any given host may have several reservations in place, some of which may be active at the same time.

Jobs are suspended only if advance reservation jobs require the slots. Jobs using a reservation are subject to all job resource usage limits, but any resources freed by suspending non-advance reservation jobs are available for advance reservation jobs to use.

Closed and open reservations

Reservations are typically closed. When a closed reservation expires, LSF kills jobs running in the reservation and allows any jobs suspended when the reservation became active to run.

Open advance reservations allow jobs to run even after the associated reservation expires. A job in the open advance reservation is only be treated as an advance reservation job during the reservation window, after which it becomes a normal job. This prevents the job from being killed and makes sure that LSF does not prevent any previously suspended jobs from running or interfere with any existing scheduling policies.

Jobs running in a one-time open reservation are detached from the reservation and suspended when the reservation expires, allowing them to be scheduled as regular jobs. Jobs submitted before the reservation became active are still suspended when the reservation becomes active. These are only resumed after the open reservation jobs finish.

Jobs running is a closed recurring reservation are killed when the reservation expires.

Jobs running in an open recurring reservation are suspended when the reservation expires, and remain pending until the reservation becomes active again to resume.

If a non-advance reservation job is submitted while the open reservation is active, it remains pending until the reservation expires. Any advance reservation jobs that were suspended and became normal jobs when the reservation expired are resumed first before dispatching the non-advance reservation job submitted while the reservation was active.

Job scheduling in advance reservations

LSF treats advance reservation like other deadlines, such as dispatch windows or run windows; LSF does not schedule jobs that are likely to be suspended when a reservation becomes active. Jobs referencing the reservation are killed when the reservation expires.

note:  
If IGNORE_DEADLINE=Y, there is no effect on advance reservations. Jobs are always prevented from starting if there is a chance that they could encounter an advance reservation.

System reservations

Reservations can also be created for system maintenance. If a system reservation is active, no other jobs can use the reserved hosts, and LSF does not dispatch jobs to the specified hosts while the reservation is active.

Configure Advance Reservation

Enable advance reservation

  1. To enable advance reservation in your cluster, make sure the advance reservation scheduling plugin schmod_advrsv is configured in lsb.modules.
  2. Begin PluginModule 
    SCH_PLUGIN               RB_PLUGIN                SCH_DISABLE_PHASES 
    schmod_default              ()                              () 
    schmod_advrsv               ()                              () 
    End PluginModule 
    

Allow users to create advance reservations

By default, only LSF administrators or root can add or delete advance reservations. To allow other users to use brsvadd to create advance reservations and brsvdel to delete advance reservations, you need to configure advance reservation user policies.

  1. Use the ResourceReservation section of lsb.resources to configure advance reservation policies for users.
  2. A ResourceReservation section specifies:

For example:

    1. Define a policy for user: user1:
    2. Policy Name: dayPolicy 
      Users: user1 
      Hosts: hostA 
      Time Window:  8:00-18:00 
      
    3. User user1 creates a reservation matching the policy ( the creator is user1, the user is user2):
    4. brsvadd -n 1 -m hostA -u user2 -b 10:00 -e 12:00  
      user2#0 is created.  
      
    5. User user1 modifies the policy to remove user1 from the users list:
    6. Policy Name: dayPolicy 
      Users: user3 
      Hosts: hostA 
      Time Window:  8:00-18:00 
      
    7. As the creator, user1 can modify the reservation with the brsvmod options rmhost, -u, -o, -on, and -d, but user1 cannot add hosts or modify the time window of the reservation.

USER_ADVANCE_RESERVATION is obsolete (lsb.params)

USER_ADVANCE_RESERVATION in lsb.params is obsolete in LSF Version 7. Use the ResourceReservation section configuration in lsb.resources to configure advance reservation policies for your cluster.

Using Advance Reservation

Advance reservation commands

Use the following commands to work with advance reservations:

brsvadd

Add a reservation

brsvdel

Delete a reservation

brsvmod

Modify a reservation

brsvs

View reservations

Add reservations

note:  
By default, only LSF administrators or root can add or delete advance reservations.
  1. Run brsvadd to create new advance reservations.
  2. You must specify the following for the reservation:

Specify hosts for the reservation

  1. Use one or both of the following brsvadd options to specify hosts for which job slots are reserved:

Add a one-time reservation

  1. Use the -b and -e options of brsvadd to specify the begin time and end time of a one-time advance reservation. One-time reservations are useful for dedicating hosts to a specific user or group for critical projects.
  2. The day and time are in the form:

    [[[year:]month:]day:]hour:minute 
     

    with the following ranges:

Add a recurring reservation

  1. Use the -t option of brsvadd to specify a recurring advance reservation. The -t option specifies a time window for the reservation. Recurring reservations are useful for scheduling regular system maintenance jobs.
  2. The day and time are in the form:

    [day:]hour[:minute] 
     

    with the following ranges:

Add an open reservation

  1. Use the -o option of brsvadd to create an open advance reservation. You must specify the same information as for normal advance reservations.
  2. The following command creates a one-time open advance reservation for 1024 job slots on a host of any type for user user1 between 6:00 a.m. and 8:00 a.m. today:

    brsvadd -o -n 1024 -R "type==any" -u user1 -b 6:0 -e 8:0
    Reservation "user1#1" is created 
     

    The following command creates an open advance reservation for 1024 job slots on hostB for user user3 every weekday from 12:00 noon to 2:00 p.m.:

    brsvadd -o -n 1024 -m "hostB" -u user3 -t "12:0-14:0" Reservation "user2#0" is created

Specify a reservation name

  1. Use the -N option of brsvadd to specify a user-defined advance reservation name unique in an LSF cluster.
  2. The reservation name is a string of letters, numeric characters, underscores, and dashes beginning with a letter. The maximum length of the name is 39 characters.

    If no user-defined advance reservation name is specified, LSF creates the reservation with a system assigned name with the form

    user_name#sequence 
     

    For example:

    brsvadd -n 3 -M "hostA hostB" -u user2 -b 16:0 -e 17:0 -d "Production AR test" Reservation user2#0 (Production AR test) is created brsvadd -n 2 -N Production_AR -M hostA -u user2 -b 16:0 -e 17:0 -d "Production AR test" Reservation Production_AR (Production AR test) is created

    If a job already exists that references a reservation with the specified name, an error message is returned: The specified reservation name is referenced by a job.

Modify an advance reservation

  1. Use brsvmod to modify reservations. Specify the reservation ID for the reservation you want to modify. For example, run the following command to extend the duration from 6:00 a.m. to 9:00 a.m.:
  2. brsvmod -e "+60" user1#0 
    Reservation "user1#0" is modified  
     

    Administrators and root can modify any reservations. Users listed in the ResourceReservation section of lsb.resources, can only modify reservations they created themselves.

Using brsvmod to modify advance reservations

Use brsvmod to make the following changes to an existing advance reservation:

For example, assume an advance reservation is the box between the time t1 and t2, as shown in the following figure:

In this figure:

Use brsvmod to shift, extend or reduce the time window horizontally; grow or shrink the size vertically.

Extending the duration

The following command creates a one-time advance reservation for 1024 job slots on host hostA for user user1 between 6:00 a.m. and 8:00 a.m. today:

brsvadd -n 1024 -m hostA -u user1 -b "6:0" -e "8:0" 
Reservation "user1#0" is created  

Run the following command to extend the duration from 6:00 a.m. to 9:00 a.m.:

brsvmod -e "+60" user1#0 
Reservation "user1#0" is modified  
Adding hosts to a reservation allocation

Use brsvmod to add hosts and slots on hosts into the original advance reservation allocation. The hosts can be local to the cluster or hosts leased from remote clusters.

Adding a host without -n reserves all available slots on the host; that is, slots that are not already reserved by other reservations. You must specify -n along with -m or -R. The -m option can be used alone if there is no host group specified in the list. You cannot specify -R without -n.

The specified slot number must be less than or equal to the available number of job slots for the host.

You can only add hosts (-m) to a system reservation. You cannot add slots (-n) to a system reservation.

For example:

The following command creates an advance reservation for 1024 slots on two hosts hostA and hostB for user group groupA every Wednesday from 12:00 midnight to 3:00 a.m.:

brsvadd -n 1024 -m "hostA hostB" -g groupA -t "3:0:0-3:3:0" 
Reservation "groupA#0" is created  
brsvs 
RSVID      TYPE    USER     NCPUS      RSV_HOSTS       TIME_WINDOW 
groupA#0   user    groupA   0/1024     hostA:0/256     3:3:0-3:3:0 * 
                                    hostB:0/768 

The following commands reserve 512 slots from each host for the reservation:

brsvmod addhost -n 256 -m "hostA" groupA#0 
Reservation "groupA#0" is modified  
brsvmod rmhost -n 256 -m "hostB" groupA#0 
Reservation "groupA#0" is modified 
Removing hosts from a reservation allocation

Use brsvmod rmhost to remove hosts or slots on hosts from the original reservation allocation. You must specify either -n or -m. Use -n to specify the number of slots to be released from the host. Removing a host without -n releases all reserved slots on the host. The slot specification must be less than or equal to the actual reserved slot number of the host.

For example:

You cannot remove slots from a system reservation. The following modification to the system reservation System#1 is rejected:

brsvmod rmhost -n 2 -m "hostA"  system#1 

How many slots or hosts can be removed also depends on the number of slots free while the reservation is active. brsvmod rmhost cannot remove more slots than free amount on a host. For example:

brsvs 
RSVID    TYPE USER    NCPUS         RSV_HOSTS    TIME_WINDOW 
user1_1  user  user1  3/4           hostA:2/2    1/24/12/2-1/24/13/0 
                                    hostB:1/2 

The following modifications are accepted, and one slot is removed from hostB:

brsvmod rmhost -m hostB user1_1 
brsvmod rmhost -n 1 -m hostB user1_1 

The following modifications are rejected:

brsvmod rmhost -n 2 user1_1 
brsvmod rmhost -m hostA user1_1 
brsvmod rmhost -n 1 -m hostA user1_1 
brsvmod rmhost -n 2 -m hostB user1_1 
Modifying closed reservations

The following command creates an open advance reservation for 1024 job slots on host hostA for user user1 between 6:00 a.m. and 8:00 a.m. today.

brsvadd -o -n 1024 -m hostA -u user1 -b 6:0 -e 8:0  
Reservation "user1#0" is created  

Run the following command to close the reservation when it expires.

brsvmod -on user1#0 
Reservation "user1#0" is modified  
Disable specified occurrences for recurring reservations

Use brsvmod disable to disable specified periods, or instances, of a recurring advance reservation.

Recurring reservations may repeat either on a daily cycle or a weekly cycle. For daily reservations, the instances of the reservation that occur on disabled days will be inactive. Jobs using the reservation are not dispatched during on those disabled days. Other reservations are permitted to use slots of the reservation on those days. For overnight reservations (active from 11 p.m. to 9 a.m. daily), if the reservation is disabled on the starting day of an instance, the reservation is disabled for the whole of that instance.

For a weekly reservation, if the reservation is disabled on the start date of an instance of the reservation then the reservation is disabled for the entire instance. For example, for a weekly reservation with time window from 9 a.m. Wednesday to 10 p.m. Friday, in one particular week, the reservation is disabled on Thursday, then the instance of the reservation remains active for that week. However, if the same reservation is disabled for the Wednesday of the week, then the reservation is disabled for the week.

The following figure illustrates how the disable options apply to the weekly occurrences of a recurring advance reservation.

Once a reservation is disabled for a period, it cannot be enabled again; that is, the disabled periods remain fixed. Before a reservation is disabled, you are prompted to confirm whether to continue disabling the reservation. Use the -f option to silently force the command to run without prompting for confirmation, for example, to allow for automating disabling reservations from a script.

For example, the following command creates a recurring advance reservation for 4 slots on host hostA for user user1 between 6:00 a.m. and 8:00 a.m. every day.

Reservation "user1#0" is created  
brsvadd -n 4 -m hostA -u user1 -t "6:0-8:0"  

Run the following command to disable the reservation instance that is active between Dec 1 to Dec 10, 2007.

brsvmod -disable -td "2007:12:1-2007:12:10" user1#0 
Reservation "user1#0" is modified  

Then the administrator can use host hostA for other reservations during the duration

brsvadd -n 4 -m hostA -u user1 -b "2007:12:1:6:0" -e "2007:12:1:8:0"  
Reservation "user1#2" is created  
Change users and user groups

Use brsvmod -u to change the user or brsvmod -g to change the user group that is able to submit jobs with the advance reservation.

Jobs submitted by the original user or user group to the reservation still belong to the reservation and scheduled as advance reservation jobs, but new submitted jobs from the removed user or user group cannot use the reservation any longer.

brun

An advance reservationa job dispatched with brun is still subject to run windows and suspending conditions of the advance reservation for the job. The job must finish running before the time window of a closed reservation expires. Extending or shrinking a closed advance reservation duration prolongs or shortens lifetime of a brun job.

bslots

bslots displays a snapshot of the slots currently not in use by parallel jobs or advance reservations. If the hosts or duration of an advance reservation is modified, bslots recalculates and displays the available slots and available run time accordingly.

How advance reservation modifications interact

The following table summarizes how advance reservation modification applies to various advance reservation instances.

 
 
 
Disable
Modification
 
 
 
Begin time
End Time
Add Hosts
Rm Hosts
User/
Usergroup
open/
closed
Pre cmd
Post cmd
One-time
 
Active
 
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Inactive
 
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Recurring
 
 
Occurrences
 
All
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Specified
Yes
No
No
No
No
No
No
No
No
Active instance
 
No
No
No
No
No
No
No
No
No

Where: "Yes" means the modification is supported in the scenario; otherwise, "No" is marked. For example, all modifications are acceptable in the case that the advance reservation is inactive.

Reservation policy checking

The following table summarizes how advance reservation commands interpret reservation policy configurations in lsb.resources:

The command ...
Checks policies for ...
Creator
Host
TimeWindow
brsvadd
 
Yes
Yes
Yes
brsvdel
 
No
No
No
brsvmod
-u or -g (changing user)
No
No
No
addhost
Yes
Yes
Yes
rmhost
No
No
No
-b, -e, -t (change timeWindow)
Yes
Yes
Yes
-d (description)
No
No
No
-o or -on
No
No
No

Reservation policies are checked when:

Reservation policies are not checked when

Remove an advance reservation

  1. Use brsvdel to delete reservations. Specify the reservation ID for the reservation you want to delete.
  2. For example:

    brsvdel user1#0
    Reservation user1#0 is being deleted 
     

    You can delete more than one reservation at a time. Administrators can delete any reservation, but users may only delete their own reservations.

    If the recurring reservation is deleted with brsvdel, jobs running in the reservation are detached from the reservation and scheduled as normal jobs.

View reservations

  1. Use brsvs to show current reservations:
  2. brsvs
    RSVID       TYPE    USER   NCPUS      RSV_HOSTS           TIME_WINDOW
    user1#0     user    user1   0/1024    hostA:0/1024   11/12/6/0-11/12/8/0
    user2#0     user    user2  0/1024     hostA:0/1024           12:0-14:0 *
    groupA#0   group   groupA  0/2048     hostA:0/1024         3:0:0-3:3:0 *
                                          hostB:0/1024
    system#0     sys   system   1024       hostA:0/1024       5:18:0-5:20:0 * 
     

    In the TIME_WINDOW column:

Show a weekly planner

  1. Use brsvs -p to show a weekly planner for specified hosts using advance reservation. The all keyword shows the planner for all hosts with reservations.
  2. The output of brsvs -p is displayed in terms of weeks. The week starts on Sunday. The timeframe of a recurring reservation is not displayed, since it is unlimited. The timeframe of one-time reservation is displayed in terms of a week. If the reservation spans multiple weeks, these weeks are displayed separately. If a week contains a one-time reservation and a recurring reservation, the timeframe is displayed, since that is relevant for one-time reservation.

    tip:  
    MAX indicates the configured maximum number of job slots for the host (MXJ defined in lsb.hosts).
    brsvs -p all
    RSVID       TYPE    USER   NCPUS      RSV_HOSTS           TIME_WINDOW
    user1#0     user    user1  0/1024     hostA:0/1024   11/12/6/0-11/12/8/0
    user2#0     user    user2  0/1024     hostA:0/1024           12:0-14:0 *
    groupA#0   group   groupA  0/2048     hostA:0/1024         3:0:0-3:3:0 *
                                          hostB:0/1024
    system#0     sys   system   1024      hostA:0/1024       5:18:0-5:20:0 *
    
    HOST: hostA  (MAX = 1024)
    Week: 11/11/2009 - 11/17/2009
    Hour:Min     Sun     Mon     Tue     Wed     Thu     Fri     Sat
    -------------------------------------------------------------------
    0:0          0       0       0       1024    0       0       0
    0:10         0       0       0       1024    0       0       0
    0:20         0       0       0       1024    0       0       0
    ...
    2:30         0       0       0       1024    0       0       0
    2:40         0       0       0       1024    0       0       0
    2:50         0       0       0       1024    0       0       0
    3:0          0       0       0       0       0       0       0
    3:10         0       0       0       0       0       0       0
    3:20         0       0       0       0       0       0       0
    ...
    5:30         0       0       0       0       0       0       0
    5:40         0       0       0       0       0       0       0
    5:50         0       0       0       0       0       0       0
    6:0          0       1024    0       0       0       0       0
    6:10         0       1024    0       0       0       0       0
    6:20         0       1024    0       0       0       0       0
    ...
    7:30         0       1024    0       0       0       0       0
    7:40         0       1024    0       0       0       0       0
    7:50         0       1024    0       0       0       0       0
    8:0          0       0       0       0       0       0       0
    8:10         0       0       0       0       0       0       0
    8:20         0       0       0       0       0       0       0
    ...
    11:30        0       0       0       0       0       0       0
    11:40        0       0       0       0       0       0       0
    11:50        0       0       0       0       0       0       0
    12:0         1024    1024    1024    1024    1024    1024    1024
    12:10        1024    1024    1024    1024    1024    1024    1024
    12:20        1024    1024    1024    1024    1024    1024    1024
    ...
    13:30        1024    1024    1024    1024    1024    1024    1024
    13:40        1024    1024    1024    1024    1024    1024    1024
    13:50        1024    1024    1024    1024    1024    1024    1024
    14:0         0       0       0       0       0       0       0
    14:10        0       0       0       0       0       0       0
    14:20        0       0       0       0       0       0       0
    ...
    17:30        0       0       0       0       0       0       0
    17:40        0       0       0       0       0       0       0
    17:50        0       0       0       0       0       0       0
    18:0         0       0       0       0       0       1024    0
    18:10        0       0       0       0       0       1024    0
    18:20        0       0       0       0       0       1024    0
    ...
    19:30        0       0       0       0       0       1024    0
    19:40        0       0       0       0       0       1024    0
    19:50        0       0       0       0       0       1024    0
    20:0         0       0       0       0       0       0       0
    20:10        0       0       0       0       0       0       0
    20:20        0       0       0       0       0       0       0
    ...
    23:30        0       0       0       0       0       0       0
    23:40        0       0       0       0       0       0       0
    23:50        0       0       0       0       0       0       0
    
    HOST: hostB  (MAX = 1024)
    Week: 11/11/2009 - 11/17/2009
    Hour:Min     Sun     Mon     Tue     Wed     Thu     Fri     Sat
    -------------------------------------------------------------------
    0:0          0       0       0       1024    0       0       0
    0:10         0       0       0       1024    0       0       0
    0:20         0       0       0       1024    0       0       0
    ...
    2:30         0       0       0       1024    0       0       0
    2:40         0       0       0       1024    0       0       0
    2:50         0       0       0       1024    0       0       0
    3:0          0       0       0       0       0       0       0
    3:10         0       0       0       0       0       0       0
    3:20         0       0       0       0       0       0       0
    ...
    23:30        0       0       0       0       0       0       0
    23:40        0       0       0       0       0       0       0
    23:50        0       0       0       0       0       0       0 
    
  3. Use brsvs -z instead of brsvs -p to show only the weekly items that have reservation configurations. Lines that show all zero (0) are omitted.
  4. For example:

    brsvs -z all 
    RSVID        TYPE      USER    NCPUS       RSV_HOSTS     TIME_WINDOW 
    user1_1      user      user1     0/3       hostA:0/2       12/28/14/30-12/28/15/30 
                                               hostB:0/1 
    HOST: hostA (MAX = 2) 
    Week: 12/23/2007 - 12/29/2007 
    Hour:Min        Sun     Mon     Tue     Wed     Thu     Fri     Sat 
    ---------------------------------------------------------------------- 
    14:30           0       0       0       0       0       1       0 
    14:40           0       0       0       0       0       1       0 
    14:50           0       0       0       0       0       1       0 
    15:0            0       0       0       0       0       1       0 
    15:10           0       0       0       0       0       1       0 
    15:20           0       0       0       0       0       1       0 
     
    HOST: hostB (MAX = 2) 
    Week: 12/23/2007 - 12/29/2007 
    Hour:Min        Sun     Mon     Tue     Wed     Thu     Fri     Sat 
    ---------------------------------------------------------------------- 
    14:30           0       0       0       0       0       2       0 
    14:40           0       0       0       0       0       2       0 
    14:50           0       0       0       0       0       2       0 
    15:0            0       0       0       0       0       2       0 
    15:10           0       0       0       0       0       2       0 
    15:20           0       0       0       0       0       2       0 
    

Show reservation types and associated jobs

  1. Use the -l option of brsvs to show each advance reservation in long format.
  2. The rows that follow the reservation information show the type of reservation (open or closed) and the status of any job associated with the specified reservation (EXIT, PEND, RUN, or SUSP) as well as:

Show reservation ID

  1. Use bjobs -l to show the reservation ID used by a job:
  2. bjobs -l
    Job <1152>, User <user1>, Project <default>, Status <PEND>, Queue 
    <normal>, Reservation <user1#0>, Command <myjob>
    
    Mon Nov 12 5:13:21: Submitted from host <hostB>, CWD 
    </home/user1/jobs>; 
    

View historical accounting information for advance reservations

  1. Use the -U option of the bacct command to display accounting information about advance reservations.
  2. bacct -U summarizes all historical modification of the reservation and displays information similar to the brsvs command:

Submit and modify jobs using advance reservations

  1. Use the -U option of bsub to submit jobs with a reservation ID. For example:
  2. bsub -U user1#0 myjob 
     

    The job can only use hosts reserved by the reservation user1#0. By default, LSF selects only hosts in the reservation. Use the -m option to specify particular hosts within the list of hosts reserved by the reservation; you can only select from hosts that were included in the original reservation.

    If you do not specify hosts (bsub -m) or resource requirements (bsub -R), the default resource requirement is to select hosts that are of any host type (LSF assumes "type==any" instead of "type==local" as the default select string).

    If you later delete the advance reservation while it is still active, any pending jobs still keep the "type==any" attribute.

    A job can only use one reservation. There is no restriction on the number of jobs that can be submitted to a reservation; however, the number of slots available on the hosts in the reservation may run out. For example, reservation user2#0 reserves 1024 slots on hostA. When all 1024 slots on hostA are used by jobs referencing user2#0, hostA is no longer available to other jobs using reservation user2#0. Any single user or user group can have a maximum of 100 reservation IDs.

    Jobs referencing the reservation are killed when the reservation expires.

Modify job reservation ID

Prerequisites: You must be an administrator to perform this task.

  1. Use the -U option of bmod to change a job to another reservation ID.
  2. For example:

    bmod -U user1#0 1234 
    
  3. To cancel the reservation, use the -Un option of bmod.
  4. For example:

    bmod -Un 1234 
     

    Use bmod -Un to detach a running job from an inactive open reservation. Once detached, the job is scheduled like a normal job.

Job resource usage limits and job chunking

A job using a reservation is subject to all job resource usage limits. If a limit is reached on a particular host in a reservation, jobs using that reservation cannot start on that host.

An advance reservation job is dispatched to its reservation even if the run limit or estimated run time of the job exceeds the remaining active time of the reservation. For example, if a job has a runlimit of 1 hour, and a reservation has a remaining active time of 1 minute, the job is still dispatched to the reservation. If the reservation is closed, the job is terminated when the reservation expires.

Similarly, when using chunk job scheduling, advance reservation jobs are chunked together as usual when dispatched to a host of the reservation without regard to the expiry time of the reservation. This is true even when the jobs are given a run limit or estimated run time. If the reservation is closed, the jobs in WAIT state are terminated when the reservation expires.

Advance reservation preemption

Advance reservation preemption allows advance reservation jobs to use the slots reserved by the reservation. Slots occupied by non-advance jobs may be preempted when the reservation becomes active.

Without modification with brsvmod, advance reservation preemption is triggered at most once per reservation period (in the case of a non-recurring reservation, there is only one period) whenever both of the following conditions are met:

If an advance reservation is modified, preemption is done for an active advance reservation after every modification of the reservation when there is at least one pending or suspended job associated with the reservation.

When slots are added to an advance reservation with brsvmod, LSF preempts running non-reservation jobs if necessary to provide slots for jobs belonging to the reservation. Preemption is triggered if there are pending or suspended jobs belonging to the reservation in the system.

When preemption is triggered, non-advance reservation jobs are suspended and their slots given to the advance reservation on the hosts belonging to the reservation. On each host, enough non-advance reservation jobs are suspended so that all of slots required by the advance reservation are obtained. The number of slots obtained does not depend on the number of jobs submitted to the advance reservation. Non-advance reservation jobs on a host can only to use slots not assigned to the advance reservation.

When a job is preempted for an advance reservation, it can only resume on the host when either the advance reservation finishes, or some other non-advance reservation job finishes on the host.

For example, a single-host cluster has 10 slots, with 9 non-advance reservation jobs dispatched to the host (each requiring one slot). An advance reservation that uses 5 slots on the host is created, and a single job is submitted to the reservation. When the reservation becomes active, 4 of the non-advance reservation jobs are suspended, and the advance reservation job will start.

Forcing a job to run before a reservation is active

LSF administrators can use brun to force jobs to run before the reservation is active, but the job must finish running before the time window of the reservation expires.

For example, if the administrator forces a job with a reservation to run one hour before the reservation is active, and the reservation period is 3 hours, a 4 hour run limit takes effect.

Host intersection and advance reservation

When ENABLE_HOST_INTERSECTION=y in lsb.params, LSF finds any existing intersection with hosts specified in the queue and those specified at job submission by bsub -m and/or hosts with advance reservation. When specifying keywords such as all, allremote, and others, LSF finds an existing intersection of hosts available and the job runs rather than being rejected.

Advance reservations across clusters

You can create and use advance reservation for the MultiCluster job forwarding model. To enable this feature, you must upgrade all clusters to LSF Version 7 or later.

See the Using Platform LSF MultiCluster for more information.

Resizable jobs and advance reservations

Like regular jobs, resizable jobs associated with an advance reservation can be dispatched only after the reservation becomes active, and the minimum processor request can be satisfied. The allocation request is treated like a regular advance reservation job, which relies on slots available to the reservation. If an advance reservation gets more resources by modification (brsvmod addhost), those resources can be used by pending allocation requests immediately.

The following table summarizes the relationship of the AR lifecycle and resizable job requests:

Advance Reservation
Resizable job
Allocation request
One-time expired/deleted
Open
RUN->SSUSP->RUN
Postponed until the job runs
Closed
Removed
Removed
Recurrent expired/deleted
Open
SSUSP till next instance
Postponed until the job runs again in next instance
Closed
Removed
Removed

By the time a reservation has expired or deleted, the status change of the resizable job to SSUSP blocks a resizable job allocation request from being scheduled.

Released slots from a resizable job can be reused by other jobs in the reservation.

Resizable advance reservation jobs can preempt non-advance reservation jobs that are consuming the slots that belong to the reservation. Higher priority advance reservation jobs can preempt low priority advance reservation jobs, regardless of whether both are resizable jobs.

Allocation requests of resizable AR jobs honor limits configuration. They cannot preempt any limit tokens from other jobs.

Compute units and advance reservations

Like regular jobs, jobs with compute unit resource requirements and an advance reservation can be dispatched only after the reservation becomes active, and the minimum processor request can be satisfied.

In the case of exclusive compute unit jobs (with the resource requirement cu[excl]), the advance reservation can affect hosts outside the advance reservation but in the same compute unit as follows:

Ideally all hosts belonging to a compute unit should be inside or outside of an advance reservation.


Platform Computing Inc.
www.platform.com
Knowledge Center         Contents    Previous  Next    Index