gtps4m0wSystem Generation

Tightly Coupled Scheduler

Function

In TPF's tightly coupled implementation, ECBs are assigned to a specific I-stream. They execute only on that I-stream unless software explicitly moves them to another I-stream, even if the I-stream they are using is heavily used and another I-stream has no work at all. For maximum performance, all I-streams must have roughly the same amount of work to do. The tightly coupled scheduler is the tool used to balance the load between the I-streams.

The scheduler is invoked when a SWISC (change I-stream) macro is issued with a target I-stream number of zero. During input message processing, the message router issues a SWISC with a target I-stream number of zero to pass control to the input message processing program. This is how input messages are assigned to an I-stream.

In some cases users may be forced to run certain user application programs only on a single I-stream, generally because the application program has not been written to operate as a reentrant program. Users can bypass the scheduler by using the COBC user exit program to explicitly specify the I-stream number for an input message. COBC is invoked by coding the COMEXIT=YES parameter on the MSGRT macro during system generation or by using the ZSYSG command. See Communications Source Message Router and System Message Processor for additional information.

Tuning the Tightly Coupled Scheduler

The scheduler attempts to keep adjusted I-stream utilization in balance across all I-streams. Adjusted I-stream utilization is the percent of the time that an I-stream spends working for all tasks minus the percent of time spent working on tasks that give up control via the DEFRC or the DLAYC macro. Adjusted I-stream utilization is reported directly to the user in the data collection/reduction I-Stream Summary Report (see TPF System Performance and Measurement Reference for more details), and can be displayed online via the ZSTAT U operator command. Adjusted I-stream utilization is used instead of actual I-stream utilization to keep looping low priority tasks from taking over an I-stream. (When a task loops using DEFRC or DLAYC, the actual I-stream utilization goes to 100%. If the scheduler used actual utilization then no further work would be scheduled on that I-stream.)

There are two conflicting requirements that must be met to obtain optimal balance: the scheduler must react instantly to any change in overall system workload composition, and the scheduler must ignore changes in load that are the result of random actions rather than an actual workload composition change. For example, if a utility program is started on a particular I-stream then the scheduler must reduce the number of tasks that are assigned to that I-stream to keep the system in balance. On the other hand, short-term changes in I-stream utilization caused by the variance in the amount of I-stream utilization required by input messages must not cause the scheduler to take any action.

The REACT parameter of the ZCNIS command determines how quickly the scheduler responds to changes in load. The higher the REACT parameter is set, the more slowly the system will respond to changes in work load. If the REACT parameter is set too high, then the system will be slow to shift the allocation of tasks in response to a change in workload composition. This condition can be seen by using the ZSTAT U command to display the size of the READY and INPUT list. During normal operation, each I-stream will maintain a certain number of tasks on each of its CPU loop work lists. Following a change in workload composition, such as starting a utility, the number of tasks on the CPU loop lists should change smoothly and reasonably quickly to a new steady-state value. If, following a change in workload composition, the scheduler has caused a workload imbalance by reacting too slowly (REACT set too high), then one or more I-streams will show substantially more tasks on its CPU loop lists than are seen in normal operation. The number of tasks will peak, then slowly start to work down to its normal steady-state value. Once the number of tasks on the CPU loop lists is back to normal, the I-stream workloads are back in balance, and will stay balanced until the next change in workload composition.

If the REACT parameter is set too low, then the system will react to statistical fluctuations in the workload, and will overreact to changes in workload composition. Both of these conditions are characterized by large READY and INPUT list queues that appear on an I-stream, remain for a few seconds, and then show up on a new I-stream. In general, when the workload composition is constant, the ratio between the sizes of the lists on the I-stream should stay roughly the same. If the ratio between the sizes of the lists swings wildly then the I-streams are not correctly balanced.

The user should exercise great care in changing the REACT parameter from the default value. It is safer to set the REACT parameter a little too high than it is to set the parameter too low. Abrupt changes in the REACT parameter may cause unexpected side effects which can lead to poor scheduler, and overall, system performance. If REACT must be changed, the change is best done in many small steps rather than one large step.

The MAXQ parameter provides a necessary safety valve which keeps the system in balance following large changes in workload composition and at extremely high utilization (>95%). MAXQ is a user-specified value that is compared to the number of tasks that are on the CROSS, READY, INPUT, DEFER and VCT lists of an I-stream. Once the system exceeds the MAXQ value on one I-stream, that I-stream will no longer have jobs assigned to it by the scheduler. (The user's ability to assign work directly to an I-stream is not affected.) If 2 or more I-streams hit MAXQ then the scheduler's reaction depends on other scheduling values (see Scheduler Algorithm). In general, the MAXQ parameter should be set so that it is not reached during normal steady-state operation, but it can be reached quickly if the system workload changes.

Notes:

  1. The ZCNIS REACT and MAXQ parameters should be set based on data observed during normal peak time operation. The scheduler is designed to operate more effectively as the overall system utilization increases. Data obtained during periods of low utilization will not help tune the scheduler for periods of high utilization. During periods of low utilization the CROSS, READY, INPUT, DEFER and VCT are too small to demonstrate the cycling behavior caused by too small a REACT parameter, the sluggish balance adjustment caused by too high a REACT parameter, or to set the correct value of MAXQ.

  2. The amount of work that is assigned to an I-stream by the user affects the ability of the scheduler to balance the system. The scheduler does not need to schedule all work to balance I-stream utilization, but it does need enough work to get the system in balance. If the user schedules about the same amount of work to all I-streams, then the scheduler will not need to schedule a substantial amount of work to bring the system into balance. If, however, 95% of the work in a system is assigned directly by the user to a single I-stream, then the scheduler can only use the remaining 5% to attempt to balance the system. It will not do a very good job.

  3. The scheduler depends upon accurate utilization data and a granular workload mix. Operation of a TPF system under VM without dedicated I-streams can lead to wildly fluctuating I-stream utilization data caused by VM time slicing of guest virtual machines. During testing, test drivers can generate workloads that change quickly and unpredictably; scheduling one item can cause a substantial amount of work for a long period of time on one I-stream with few or no other pieces of work to schedule on the other I-streams. In general, do not expect to be able to obtain a stable balance in a small scale test environment.

Scheduler Algorithm

The tightly coupled scheduler has two components. The first component, the monitor, is invoked once a second. It determines I-stream utilization and adjusted I-stream utilization for each of the available I-streams. The monitor then sets routing weights based on the adjusted I-stream utilization and a reaction time constant set by the user. The routing weights are used to schedule messages over the next one-second interval. The second component, the scheduler, uses the routing weights set by the monitor, and a MAXQ value provided by the user, to actually assign an I-stream.

I-Stream Utilization Computation

Once a second, raw I-stream utilization (RAWU) and adjusted raw I-stream utilization (ARAWU) are computed using the exact elapsed time (ELAPSE), the time spent idle (IDLE), and the time the system spent working on tasks that gave up control via either the DEFRC or DLAYC macro (DTIME). The following equations show how raw I-stream utilization and adjusted raw I-stream utilization are computed:

        ELAPSE - IDLE                 ELAPSE - (IDLE + DTIME)
RAWU = ---------------       ARAWU =  ------------------------
          ELAPSE                           ELAPSE

I-stream utilization (UTIL) is computed by exponential smoothing using the previously computed utilization (OLDU) and the raw measured I-stream utilization (RAWU):

UTIL =  ( (.8) × OLDU ) + ( (.2) × RAWU )

Adjusted I-stream utilization (AUTIL) is computed in the same way using previously computed adjusted utilization (AOLDU) and the raw measured adjusted I-stream utilization (ARAWU):

AUTIL =  ( (.8) × AOLDU ) + ( (.2) × ARAWU )

Both I-stream utilization and adjusted I-stream utilization are reported in the I-stream Summary Report produced by the data collection package and can be displayed online via the ZSTAT U operator command.

Routing Weight Computation

Routing weights are initialized to 100. Once a second the I-stream with the highest adjusted I-stream utilization and the I-stream with the lowest adjusted I-stream utilization are selected. The routing weights for these I-streams are then adjusted to change the relative amount of work that these 2 I-streams receive. Even on a processor with more than 2 I-streams, only the weights for the I-streams having the maximum and minimum adjusted utilization are changed.

The following formulae give the new weights for the I-streams that had the minimum and maximum adjusted I-stream utilization:

                      MAX + MIN + REACT + 1
MIN.WT = MIN.WT.P  ×  --------------------- (ROUNDED DOWN)
                      MAX + MAX + REACT + 1
 
 
                      MAX + MIN + REACT + 1
MAX.WT = MAX.WT.P  ×  --------------------- (ROUNDED UP)
                      MIN + MIN + REACT + 1
 

where:

MIN.WT
The weight that will be assigned to the I-stream with the minimum adjusted utilization.

MAX.WT
The weight that will be assigned to the I-stream with the maximum adjusted utilization.

MIN.WT.P
The weight assigned during the previous period to the I-stream with the minimum adjusted utilization.

MAN.WT.P
The weight assigned during the previous period to the I-stream with the maximum adjusted utilization.

MAX
The maximum adjusted I-stream utilization found in last period times 1000.

MIN
The minimum adjusted I-stream utilization found in last period times 1000.

REACT
A tuning parameter which changes the rate at which the weights change for a change in utilization.

Scheduling Tasks

A total field (TOTAL) is maintained for each I-stream. These fields are initialized to zero. When the scheduler is invoked to select an I-stream for a task, each I-stream's total field is checked to select the I-streams with the lowest and second lowest total. The I-stream with the lowest total is chosen tentatively as the I-stream on which the task will run. The selected I-stream has its total field increased by its weight. (The weight for an I-stream is computed once a second as described above.) The number of tasks on the CROSS, READY, INPUT, DEFER and VCT lists for the selected I-stream is then compared with the user specified MAXQ value. If the number of tasks queued on the CROSS, READY, INPUT, DEFER and VCT lists is less than the MAXQ parameter, the scheduler returns the selected I-stream number to the caller.

If, however, the number of tasks on the lists exceeds MAXQ then the scheduler selects the I-stream with the fewest tasks queued on its lists. The scheduler then returns the I-stream number to the caller.

User Specified Scheduling Constants

Two constants are available to tune the action of the scheduler. The ZCNIS MAXQ message sets the value MAXQ described in the section above. The ZCNIS REACT command sets the REACT parameter given in Routing Weight Computation. Refer to Tuning the Tightly Coupled Scheduler for additional information.