The parallel job manager

The parallel job manager (PJM) provides a facility and framework for submitting and managing transactional batch jobs that execute as a coordinated collection of independent parallel subordinate jobs.

PJM Architecture

Compute Grid applications provide a parallel job execution capability. The parallel job execution works using the concept of a top-level job and one or many subordinate jobs. When you submit a top-level job, the top-level job creates its own subordinate jobs. The subordinate jobs belong to the top-level job instance and are logically part of it. You can manage only the top-level jobs. The top-level jobs in turn submit and manage their subordinate jobs. The number of subordinate jobs created by a top-level job determines the degree of parallelization for that parallel job.

The role of the top-level job is to determine the number of subordinate jobs it requires, and then create and manage the required number of subordinate jobs. The life-cycle of the top-level job is synchronized with the life-cycle of its subordinate jobs. For example, if you cancel a top-level job, all its subordinate jobs are automatically canceled; if you restart a top-level job, the subordinate jobs are automatically restarted; and so on.

A parallel job consists of at least two parts: a top-level job definition and a subordinate job definition. All WCG job definitions are defined by an xJCL document. A parallel job can also invoke a set of System Programming APIs that perform special functions that assist in the execution of the parallel job.

To better understand the Parallel Job Architecture, familiarize yourself with the logical parts of a parallel job and the runtime components that supports the parallel job:

PJM architecture and programming model

The following image summarizes the PJM architecture, and the image shows where the SPI's are called.

The image depicts the PJM architecture and the sequence of a parallel job. First, a top-level job is submitted to the job scheduler and the job scheduler determines if it is a parallel job and dispatches it to the PJM. The PJM invokes the parameterizer SPI to help divide the job into subordinate jobs. The PJM then invokes the LogicalTX synchronization SPI to indicate the beginning of the logical transaction. The PJM then uses the subordinate job xJCL stored in the repository to submit the subordinate jobs to the job scheduler. The job scheduler then dispatches the subordinate jobs to the batch container endpoints for execution. After the PJM detects that the subordinate jobs started running, it invokes the life cycle SPI. No context information is available as part of this invocation, unlike the other SPI invocations. Next, the batch container runs the subordinate job. When a checkpoint is taken, the subordinate job collector SPI is invoked. This SPI collects relevant state information about the subordinate job. This data is sent to the subordinate job analyzer SPI for interpretation. After all subordinate jobs reach a final state, the synchronization SPIs beforeCompletion and afterCompletion are invoked. The analyzer SPI is also invoked to calculate the return code of the job.

Parallel Job Execution Flow

The following diagram depicts the sequence of flow of the SPIs that the PJM calls for a top-level-job instance. The PJM runs under control of the Batch Container. Recall the Batch Container bounds checkpoints within a global transaction. You can think of “checkpoint begin” and “checkpoint end” as begin and end of the global transactions. Note the PJM makes all its SPI calls in the scope of a global transaction.

The following image showed the order of events in a parallel job:

Sequence of a parallel job

The PJM job thread runs in a loop that waits for subordinate jobs to end. The loop ends after all subordinate jobs end. During initialization, the PJM establishes a pair of listeners: one listener that the Job Scheduler notifies when a subordinate job ends and the other listener that Batch Container instances notify when a SubJobCollector externalizable payload is collected from a subordinate job. These listeners wake up the top-level-job PJM instance when they have a notification to deliver.

PJM system application and parallel jobs

The PJM is an Enterprise JavaBeans (EJB) application which monitors and manages parallel sub jobs. Parallel subordinate jobs are batch J2EE applications. The PJM is a one-step batch Java 2 Platform, Enterprise Edition (J2EE) job. The PJM does not process batch data streams, but instead submits or restarts subordinate jobs under the control of step properties which identify the subordinate job in the job repository and the count of subordinate jobs to process.

A parallel job is composed of a top-level job that runs the ParallelJobManager application, and a set of subordinate jobs that run the actual business logic. Subordinate jobs run the same job definition, but each with potentially distinct inputs. All subordinate jobs are managed together as a single logical job.

Separate xJCL definitions are required for both the top-level and subordinate jobs. All subordinate jobs run using the same xJCL definition; each subordinate job instance can be parameterized with distinct substitutions properties.

A logical transaction is a unit of work demarcation that spans the running of a parallel job. Its lifecycle corresponds to the combined lifecycle of the parallel job's subordinate jobs. An extension mechanism enables customization so that application-managed resources can be controlled in this unit of work scope for commit and rollback purposes.

Job naming convention

The job name of a parallel job is specified in the xJCL that defines a top-level job. The top level job xJCL can originate from the file system or from the Compute Grid job repository. The job name can be parameterized using standard substitution property notation: <job name=”${jobname}” … >.

A job ID is formed by concatenating a job name with a system generated sequence number. A job name and sequence number are colon separated. The job name of a subordinate job is the parallel job's job ID. The xJCL for a subordinate job can originate only from the Compute Grid job repository. Subordinate job xJCL must supply the job name substitution property: <job name=”${jobname}” … >

PJM job management

The top-level job submits the subordinate jobs and monitors their completion. The top-level job end state is influenced by the outcome of the subordinate jobs as follows:
  1. If all subordinate jobs complete in the ended state, that is, in a successful completion, then the top-level job will complete in the ended state.
  2. If anysubordinate job completes in the restartable state and no subordinate job has ended in the failed state, then the top-level job will complete in the restartable state.
  3. If any subordinate job completes in the failed state, then the top-level job will complete in the failed state.
  4. If the top-level job and subordinate jobs are in the restartable state, only the top-level job should be restarted. If any subordinate jobs are restarted manually, then the top-level job will not process the logical transaction properly.