Message processing

The adapter supports two primary operations:

  1. The retrieval of messages from a JMS destination
  2. The delivery of a message to a JMS destination

The adapter performs both operations by establishing a connection to a JMS provider (such as WebSphere MQ) and then using the JMS API to:

These two operations are described in detail in Event message processing and Request message processing.

Event message processing

The connector periodically checks for new messages delivered to one or more JMS destinations. During each poll cycle, the connector:

  1. Uses the JMS API to retrieve any waiting messages.
  2. Calls a configured data handler to convert the message content to a business object.
  3. Delivers or publishes the event business object to the configured integration broker for processing by any subscribing business processes.

These steps are illustrated in Figure 1and described in depth in:

Figure 1. Event message flow




Event detection

During each event polling cycle, the connector performs a non-blocking read of messages at the destination specified by connector property InputDestination (for further information on connector properties, see Configuring connector properties). The connector retrieves messages and then publishes them to the broker.

The connector uses the pollForEvents() method to poll at regular intervals for messages. For each poll cycle, message retrieval is limited to the maximum number specified by connector property PollQuantity. If it retrieves all available messages before reaching the specified maximum, the connector does not wait for more messages but instead returns immediately from the poll cycle.

If multiple destinations are specified in connector property InputDestination, the connector polls each destination specified in a round-robin manner. It retrieves and publishes to the broker a maximum of PollQuanity number of messages from each destination. If it empties all destinations before reaching the maximum specified by the PollQuantity, the connector returns immediately from the poll cycle.

For example, in a scenario where

the adapter would retrieve messages in the following order in a single poll cycle:

  1. Next message from queue A (leaving 1 message remaining)
  2. Next message from queue B (now empty)
  3. Next message from queue C (leaving 4 messages remaining)
  4. Next message from queue A (now empty)
  5. Connector checks queue B but it's still empty.
  6. Next message from queue C (leaving 3 messages remaining)

The adapter then returns from the polling cycle because it has now polled a maximum (as set by the PollQuanity) of 2 messages from each queue.

Event status and recovery

Event message retrieval is part of a transaction. If the connector terminates unexpectedly before committing the transaction, the transaction is rolled back and the original message restored. Because the connector framework does not currently support distributed transactions, the connector may publish an event to the broker but either terminates unexpectedly or loses communication before an acknowledgement from the broker is received. In such cases, whether or not the event was received by the broker cannot be determined by any information available to the connector. To avoid loss of event messages, the connector does not commit the transaction until after the connector has received a response from the broker confirming receipt of the event. If there is a failure between the time the connector publishes an event and when it receives an acknowledgement, the transaction is rolled back automatically and the original message is restored. Because it's unknown whether or not the message was processed by the broker, such events are referred to as in-doubt events

Upon restart, the connector will start processing messages from the input destination and resubmit the in-doubt event. Although this strategy eliminates any risk that an event could be lost, it cannot prevent the same event from being published twice.

There are two means of reducing or eliminating the risk of duplicate event delivery: use of an in-progress destination (see Recovery with an in-progress destination) or via guaranteed event delivery (see Recovery with guaranteed event delivery).

Recovery with an in-progress destination

To control how in-doubt events are handled, you can create a separate, temporary destination by specifying the connector property InProgressDestination.

Note:
Recovery with an in-progress destination is not supported with Pub/Sub style messaging.
Before publishing an event to the broker, the connector moves the event message from the input destination to the in-progress destination. Once it receives acknowledgement from the broker, the connector will remove the message from the in-progress destination. This isolates in-doubt messages that have not been processed. Upon startup, if the connector finds messages in the in-progress destination, the connector can safely assume that these were left from a previous instance of the connector that terminated unexpectedly. You can specify different actions for the connector to take on such messages (if duplicate event notification is unacceptable). You do this by specifying one of four options for the connector configuration property InDoubtEvents as follows:

For further information, see InDoubtEvents.

Recovery with guaranteed event delivery

The guaranteed-event-delivery feature enables the connector framework to ensure that events are never lost and never sent twice. The connector framework supports guaranteed event delivery through two mechanisms: container managed events (CME) and duplicate event elimination (DEE).

Container managed events (CME)

You can use CME when the connector is configured for PTP-style messaging; when configured for Pub/Sub-style messaging, the connector does not support CME. For more on how this method of guaranteed-event-delivery works, see the Connector Development Guide for Java. For further information on the ContainerManagedEvents connector property, see ContainerManagedEvents.

Duplicate event elimination (DEE)

DEE is the recommended approach to implement guaranteed event delivery for the JMS adapter. DEE also is the only approach supported for Pub/Sub-style messaging.

With DEE, a connector includes a unique ID with each event it publishes to the broker. The framework checks that the connector does not submit the same event ID consecutively. If this does occur, the framework assumes that the connector is publishing the same event twice and discards the second submission. For PTP-style messaging, DEE reduces the substantial overhead involved in copying messages to and from an in-progress destination

This connector includes the message ID of all events when publishing business objects to the broker. If the connector fails to successfully post an event to the broker due to communication failure or unexpected termination, the original message is rolled back to the input queue as described previously. Upon restart, the connector begins resubmitting events from the queue including any in-doubt messages. If DEE is enabled, any in-doubt message that successfully reached the broker in the past will be discarded. This assures that each message is posted once and only once to the broker.

When using DEE, you should avoid manipulating the order of messages in destinations while the connector is off-line. DEE records only the last message ID retrieved by the adapter. DEE will fail in situations where, for example, new messages with higher priorities have pushed the last in-doubt message down in the queue before the adapter could restart.

For information on DEE and enabling it, see the Connector Development Guide for Java. For further information on the DuplicateEventElimination connector property, see DuplicateEventElimination.

Event retrieval

Event retrieval encompasses the typical processing of events by the connector. It begins when an incoming event is detected and ends when it has been converted into a format suitable for the target application and successfully delivered to the designated integration broker. The connector delivers all events asynchronously ("fire and forget") to the broker.

The following sections discuss event retrieval:

Metadata and meta-objects

In order for the connector to successfully convert messages to business objects and vice-versa, it needs additional information known as metadata. Metadata describes how data in an object or message or application is represented or should be processed. Meta-data includes such details as which business object to create if the connector retrieves a message from destination XYZ, or which data-handler should be used to serialize a request business object of type Customer with verb Create.

Attributes, properties, verbs, and application-specific information constitute the metadata for a business object definition. In addition, you can specify one or more meta-objects that contain metadata about destinations, data formats, data handlers, and more.

There are two types of meta-objects: static and dynamic. You create a static meta-object during implementation. It contains attributes that provide meta-data for each business object type the connector must support. The static meta-object is specified in connector-specific properties and is read by the connector during initialization. For an overview of meta-object properties and how they affect message transformation, see Business object mapping and Understanding message header mapping.

The second type of meta-object is dynamic. This meta-object allows you to change the metadata used by the adapter to process a business object on a per-request basis during request processing. During event processing, the dynamic meta-object acts as a container to hold transport-specific information about the event (for example, message ID, priority, etc.) so that downstream business processes can use the information in their business logic. The dynamic meta-object is represented as a specially marked child object defined in the event (or request) top-level object.

You may opt to use one or both types of meta-objects in the same implementation. Values provided in a dynamic meta-object generally take precedence over any values provided in the static meta-object. For further information on metadata, see the Connector Development Guide for Java. For information on configuring static and dynamic meta-objects, see Configuring meta-objects.

Business object mapping

Upon retrieval of a message, the connector attempts to identify which business object the message should be mapped to.

By default, the connector allows the data handler configured in the connector properties to determine the business object type. It will pass the message body to the data handler and publish the business object returned by the data handler to the broker. If the data handler cannot determine the appropriate business object, the connector will fail the event.

If a static meta-object is specified for connector configuration property ConfigurationMetaObject, the connector searches this object to find a rule that matches the message in terms of input format or input destination. If the rule specified in the meta-object specifies both an input format and input destination, the connector observes this rule only if the message matches both those properties. If one of these properties is missing, the connector uses the specified property only.

For example, a message with input format Cust_In from input destination MyInputDest would match the following static meta-object rules:

  1. InputFormat=Cust_In;InputDestination=MyInputDest
  2. InputFormat=Cust_In
  3. InputDestination=MyInputDest

If it can match the event message to a single rule, the connector will dictate the business object by creating a new instance of this business object and passing it along with the message body to the data handler specified in the rule. If no data handler is specified in the rule, the connector will use the default data handler specified in the connector configuration properties.

If the adapter can match the event message to multiple rules or to none at all, the connector allows the data handler to determine the business object type by passing only the message body to the data-handler specified in the connector configuration properties.

Understanding message header mapping

To transform an event message into a business object, the connector compares metadata about the business object to metadata about the message, mapping one to the other. As described in Metadata and meta-objects, metadata about business objects resides in business object definitions (the application-specific information as well as child dynamic meta-objects), connector properties, and in static meta-objects. Message meta-data is contained in message headers.

To gain access to transport-specific message header information, and to get more information about, and more control over, the message transport, you can add attributes to a dynamic meta-object that is a child of a business object definition. Adding such attributes allows you to read from and optionally write to message headers, thereby modifying message metadata. Such modifications may include changing JMS properties, controlling the destination on a per-request basis (rather than using the default destination specified in the adapter properties), re-targeting a message CorrelationID, and more. When you specify such properties in a dynamic meta-object that is a child of a business object definition, the connector will check for their counterparts in message headers and then populate a dynamic meta-object based on the contents of the message header. You can define one or all of the supported dynamic meta-object attributes; the connector will populate the meta-object accordingly. For further information, including a list of the message header properties that you can read or write to, see Population of the dynamic child meta-object during polling.

Archiving

If you specify the connector-specific property ArchiveDestination, the connector places a copy of all successfully processed messages in this destination. If ArchiveDestination is undefined, successfully processed messages are discarded. For further information see Configuring connector-specific properties.

Error recovery

If it encounters errors reading from the input destination(s), the connector will immediately return a constant value APPRESPONSETIMEOUT to the broker resulting in the termination and possible restart of the connector. Such unrecoverable errors are generally caused by either loss of connection to the JMS provider or internal errors reported by the JMS provider that the connector either does not recognize, or recognizes but deems unrecoverable (for example, transaction failure).

If it encounters errors converting the inbound message to an event business object (for example, the data handler reports invalid message format), the connector will fail the event and log an appropriate error message explaining the reason. If connector property ErrorDestination is defined and valid, the connector will place a copy of the failed message in this error destination; otherwise, the message is discarded.

If the broker reports an error after the connector publishes the event business object, the connector will fail the event and log the error message reported by the broker. If connector property ErrorDestination is defined and valid, the connector will put copy of the failed message in this destination; otherwise, the message is discarded.

If it is unable to determine a business object for a message, or if it publishes a message to a broker and the broker reports that the message is not supported, the connector will consider it unsubscribed. If connector property UnsubscribedDestination is defined and valid, the connector will put a copy of the unsubscribed message in this destination; otherwise, the message is discarded.

Request message processing

When a business object request is sent to the connector, it creates a new message in the target destination. The message header is populated with a combination of user-defined values specified in the request meta-object(s) and default parameters specified by connector properties. The body of the message is populated with the resulting content generated by passing the request business object through the configured data handler.

Figure 2 illustrates a message request communication. When the doVerbFor() method receives a business object from a broker, the connector passes the business object to the data handler. The data handler converts the business object into a suitable message, and the connector issues it as a message to a destination.

Figure 2. Request flow




There are two types of actions the connector can take during request processing. In the first, described below as asynchronous processing, the connector will put a message in the target destination and return successfully. This is commonly referred to as 'fire-and-forget'. In the second, described below as synchronous processing, the connector will again put a message in the target destination but will also wait for a response to be returned by the target application.

The mode of processing is determined by the numeric property ResponseTimeout, which is specified in either the dynamic or static meta-object for the business object request. If this property is not defined or is equal to -1, the connector delivers the request asynchronously. If this property is 0 or greater, the adapter processes the request synchronously, waiting at least that many milliseconds for a response message to be returned by the target application. The request processing illustrated in Figure 2 is described in detail in:

Verb support

The connector places no semantic value on the verb specified in the request business object. It performs the same action, namely putting a message in a JMS destination, regardless of the verb specified.

Asynchronous processing

In asynchronous processing, the connector converts the request business object to a message, places that message in the target destination, and then returns immediately to the broker. The success or failure of the request is based entirely on the ability of the connector to put the message to the JMS destination. Note that the success of this delivery does not imply that the target application has or even will receive the message. Because of the asynchronous nature of messaging systems, a message may remain with the JMS provider indefinitely until the target application is able to process it or expires (if so configured).

The connector first serializes the request business object to text using the configured data handler. The connector uses the data handler that is specified, in order of preference, by:

  1. the dynamic meta-object
  2. the static meta-object
  3. the connector configuration properties

The connector creates a new message containing the serialized business object data as the body of the message. It populates the message headers as described in the following table. In all cases where a property can be specified in either the dynamic or static meta-object, the value specified in the dynamic meta-object takes precedence over any value specified in the static meta-object. For descriptions and a list of which properties you can specify in meta-objects, see Configuring meta-objects.

Table 1. JMS message header population during asynchronous request processing

Meta-object property Default action if property is undefined Action taken if property is defined
OutputFormat
 
Connector does not specify a message format Connector specifies this value for message format.
CorrelationID
 
Connector leaves this value blank in message header. Connector specifies this value for correlation ID in request message header.
ReplyToDestination
 
Connector leaves this value blank in message header. Connector specifies this value for reply destination in request message header.
Priority
 
Connector allows JMS provider to use default priority. Connector sets numeric message priority using this value.
JMSProperties None Connector maps JMS properties specified to JMS properties in message header.

The following attributes in the meta-object determine how the message is delivered:

Table 2. Asynchronous delivery to destination

Meta-object property Default action if property is undefined Action taken if property is defined
OutputDestination
 
A value is required. Target destination for message.
DeliveryMode
 
Connector allows JMS provider to dictate message persistence. Connector writes message persistently/non-persistently as indicated by user.

Depending on the connector's ability to successfully deliver the request message to the output (target) destination, one of the following codes is returned to the broker:

Table 3. Asynchronous return codes

Connector action Return code to broker
Successfully delivers message to target destination. SUCCEED
Fails to deliver due to recoverable errors such as improper or incomplete meta-data, failure of data handler, or general processing problems. FAIL
Fails to deliver due to unrecoverable errors reported by the JMS provider such as connection failure APPRESPONSETIMEOUT

Synchronous processing

In synchronous processing, the connector delivers the request to the target destination and then waits on a second destination for a response message. Creation of the request message is identical to that described in asynchronous processing. However, the connector also checks the following additional attributes in the meta-object:

Table 4. Synchronous meta-object properties

Meta-object property Default action if property is undefined Action taken if property is defined
ResponseTimeout
 
A value is required. Minimum amount of time (in milliseconds) that the adapter should wait for a response message to be returned.
TimeoutFatal
 
If it does not receive response by time specified by ResponseTimeout, connector returns APPRESPONSETIMEOUT to the broker, which typically results in connector termination. If it does not receive response, connector fails request (return FAIL to broker) but does not terminate.

The delivery of the message to the target destination is the same as that described for asynchronous processing except for the following:

Table 5. Synchronous delivery to destination

Meta-object property Default action if property is undefined Action taken if property is defined
ReplyToDestination
 
Same as that for asynchronous. Connector populates this field in request message with value of connector-specific property ReplyToDestination.

The connector waits for a response message from the target application specified by ReplyToDestination for at least as much time as specified by the meta-object attribute ResponseTimeout. If a response is not returned in that time, the connector will timeout and report an error.

Response criteria

The connector does not assume that the first message in the reply destination is the correct response message. Instead, it follows JMS request-response conventions and looks for the first message that has a correlation ID that matches the message ID of the request. In other words, the application that receives the request message must create a response message whose correlation ID equals the request message ID and it must put that message in the reply destination specified by the request message.

Not all applications follow the convention of using the correlation ID to map request and response messages. In such cases, the connector accepts custom criteria for identifying a response message.

Upon receipt of a business object for synchronous request processing, the connector checks for the presence of the name-value pair response_selector= in the application-specific information of the verb. If no such name-value pair exists, the connector identifies the response message using the message correlation ID as described above.

If a response selector name-value pair is defined, the connector considers the value to represent a JMS message selector string that can identify the response message. The following are a few examples of usage; for further information on JMS message selector syntax, see the JMS API specification. Note that the JMS message selector syntax is not parsed by the connector. Rather, the syntax is understood by the JMS provider. The connector makes available the selector to the JMS provider as a means of filtering messages (akin to a database query).

For example, verb application-specific information containing the name-value pair

response_selector=JMSType = 'xmlResponse'
 

informs the connector that the response message must match selector string JMSType = 'xmlResponse'. The connector provides this selector to the JMS provider, which then returns the first message delivered to the reply destination where the JMS type field of the message equals xmlResponse.

In all cases, the message selector string must be capable of uniquely identifying one and only one response. If multiple messages were to be delivered to the reply destination that met the criteria of the response selector, the adapter would retrieve only the first. Any other potential response message matching the criteria would be ignored.

To allow for unique message selectors at run-time, the connector supports dynamic substitutions of attribute values into the message selector itself. To do so, you must specify a placeholder in the form of an integer surrounded by curly braces ("{1}") in the response selector. This must be followed by a colon and a list of comma-separated attributes to use for the substitution. The integer in the placeholder acts as an index to the attribute to use for the substitution.

For example, the following message selector

response_selector=JMSCorrelationID LIKE '{1}':MyDynamicMO.CorrelationID
 

informs the connector to replace the token {1} with the value of attribute CorrelationID in child-object MyDynamicMO. If attribute CorrelationID had a value of 123ABC, the connector would generate and use message selector

JMSCorrelation LIKE '123ABC'
 

You can also specify multiple substitutions as shown below:

response_selector=Name LIKE '{1}'AND Zip LIKE '{2}':PrimaryID,Address[4].AddressID
 

In this example, the connector would substitute '{1}' with the value of attribute PrimaryID from the top-level business object and '{2}' with the value of AddressID from the fifth position (base 0) of child container object Address. With this approach, you can reference any attribute in the business object and meta-object in the response message selector.

To specify the literal value "{" in the message selector, use "{{" instead. For example, the following selector

response_selector=PrimaryID LIKE {{1}
 

would be recognized by the adapter as the literal value

PrimaryID LIKE {1}
 

The connector would not perform any substitution on the value '{1}' in this case.

When the connector encounters special characters such as '{', '}', ':' or ';' in attribute values, they are inserted directly into the query string. This allows you to include special characters in a query string that also serve as application-specific information delimiters. For example, the following selector

Response_selector=PrimaryID = '{1}':Foo
 

when attribute Foo has a value of {A:B};{C:D} would be converted into a literal message selector like

PrimaryID = '{A:B};{C:D}'
 
Response processing

To determine what action to take upon receipt of the response message, the connector checks the JMS result property specified by connector property MessageResponseResultProperty. Depending on the value of this JMS property, the connector expects the response message to contain either a business object or an error message in the message body (see table below). In all cases, the connector returns the corresponding return code to the broker; for example, if the JMS result property equals VALCHANGE in the message, the connector takes the action described below for VALCHANGE and returns the numeric value corresponding to broker constant VALCHANGE to the broker.

Table 6. Response message processing

Value of JMS result property Connector action
SUCCESS Makes no changes to request business object and simply returns successfully to broker.
VALCHANGE
MULTIPLE_HITS
Repopulates request business object with content of response message body. If response message body is empty, request business object is left unchanged.
Repopulates the dynamic meta-object of the request business object with the JMS header fields of the response message.
FAIL FAIL_RETRIEVE_BY_CONTENT BO_DOES_NOT_EXIST UNABLE_TO_LOGIN VALDUPES If response is populated, connector assumes it is an error message and returns it to the broker. If response message body is empty, connector returns generic error message to broker.
APPRESPONSETIMEOUT Same as above except that return of APPRESPONSETIMEOUT to broker normally results in the termination of the adapter agent.
undefined or unrecognized value Connector fails the request.
Error handling

If it encounters errors when reading or writing the request message to the target destination or checking for the response message (as applicable), the connector immediately returns APPRESPONSETIMEOUT to the broker. This results in the termination or possible restart of the adapter. Such unrecoverable errors are generally caused by either loss of connection to the JMS provider or internal errors reported by the JMS provider that the connector does not recognize or recognizes but deems unrecoverable (for example, transaction failure).

If it encounters errors converting a business object to a message or vice-versa (for example, the data handler reports an invalid message format), the connector fails the request and logs an appropriate error message explaining the reason.

For further information including event failure scenarios, see Error handling.

Copyright IBM Corp. 1997, 2003