[Java programming language only]

Configuring request and retry timeout values

You can supply tuning options to control how long the eXtreme Scale client code waits for requests to complete, and for how long it attempts to retry requests that are associated with accessing the data grid.

About this task

You can configure settings for the eXtreme Scale client that control how long the client attempts to create network connections, how long the client attempts to process a data grid request to a partition, and how long it attempts to retry that request to the partition, before it returns an exception to your application.

Factors for tuning request and retry timeout values in XIO and ORB

For some tuning options, where you set the values depends on which transport you are using, either eXtremeIO (XIO) or Object Request Broker (ORB). These transport-level tuning options have the initial impact on interactions with your client because they govern how long the transport attempts network socket connections and how long an individual remote procedure call (RPC) analogous to a data grid operation is given to complete.

When you tune these values, consider what your environment can tolerate under peak load conditions as well as steady state conditions. If you tune the intervals too far under the default values (30 seconds for request retry timeout, for example), your operations might fail prematurely. Consider the following factors:
  • Network latencies
  • Coupling of grid interactions with external resources like databases
  • Garbage collection pauses resulting from your combination of heap size, heap usage, and garbage collection tuning policies

ORB settings for tuning request and retry timeout values

The following timeout settings exist for the ORB:
com.ibm.CORBA.ConnectionTimeout
Specifies the amount of time that the ORB attempts to create a socket connection with the remote location. The ORB caches these connections, and therefore, this operation is not done on every request.
com.ibm.CORBA.RequestTimeout
Specifies the amount of time that the ORB waits for an RPC to complete.
com.ibm.CORBA.FragmentTimeout
Reference the IBM ORB documentation for precise details. The product provides default settings for this value.
com.ibm.CORBA.LocateRequestTimeout
Reference the IBM ORB documentation for precise details. The product provides default settings for this value.

As you tune the RequestTimeout and ConnectionTimeout settings, adjusting them based on the default recommendations can be appropriate. You can also set these settings with the same value, where you define these settings that are based on how long you want the request timeout to be.

The next level of tuning is the requestRetryTimeout. With each transport type, after it throws a system exception because an RPC did not complete in time, the data grid can use the additional time that is defined by the requestRetryTimeout setting (for example, the request timeout is 10 seconds, and the retry request timeout is 20 seconds) to specify how long it completes the following actions:
  • Asynchronously asks the catalog server for the latest routing table in case partitions are located elsewhere because of a failover.
  • Takes new routes and retries the request, or stops trying and throws an exception to your application.

The requestRetryTimeout property is set in milliseconds. Set the value greater than zero for the request to be retried on exceptions for which retry is available. Set the value to 0 to fail without retries on exceptions. To use the default behavior, remove the property or set the value to -1.

XIO settings for tuning request retry timeout

With XIO, the following consolidated settings exist:
  • The xioTimeout setting determines how long the XIO transport attempts to establish a network socket connection.
  • There is no equivalent to the LocateRequest setting and the FragmentTimeout setting in the ORB.
  • The requestRetryTimeout setting controls the maximum time that an RPC attempts to succeed, and it helps control how long retries of the RPC occur when new routing information is retrieved from the catalog server. The difference with XIO is that when you use it as a transport, it often reports much sooner than the ORB when an RPC is about to fail. This failure happens sooner than the requestRetryTimeout value expires.

Ways to set request retry timeout

You can configure the request retry timeout value on the client properties file or in a session. The session value overrides the client properties setting. If the value is set to greater than zero, the request is tried until either the timeout condition is met or a permanent failure occurs. A permanent failure might be a DuplicateKeyException exception. A value of zero indicates the fail-fast mode setting and the data grid does not attempt to try the transaction again after any type of transaction.

Transaction timeout and request retry timeout

During run time, the transaction timeout value is used with the request retry timeout value, ensuring that the request retry timeout does not exceed the transaction timeout.

Two types of transactions exist: Autocommit transactions, and transactions that use explicit begin and commit methods. The valid exceptions for retry differ between these two types of transactions:
  • For transactions that are called within a session, transactions are tried again for ORB CORBA SystemException (TransportException for XIO) and eXtreme Scale client TargetNotAvailable exceptions.
  • Autocommit transactions are tried again for CORBA SystemException and eXtreme Scale client availability exceptions. These exceptions include the ReplicationVotedToRollbackTransactionException, TargetNotAvailable, and AvailabilityException exceptions.

Application or other permanent failures return immediately and the client does not try the transaction again. These permanent failures include the DuplicateKeyException and KeyNotFoundException exceptions. Use the fail-fast setting to return all exceptions without trying transactions again after any exceptions.

Exceptions where the client tries the transaction again:
  • ReplicationVotedToRollbackTransactionException (only on autocommit)
  • TargetNotAvailable
  • org.omg.CORBA.SystemException (TransportException is the XIO equivalent of this ORB system exception.)
  • AvailabilityException (only on autocommit)
  • LockTimeoutException (only on autocommit)
  • UnavailableServiceException (only on autocommit)
Permanent exceptions, where the transaction is not tried again:
  • DuplicateKeyException
  • KeyNotFoundException
  • LoaderException
  • TransactionAffinityException
  • LockDeadlockException
  • OptimisticCollisionException

Procedure

Example

Consider the following example, where the client can handle network latency, garbage collection, general contention on the server as a result of setting short timeout values. The requestRetryTimeout property is 10 seconds, and the xioTimeout property matches the ORB ConnectionTimeout value, which is 5 seconds.
Table 1. Data grid configurations for ORB and eXtremeIO transport types
Grid Type ORB XIO
A Java™ or .NET client application that accesses an eXtreme Scale API directly
  • Modify the orb.properties file for your client application. Set the following values:
    • com.ibm.CORBA.RequestTimeout=5
    • com.ibm.CORBA.ConnectTimeout=5
    • com.ibm.CORBA.FragmentTimeout=5
    • com.ibm.CORBA.LocateRequestTimeout=5
    Note: With WebSphere Application Server, you control the ORB settings through the deployment manager, and not through an orb.properties file.
  • Modify the objectGridClient.properties file for the client application with requestRetryTimeout=7000.
Modify the objectGridClient.properties file for your client application with the following values:
  • xioRequestTimeout=50000. This value is in milliseconds and is equivalent to com.ibm.CORBA.RequestTimeout.
  • xioTimeout=5. This value is in seconds and is equivalent to com.ibm.CORBA.ConnectTimeout.
  • requestRetryTimeout=70000. This value is in milliseconds and is also used for the ORB transport.
  • ORB FragmentTimeout and LocateRequestTimeout have no XIO equivalent values.
HTTP session Place the client properties file in the classpath, with requestRetryTimeout=10000. Modify the client properties file for your client application with the following values:
  • xioRequestTimeout=10000. This value is in milliseconds and is equivalent to com.ibm.CORBA.RequestTimeout.
  • xioTimeout=5. This value is in seconds and is equivalent to com.ibm.CORBA.ConnectTimeout
  • requestRetryTimeout=10000. This value is in milliseconds and is also used for the ORB transport.
  • ORB FragmentTimeout and LocateRequestTimeout have no XIO equivalent values
Dynamic cache Set the property on the cache instance:
  • com.ibm.websphere.xs.dynacache.request_retry_timeout_override=2000
You can use the requestRetryTimeout setting with the client properties in the class path instead of this cache instance property, if you want it to be the same for every dynamic cache instance.