Tuning TCP/IP buffer sizes

WebSphere® Application Server uses the TCP/IP sockets communication mechanism extensively. For a TCP/IP socket connection, the send and receive buffer sizes define the receive window. The receive window specifies the amount of data that can be sent and not received before the send is interrupted. If too much data is sent, it overruns the buffer and interrupts the transfer. The mechanism that controls data transfer interruptions is referred to as flow control. If the receive window size for TCP/IP buffers is too small, the receive window buffer is frequently overrun, and the flow control mechanism stops the data transfer until the receive buffer is empty.

About this task

[AIX Solaris HP-UX Linux Windows] Flow control can consume a significant amount of CPU time and result in additional network latency as a result of data transfer interruptions. It is recommended that you increase buffer sizes avoid flow control under normal operating conditions. A larger buffer size reduces the potential for flow control to occur, and results in improved CPU utilization. However, a large buffer size can have a negative effect on performance in some cases. If the TCP/IP buffers are too large and applications are not processing data fast enough, paging can increase. The goal is to specify a value large enough to avoid flow control, but not so large that the buffer accumulates more data than the system can process.

[AIX Solaris HP-UX Linux Windows] The default buffer size is 8 KB. The maximum size is 8 MB (8096 KB). The optimal buffer size depends on several network environment factors including types of switches and systems, acknowledgment timing, error rates and network topology, memory size, and data transfer size. When data transfer size is extremely large, you might want to set the buffer sizes up to the maximum value to improve throughput, reduce the occurrence of flow control, and reduce CPU cost.

[AIX Solaris HP-UX Linux Windows] Buffer sizes for the socket connections between the Web server and WebSphere Application Server are set at 64KB. In most cases this value is adequate.

[AIX Solaris HP-UX Linux Windows] Flow control can be an issue when an application uses either the IBM® Developer Kit for Java(TM) JDBC driver or the IBM Toolbox for Java JDBC driver to access a remote database. If the data transfers are large, flow control can consume a large amount of CPU time. If you use the IBM Toolbox for Java JDBC driver, you can use custom properties to configure the buffer sizes for each data source. It is recommended that you specify large buffer sizes, for example, 1 MB.

[AIX Solaris HP-UX Linux Windows] Some system-wide settings can override the default 8 KB buffer size for sockets. With some applications, for example, WebSphere Commerce Suite, a buffer size of 180 KB reduces flow control and typically does not adversely affect paging. The optimal value is dependent on specific system characteristics. You might need to try several values before you determine the ideal buffer size for your system.

[z/OS] TCP/IP can be the source of some significant remote method delays.

[AIX] For more information, see section 4.6 in the http://www.redbooks.ibm.com/redbooks/SG247347/wwhelp/wwhimpl/js/html/wwhelp.htm document. In addition, see the http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.prftungd/doc/prftungd/tcp_streaming_workload_tuning.htm document.

[HP-UX] For more information, see the ndd command information in the following documents:
  • http://docs.hp.com/en/B2355-91020/B2355-91020.pdf
  • http://docs.hp.com/en/TKP-90203/index.html

[HP-UX] Also, see the _recv_hiwater_def and tcp_xmit_hiwater_def parameter information in the following document: http://docs.hp.com/en/11890/perf-whitepaper-tcpip-v1_1.pdf

[Linux] For more information, see the following documents:
  • http://www.ibm.com/developerworks/linux/library/l-hisock.html
  • http://fasterdata.es.net/TCP-tuning/linux.html
  • http://www.onlamp.com/pub/a/onlamp/2005/11/17/tcp_tuning.html?page=2

[Solaris] For more information, see section 2.2 in the following document: http://www.redbooks.ibm.com/redbooks/SG247584/wwhelp/wwhimpl/java/html/wwhelp.htm

[Windows] For information on tuning TCP/IP buffer sizes, see the "TCP window size" section of the http://support.microsoft.com/kb/224829 document. Consider setting the TcpWindowSize value to either 8388608 or 16777216.

[z/OS] To change the system wide value, perform the following steps:

Procedure [z/OS]

[z/OS] Tune the TCP/IP buffer sizes.
  1. First, ensure that you have defined enough sockets to your system and that the default socket timeout of 180 seconds is not too high. To allow enough sockets, update the BPXPRMxx parmlib member:
    1. Set MAXSOCKETS for the AF_INET filesystem high enough.
    2. Set MAXFILEPROC high enough.

    We recommend setting MAXSOCKETS and MAXFILEPROC to at least 5000 for low-throughput, 10000 for medium-throughput, and 35000 for high-throughput WebSphere transaction environments. Setting high values for these parameters should not cause excessive use of resources unless the sockets or files are actually allocated.

    Example:
    /* Open/MVS Parmlib Member                                           */
    /* CHANGE HISTORY:                                                   */
    /*   01/31/02 AEK Increased MAXSOCKETS on AF_UNIX from 10000 to 50000*/
    /*                per request from My Developer                      */
    /*   10/02/01 JAB Set up shared HFS                                  */
    
    /* KERNEL RESOURCES              DEFAULT             MIN MAX         */
    /* ========================      =================== === =========== */
       .
       .
       MAXFILEPROC(65535)            /* 64               3   65535       */
    
       .
       .
        NETWORK DOMAINNAME(AF_INET) DOMAINNUMBER(2) MAXSOCKETS(30000)
       .
    
  2. Next check the specification of the port in TCPIP profile dataset to ensure that NODELAYACKS is specified as follows:
    PORT 8082 TCP NODELAYACKS
    
    

    In your runs, changing this could improve throughput by as much as 50% (this is particularly useful when dealing with trivial workloads). This setting is important for good performance when running SSL.

  3. You should ensure that your DNS configuration is optimized so that lookups for frequently-used servers and clients are being cached.

    Caching is sometimes related to the name server's Time To Live (TTL) value. On the one hand, setting the TTL high will ensure good cache hits. However, setting it high also means that, if the Daemon goes down, it will take a while for everyone in the network to be aware of it.

    A good way to verify that your DNS configuration is optimized is to issue the oping and onslookup USS commands. Make sure they respond in a reasonable amount of time. Often a misconfigured DNS or DNS server name will cause delays of 10 seconds or more.

  4. Increase the size of the TCPIP send and receive buffers from the default of 16K to at least 64K. This is the size of the buffers including control information beyond what is present in the data that you are sending in your application. To do this specify the following:
    
    
    TCPCONFIG TCPSENDBFRSIZE 65535
              TCPRCVBUFRSIZE 65535
    
    
    Avoid trouble Avoid trouble: It is unreasonable, in some cases, to specify 256 KB buffers. gotcha
  5. Increase the default listen backlog.
    The default listen backlog is used to buffer spikes in new connections which come with a protocol like HTTP. The default listen backlog is 10 requests. We recommend that you use the TCP transport channel listenBacklog custom property to increase this value to something larger. For example:
    listenBacklog=100
    [z/OS] Note: [Updated in September 2012] The value you use for listenBacklog can be limited by the specification of the SOMAXCONN statement in the TCP/IP profile. If you use a listenBacklog value greater than the SOMAXCONN value, the listenBacklog value is not be used; the value for SOMAXCONN is used.

    IMPORTANT: If listenBacklog is not set for channel types of HTTP, HTTP SSL, IIOP and IIOP SSL, the listenBacklog is set from deprecated environment values: protocol_http_backlog, protocol_https_backlog, protocol_iiop_backlog, and protocol_iiop_backlog_ssl. If the associated deprecated environment value is not specified, a default of 10 is used.

    For channel types that are not HTTP, HTTP SSL, IIOP and IIOP SSL, the default for listenBacklog is 511.

    [Updated in September 2012]
    sep2012
  6. Reduce the finwait2 time. In the most demanding benchmarks you may find that even defining 65K sockets and file descriptors does not give you enough 'free' sockets to run 100%. When a socket is closed abnormally (for example, no longer needed) it is not made available immediately. Instead it is placed into a state called finwait2 (this is what shows up in the netstat -s command). It waits there for a period of time before it is made available in the free pool. The default for this is 600 seconds.
    Avoid trouble Avoid trouble: Unless you have trouble using up sockets, we recommend that you leave this set to the default value. gotcha
    If you are using z/OS® Version 1.2 or above, you can control the amount of time the socket stays in finwait2 state by specifying the following in the configuration file:
    
    
    FINWAIT2TIME 60
    
    

Results [z/OS]

[z/OS] Repeat this process until you determine the ideal buffer size.

What to do next [z/OS]

[z/OS] The TCP/IP buffer sizes are changed. Repeat this process until you determine the ideal buffer size.




In this information ...


IBM Redbooks, demos, education, and more

(Index)

Use IBM Suggests to retrieve related content from ibm.com and beyond, identified for your convenience.

This feature requires Internet access.

Task topic Task topic    

Terms and conditions for information centers | Feedback

Last updatedLast updated: Jun 11, 2013 8:40:09 AM CDT
http://www14.software.ibm.com/webapp/wsbroker/redirect?version=v701sca&product=was-nd-mp&topic=tprf_tunetcpip
File name: tprf_tunetcpip.html