================================= MACHINE SPECIFIC NOTES FOR IBM Informix Extended Parallel Server 8.32.FC2 PRODUCT RELEASE DATE: 07 Feb 2002 ================================= PORT NUMBER: 205052 (CD ROM) 205053 (ELEC. DELIVERY) I. System Requirements ======================== 1. Supported Operating System Release and Platforms This version of the IBM Informix Extended Parallel Server is certified to run on Digital UNIX V4.0D or higher operating system releases. IBM Informix Extended Parallel Server has been tested by IBM, using the following configurations: Operating System: Digital UNIX V4.0D (Rev. 878) Clustering Software: TruCluster Production Server V1.5 Patches: See note 14 below. Hardware Platforms: AlphaServer 8400 and AlphaServer 4100 Memory Channel: MC 1.5 Operating System: Compaq Tru64 UNIX V4.0F (Rev. 1229) Clustering Software: TruCluster Production Server V1.6 Patches: See note 15 below. Hardware Platforms: AlphaServer 8400 Memory Channel: MC 2.0 Operating System: Compaq Tru64 UNIX V5.1 (Rev. 732) Clustering Software: TruCluster V5.1 Hardware Platforms: AlphaServer ES40 Memory Channel: MC 2.0 2. Required Patches a) V4.0D Systems Patches installed on V4.0D systems came from following patch kits: - DUV40DAS0004-19990723 OSF425 - DUV40DAS0004-19990723 TCR150 Please identify the RDG patch as it ships in this patch kit: - TruCluster_V1.5 / Cluster Patches: Patch 0064.00 - Reliable Datagram (RDG) Messaging Support - DIGITAL_UNIX_V4.0D / Commands, Shells, & Utility Patches: Patch 0585.00 - Fix For dbx Cmd Patch 0607.00 - sys_check Utility Updated to Version 111 Patch 0644.00 - Incorrect Mount Of Memory Copied In LMF - DIGITAL_UNIX_V4.0D / Filesystem Patches: Patch 0625.00 - Fix For mount Command - DIGITAL_UNIX_V4.0D / Kernel Patches: Patch 0116.02 - Kernel Class Scheduler And libclass.a Correction - DIGITAL_UNIX_V4.0D / Library Patches: Patch 0422.00 - libaio Shared Library Fix Patch 0423.00 - libaio Static Library Fix Patch 0601.00 - libtli/libxti Shared Library Correction Patch 0627.00 - malloc Performance Enhancement Patch 0632.00 - libtli/libxti Static Library Correction - DIGITAL_UNIX_V4.0D / Security Related Patches: Patch 0046.01 - Security, (SSRT0495U) Patch 0054.00 - Security Correction Patch 0079.00 - libDtSvc Fix, Security (SSRT0498U) Patch 0172.00 - Security (SSRT0498U) Patch 0179.00 - Security, (SSRT0456U) Patch 0342.02 - Security (SSRT0587U) Patch 0346.02 - Security (SSRT0548U, SSRT0412U) Patch 0382.03 - CDE Fixes, Security (SSRT0547U) Patch 0412.02 - Security (SSRT0547U) Patch 0441.00 - Security Patch (SSRT0589U) Patch 0451.00 - Security (SSRT0588U) Patch 0452.00 - Security (SSRT0588U) Patch 0504.00 - Security (SSRT0566U) Patch 0506.00 - Security (SSRT0585U) Patch 0507.00 - Security (SSRT0525U, SSRT0580U) Patch 0512.00 - Security (SSRT0565U) Patch 0531.00 - Security (SSRT0487U, SSRT0567U, SSRT0583U) Patch 0533.00 - Security (SSRT0556U) Patch 0541.00 - Security (SSRT0546U, SSRT0542U) Patch 0553.00 - Security (SSRT0571U) Patch 0610.00 - Security (SSRT0583U) Patch 0634.00 - Security (SSRT0546U, SSRT0542U) Patch 0639.00 - Security (SSRT0546U, SSRT0542U) Patch 0640.00 - Security (SSRT0521U, SSRT0537U) - DIGITAL_UNIX_V4.0D / Software Development Environment Patches: Patch 0057.01 - Segfaults In nm For C++ Compiler Correction Patch 0288.02 - Compiler Correction Patch 0348.02 - lex Command Correction Patch 0599.00 - Various Compiler Fixes Patch 0603.00 - Support for Developers Toolkit Update Kit Patch 0604.00 - Support for VTI Tool Patch 0633.00 - Update for Developers Toolkit - TruCluster_V1.5 / ASE Availability Manager (AM) Patches: Patch 0035.01 - ASE Service Modification Patch 0046.00 - LSM Disk Not Updated in ASE Database - TruCluster_V1.5 / Cluster Patches: Patch 0064.00 - Reliable Datagram (RDG) Messaging Support b) V4.0F Systems Patches installed on V4.0F systems came from following patch kits: - DUV40FAS0001-19990609 OSF440 - DUV40FAS0001-19990609 TCR160 Patch 2.00 - Security (SSRT0571U) Patch 3.00 - Security (SSRT0585U) Patch 4.00 - Security (SSRT0580U) Patch 6.00 - Security (SSRT0600U) Patch 11.00 - Security (SSRT0596U) Patch 12.00 - Security (SSRT0567U) Patch 16.00 - Fixes Kernel Memory Fault Caused By SMP Race Patch 18.00 - Resolves Corrupt EV6 Binary Error Log Entries Patch 28.00 - Security (SSRT0556U) Patch 31.00 - Security (SSRT0563U) Patch 34.00 - Fix For kio Subsystem Panic Patch 44.00 - Fix For Kernel Memory Fault Patch 48.00 - DECthreads Library Fix Patch 54.00 - AdvFS volumes Not Setting I/O Byte Transfer Size Patch 58.00 - Additional Error Detection for FC Driver Patch 66.00 - libxti/libtli Static Library Fix c) TruCluster_V1.6 / Cluster Patches: Patch 1.00 - Support for Reliable Datagram Messaging d) V5.0A Systems Patches installed on V5.0A systems came from following patch kits: Compaq Tru64 UNIX V5.0A Initial Patch Kit-0001 - T64V50AAS0001-20000718 Tru64 Patch 96.00 Fix for sendmsg, nsendmsg, recvmsg, nrecvmsg (OSF505-084) 3. DG Information This version of IBM Informix Extended Parallel Server supports two system interconnect communications transports, RDG and TCP. RDG is the default DG configuration. UDP can also be used as the transport, but UDP is not supported across the cluster. You can use UDP as the transport on a single node that hosts multiple coservers. Both RDG and TCP are "Hybrid DataGram Layer", meaning that they also utilize a shared memory segment for intra-node message passing. RDG and TCP addressing can be automatically configured by default, with the port numbers starting at 16000, and incremented by 100 for each coserver. This will create one endpoint for each size of message for each cpuvp. If there is port number contention, which might occur when you start 2 servers on one machine, the starting port number can be changed. Refer to the DGINFO section below. You can also explicitly set the TCP addressing for each coserver in the COSERVER section of the ONCONFIG file. To override the automatic configuration setting, specify the SADDR, LADDR, and HADDR strings. Do this if the hostname of the interface you want to use is NOT the same name as returned by the 'hostname' command. NOTE: Hostname interface for "memory channel" is different between Tru64 UNIX Version V4.x and V5.x. In Tru64 UNIX version V4.x, "memory channel" hostname interface is prefixed with "mc" followed by the return value of the 'hostname' command. In Tru64 UNIX version V5.x, "memory channel" hostname interface begins with the return value of 'hostname' command and suffixed with "-mc0". For example, if return value of 'hostname' is node1, the "memory channel" hostname interface would be the followings in the two versions of the OS: In Tru64 UNIX V4.x: mcnode1 In Tru64 UNIX V5.x: node1-mc0 The format for the RDG and TCP addressing is: :,, SIZE = SADDR - Small endpoint size (default 1K) LADDR - Large endpoint size (default 8K) HADDR - Huge endpoint size (default 48K) Host = Hostname of IP interface for inter-connect (NOTE: RDG will use the memory channel by default) Port = Starting IP port number of endpoint to open Number = Number of consecutive endpoints to open Domain = Communications domain that owns endpoints (NOTE: Domain numbering starts at 0, not 1) Here is an example of the COSERVER section configured with 2 coservers, each having 2 domains, with 1 small, 1 large, and 1 huge endpoint per domain: # ....................................................... COSERVER 1 NODE node1 SADDR node1:8000,1,0 LADDR node1:8001,1,0 HADDR node1:8002,1,0 SADDR node1:8010,1,1 LADDR node1:8011,1,1 HADDR node1:8012,1,1 END # ....................................................... COSERVER 2 NODE node2 SADDR node2:8000,1,0 LADDR node2:8001,1,0 HADDR node2:8002,1,0 SADDR node2:8010,1,1 LADDR node2:8011,1,1 HADDR node2:8012,1,1 END # ....................................................... The DGINFO directive in the ONCONFIG file is used to pass in platform- specific information to the DG layer. The format of DGINFO is a string of comma-separated parameters and values. Below is a listing of available DGINFO parameters and default values. DG, Specify DG library choice. Default: rdg Also available: tcp, udp PORT, Specify starting port number Default: 16000 SB_MAX, Specify socket buffer size (TCP only). Default: 262144 NO_SHM Disable the use of DG shared memory. SHM_SBUFS, Specify number of small shared memory buffers per domain. Default: 64 SHM_LBUFS, Specify number of large shared memory buffers per domain. Default: 96 SHM_HBUFS, Specify number of huge shared memory buffers per domain. Default: 8 Below is an example of a DGINFO line that sets the DG library to use TCP, lowers the SB_MAX to 32K bytes, and disables DG shared memory usage: DGINFO DG,tcp,SB_MAX,32768,NO_SHM Please note that configuration of RDG support with the indicated patches (Section 1 above) will require a Digital UNIX kernel rebuild. When more than one coserver is used, at initialization an informational message is printed in the online msglog file to indicate the protocol used by the IBM Informix Extended Parallel Server instance. RDG-Specific information ======================== a) Ensure that the RDG patch is installed from either the V4.0D or V4.0F patch kits as detailed in sections 13 and 14 below. b) The setting of ASYNCRQT in the onconfig file must match the parameter max_async_req in /etc/sysconfigtab under the RDG stanza. For instance, if we need to set ASYNCRQT to 150, we need to add the following to /etc/sysconfigtab: in /etc/sysconfigtab: rdg: max_async_req=150 in ONCONFIG file: ASYNCRQT 150 c) To enable HADDR, the following should to be added t /etc/sysconfigtab under the RDG stanza rdg: msg_size=49152 NOTE: If we do not set this variable all huge buffer packets are split and transmitted in 8K buffer sizes. II. System Configuration ========================= 1. Shared Memory ONCONFIG Parameters The ONCONFIG variable SHMBASE should be set to the following for high memory. We recommend high memory setup SHMBASE 0x200000000L or for low memory SHMBASE 0x800000 2. OS Specific Parameters OS Specific Parameters for IBM Informix Extended Parallel Server on Compaq AlphaServers running Digital UNIX V4.0D. In the file /etc/rc.config, please change the following two parameters: IMC_MAX_ALLOC="110" export IMC_MAX_ALLOC IMC_MAX_RECV="110" export IMC_MAX_RECV You might also want to adjust parameters/values in /etc/sysconfigtab. We use the following values to build and test the server: rm: rm-no-inheritance = 1 rt: aio-max-num = 16384 aio-max-percent = 3 aio-max-retry = 10 aio-task-max-num = 8192 sigqueue-max-num = 8192 io: dma-sg-map-unload-zero = 0 ipc: ssm-threshold = 0 sem-ume = 500 sem-opm = 500 sem-msl = 1000 sem-mni = 500 msg-tql = 1024 msg-mni = 128 shm-seg = 2048 shm-mni = 512 shm-max = 2147483647 vm: vm-mapentries = 2048 new-wire-method = 0 vm-maxvas = 17179869184 vm-heappercent = 60 proc: max-proc-per-user = 27990 max-threads-per-user = 27990 per-proc-stack-size = 33554432 max-per-proc-stack-size = 67108864 per-proc-data-size = 8589934592 max-per-proc-data-size = 274877906944 max-per-proc-address-space = 17179869184 per-proc-address-space = 17179869184 maxusers = 1024 task-max = 28000 thread-max = 28000 num-wait-queues = 1024 num-timeout-hash-queues = 1024 socket: somaxconn = 65000 sominconn = 65000 inet: tcbhashsize = 32768 lsm: max-vol = 4093 generic: msgbuf_size = 16384 NOTE: for Tru64 UNIX version 5.0 and above, naming convention of attributes is changed. Dashes are replaced by underlines. Legacy attributes with dashes are supported for backward-compatibility. 3. Reserving Physical Memory for Shared Memory using gh-chunks gh-chunks is a kernel attribute that can be specified in the /etc/sysconfigtab kernel file to enable granularity hints. The value specified in gh-chunks pre-configured memory pages from which shared memory allocations can be made. The advantage of using gh-chunks is in the mapping of large (4-MB) pages compared to the default OS page size of 8 KB, resulting in substantial performance gains. How to configure gh-chunks -------------------------- When running IBM Informix Extended Parallel Server, observe the amount of shared memory used for a good performing system using "onstat -g seg". Compute the total shared memory from the "size" column. Divide this value by 4194304 (4 MB) to arrive at a value to be used for the gh-chunks kernel attribute. Things to Note: --------------- Depending on the demand for the server memory based on the applications running on the system, using gh-chunks may not always be beneficial. This is because VM will have less memory available in which to manage the working application's memory requirements. If IBM Informix Extended Parallel Server is not exclusively using the node, do not allocate an excessive amount of memory for gh-chunks. Also, monitor the memory usage and paging activity on the system using 'vmstat' or 'sar' to make sure that you do not suffer paging penalties because you have left too little memory for VM. VM available (user+kernel) memory = total physical - gh-chunks memory 4. Reading current time from Kernel memory Digital UNIX provides a mechanism to directly read the current time from kernel memory. In order to use this feature, it is necessary for the device /dev/sysdev0 to be available on all machines where IBM Informix Extended Parallel Server is running. If this device is not available, the DBA/Admin should create it using the following command: As user root; cd /dev; ./MAKEDEV sysdev0; ls -l /dev/sysdev0 III. Features Supported on This Platform ======================================== 1. Supported Client Protocols and Interfaces The following protocols and interfaces are supported for this platform: a) Berkeley sockets using TCP/IP To use TCP/IP, the NETTYPE in the ONCONFIG file and the nettype field in the sqlhosts file entry must be set to "onsoctcp". b) IPC using shared memory To use IPC with shared memory, the NETTYPE in the ONCONFIG file and the nettype field in the sqlhosts file entry must be set to "onipcshm". NOTE: ontlitcp is not supported. 2. Kernel AIO is supported on this platform. On a heavily loaded machine, you may run out of KAIO resources. Please refer to the Compaq/Digital Documentation to reconfigure the AIO resources used by the Kernel. If you run out of KAIO resources, please check the following kernel configuration attributes: aio-max-num aio-max-percent aio-max-retry aio-retry-scale aio-task-max-num sigqueue-max-num NOTE: For Tru64 UNIX version 5.0 and above, the naming convention of attributes is changed. Dashes are replaced by underlines. Attribute aio-max-num is removed; use aio_task_max_num instead. The new related attributes are: aio_max_percent aio_max_retry aio_retry_scale aio_task_max_num sigqueue_max_num Legacy attributes with dashes are supported for backward-compatibility. 3. TERMINFO is supported on this platform. 4. Chunk Size This version of IBM Informix Extended Parallel Server supports maximum chunk size of up to 128 GB, increased from the previous limit of 2 GB. This limit applies to files created on file systems (cooked files) or raw partitions. NOTE: dbspaces with large chunks cannot be reverted to previous version 8.21 servers. 5. Conversion and reversion of IBM Informix Extended Parallel Server from 8.30 (FC4) to 8.32 is supported. Please read the Migration Guide for more detailed information. 6. Research SNMP based OnSnmp is supported in this release. The SNMP Research package needs to be installed on the machine and the /etc/srconf/snmpinfo.dat file needs to be merged/updated with sqldist/snmp/snmpr/snmpinfo.dat for OnSnmp to work correctly. 7. The IBM Informix Extended Parallel Server GUI installer is supported. 8. Operating System Auditing Support This section gives the necessary configuration changes to the O audit subsystem in order for IBM Informix Dynamic Server audit records to be recorded in the system audit log. a) The OS audit subsystem must be operational, and auditing enabled. b) A new event INFORMIX_DBMS must be introduced to the audit subsystem. c) The auditing must be configured for binary or stream mode. To receive IBM Informix OnLine audit data, the operating system audit subsystem must be configured to accept events of the type INFORMIX_DBMS. The event type INFORMIX_DBMS must be introduced to the OS, by inserting a line defining this event in the file /etc/sec/site_events. (Please pay careful attention to the format, even of the comments, because they are parsed by the auditselect and perhaps by other audit library functions at runtime.) Lines extracted from /etc/sec/site_events: INFORMIX_DBMS 2048, informix_event1 0; Important files in configuring OS audit subsystem: /etc/sec/site_events, /var/audit/auditlog After configuring the system, enable auditing with the new configuration. Refer to the operating system documentation on how to accomplish this. The OS auditing must be enabled for all users who might use the IBM Informix Extended Parallel database server, and for the event type INFORMIX_DBMS. Please see the man pages for "auditsetup", "auditmask", "auditgen", "audittool" and other related commands for details on configuring auditing on the system. To select which IBM Informix Extended Parallel Server events are to be audited, specify audit masks using the "onaudit" utility. Refer to Informix OnLine Trusted Facility Manual (TFM) and Security manual of Compaq/Digital UNIX for further details. To extract audit records from operating system audit trail files, "audit_tool" utility can be used. To extract IBM Informix Extended Parallel Server records use "onshowaudit -O" which is configured to use the current audit log by default. If the system is configured to use a different audit file, use "onshowaudit" utility with -f option to specify the required audit file. For example, to use /var/audit/auditlog.001 file, specify: onshowaudit -O -f "/var/audit/auditlog.001". The following steps should be taken to configure the Operating System Audit subsystem: 1. Verify that the enhanced security subsets (OSFC2SEC425 and OSFXC2SEC425) are installed. If the subsets are not installed, install them, using the Installation Guide, if you need more information. 2. Log in as root. 3. Run the interactive secsetup command selecting ENHANCED security when prompted. 4. Bring your system down to single user and reboot. 5. Run the audit_setup script to configure the auditd and auditmask parameters. Make sure that the destination of the audit data is /var/audit/auditlog, which is the default when you run the audit setup script. 6. As root, edit the file /etc/sec/site_events. Add the following entry: INFORMIX_DBMS 2048, informix_event1 0; If the file /etc/sec/site_events does not exist, the system administrator must create it. Run 'auditmask -f' to enforce the new audit mask. NOTE 1: OS Auditing system has some limitations (Defect 97000). Please refer to the Release Notes for more details. NOTE 2: If the operating system is configured to run in enhanced security mode, oninit must be either started by user "root" or by user informix to which the membership of the group "auth" is added. If the above condition is not met, any database connections to the IBM Informix server will fail with errors indicating problems in user/password information. For example, 951: Incorrect password or user informix is not known on the database server The connection failure is caused by the extra protection enforced by the operating system for the user password information under enhanced security.