ptx/SDI provides the distributed message passing (DMP) infrastructure to enable the scaling of applications on IBM NUMA-Q systems. It provides for a high performance, low latency, reliable, and scaleable communication facility within a system.
ptx/SDI provides a solution for environments that require scaling beyond a single instance of an application (for example, a relational database).
The following products are required by ptx/SDI:
For information about installing Oracle on a system that is running ptx/SDI, read IBM NUMA-Q's "technical tips" documents for the database that you will be installing. Contact your sales representative for access to the "technical tips."
ptx/SDI can be installed along with other DYNIX/ptx products through the standard ptx/INSTALL installation procedure. See the DYNIX/ptx and Layered Products Software Installation Release Notes for instructions on how to install software with ptx/INSTALL.
By default, the ptx/SDI special device files (/dev/sdi/master and /dev/sdi/clone* are created with root-only access (permissions are set to 600 or rw-------). This setting is adequate for all applications that are executed as the root user.
In order to run a database or other application with ptx/SDI that does not execute as the root user, you must change the access permissions, file ownership, or group ownership of the ptx/SDI special device files. Otherwise, an error condition will result and the database or application attempting to use ptx/SDI may fail.
There are four possible ways to set the permissions on the ptx/SDI special device files:
Change the ownership and group of the file.
Establish a common group for all applications that use ptx/SDI.
Change only the ownership of the file.
Change the access permissions only.
These options are discussed in the following sections.
Changing the ownership and group of the ptx/SDI special device fils gives ptx/SDI access to applications executing as the same unique user ID or belonging to the same group. For example, to have applications using ptx/SDI execute as user rdbms1 or be members of the group dba, set the permissions of the device special files, as follows:
As root, change the permissions on the device special files to allow group access:
# chmod 660 device_special_file
As root, change the owner and group of the device special files to match the user ID and group of the applications using ptx/SDI. (In this example, the user ID is rdbms1 and the group is dba.)
# chown rdbms1 device_special_file
# chgrp dba device_special_file
Not all applications using ptx/SDI will execute as the same user ID. For example, the applications that access the rdbms1 database will most likely be executed as user rdbms1. Other applications that access the rdbms2 database will probably be executed as user rdbms2. In these cases, establish a common group name for all applications that use ptx/SDI. Create the new group by editing the /etc/group file and adding an entry specifying the group name (or ID) and the user IDs that will have access to the new group. In addition, the group ID and permissions of the ptx/SDI special device files must be modified to reflect the new group name and updated access permissions.
In the following example, all applications using ptx/SDI are members of the group sdi:
As root, edit the /etc/group file and add the new group name and the user IDs that are group members. (The entry should be in the form groupname:userid1, userid2, userid3....) To add group name sdi and user IDs rdbms1 and rdbms2, enter the following in the /etc/group file:
sdi:rdbms1,rdbms2
As root, change the access permissions to allow group access and change the group ID to sdi on the special device files using the chmod and chgrp commands, respectively:
# chmod 660 device_special_file
# chgrp sdi device_special_file
ATTENTION This option is very restrictive and requires that all applications using ptx/SDI be executed with the same unique user ID.
Changing the ownership ID on the ptx/SDI special device files permits applications executing as the same unique user ID to utilize ptx/SDI. This is sufficient if a particular application or group of applications needing access to ptx/SDI executes as a unique user. For example, if applications using ptx/SDI execute as user rdbms1, set the permissions on the device special files as follows:
# chown rdbms1 device_special_file
ATTENTION This option allows access to ptx/SDI by any application and creates a security hole because an application can extract or corrupt data. IBM NUMA-Q does not recommend this option.
By default, the access permissions on the ptx/SDI special device files are set to 600 (root-only access). Changing the access permissions to 666 will allow any application to access the ptx/SDI device.
To change the access permissions to 666, issue the following command:
# chmod 666 device_special_file
Whichever of the previous strategies you choose to secure the ptx/SDI device special files, the associated commands must be added to the /etc/rc2.d/S53sdi file, in order to make the selected strategy available across reboots. All commands that change owner, group, or file permissions need to be added to the file after the "if" statement following the invocation of the /usr/bin/sdi_devbuild command.
By default, 200 clone devices are created at boot-time. This number may be insufficient if a large number of Oracle instances are run on each node. In that case, create the number of clone devices you wish by running this command:
# /usr/bin/sdi_devbuild number_of_devices
The devices created this way are not persistent across boots. In order to avoid having to run the command on each boot, modify the invocation of the command in /etc/rc2.d/S53sdi to specify the desired number of clone devices.
The following administration utilities are provided with ptx/SDI. See the man page for each utility for more information.
Syntax of lsdom is the following:
/usr/bin/lsdom [-i interval] [-c count] -me
The -i option causes lsdom to loop and repeat its output every interval seconds.
The -c option allows the user to specify the number of times lsdom will loop. If no interval is specified, an interval of 5 seconds is used.
The -m option lists registered memory segments; the -e option lists endpoints.
The endpoint handle returned to the application.
The host and port discriminator of the port.
The type of endpoint.
The domain ID of the domain containing the endpoint.
The process ID of the process containing the endpoint.
The number of endpoints associated with a port.
The number of pending requests queued on the endpoint.
The number of completed requests queued to the domain.
Syntax of lsep is the following:
/usr/bin/lsep [-i interval] [-c count] [-IPCD]
The -i option causes lsep to loop and repeat its output every interval seconds.
The -c option lets the user specify the number of times lsep will loop. If no interval is specified, an interval of 5 seconds is used.
The IPCD arguments specify the type of endpoint to list. -I lists inactive endpoints. -P lists ports. -C lists connections. -D lists disconnected ports. The default is to list all types.
The syntax of lshst is the following:
/usr/bin/lshst [-i interval] [-c count]
The -i option causes lshst to loop and repeat its output every interval seconds. If no interval is specified, an interval of 5 seconds is used.
The -c option allows the user to specify the number of times lshst will loop.
The syntax of sdi_online is:
/usr/bin/sdi_online [hostname]
If hostname is not supplied, sdi_online checks the /var/tcp/hostname file for a valid entry. If a valid entry exists, that entry is used as the hostname. Otherwise, the result of the uname system call is used. If this also fails, an error is displayed.
The syntax of sdi_offline is:
/usr/bin/sdi_offline
The syntax of sdiping is:
/usr/bin/sdiping [-c message_count] [-i time_interval] [-p pattern] [-s size] hostname
The -c option specifies the number of messages to send. If you do not use this option, the default number of message that will be sent is 10.
The -i option specifies the interval to wait between successive messages (in seconds). If you do not use this option, the default interval to wait between successive messages is 1 second.
The -p option specifies a character pattern of up to 16 characters to be used in the message.
The -s option specifies the size of the message to send.
hostname is the name of the host to which the test messages are sent. Only the local host can be pinged.
The syntax for sdisar is:
/usr/bin/sdisar [-o outputfile] [-ADHT] interval [count]
/usr/bin/sdisar -f inputfile [-ADHT] [-s hh:mm:ss] [-e hh:mm:ss] [-i ss]
When used with the -o option, sdisar stores time-stamped statistics in outputfile. When the -o option is not specified, utilization statistics are formatted to the standard output.
Prepended to the above fields is the timestamp of the sample. The -f option formats data previously recorded in inputfile. The -s and -e options select which samples are to be printed.
In the first form of the command, if the interval count option is not specified, sdisar is run only once.
In the second form of the command, you can tell sdisar to sample data in inputfile every ss number of seconds. For example, to look at data collected every 30 seconds, specify -i 30. If the -i option is not specified, sdisar by default displays every record between the specified start and end times, if any.
In the second form of the command, the starting and ending times of the report can be bounded with the -s and -e hh:mm:ss arguments.
The -D option displays statistics for the domain management layer.
The -H option outputs statistics for the local host.
The -T option outputs per-adapter statistics for all of the hardware adapters used by ptx/SDI.
The -A option does nothing. It is provided for compatibility with sar(1) and can be used in shell scripts that are executed with either sar or sdisar.
There are no open software defects.
This section contains kernel error messages specific to ptx/SDI. Kernel error messages are grouped by type: PANIC, WARNING, and NOTICE.
Variables in the error strings are specified by the following conversion strings:
When such a variable appears in an error message, you will see an actual value in that position when the message is displayed on your screen or console.
For more information about conversion strings, refer to the printf(3S) man page.
The kernel "panics" whenever it detects an irreparable problem in one of its data structures or encounters a hardware error that causes the entire system to fail catastrophically. A panic usually proceeds as follows:
The CPU that panicked sends NMI (nonmaskable interrupt) signals to the other CPUs to shut them down.
The kernel prints the word PANIC: on the console, along with a message describing the reason for the panic.
The kernel prints a hexadecimal stack dump and a stack trace from the process that the CPU was executing at the time of the panic. This is usually, but not always, the process that caused the panic.
The kernel attempts to sync the disks (that is, to copy all cached disk blocks from memory to disk). Occasionally, it will be impossible to complete the sync operation. For example, a data structure needed for the operation might have been locked by one of the other (not paused) CPUs. In this case, the message sync timeout will appear.
The kernel terminates, returning to the power-up monitor. If you have set the monitor's secondary reboot string (the dump string) and have enabled auto-reboot (by using the AUTO button), the monitor automatically invokes the stand-alone dump utility to save a copy of system memory. If auto-reboot is not set, you will see the power-up monitor prompt. At this point, you should manually obtain a dump of system memory.
Panics can be caused by an incorrect system configuration. If you have just installed a new kernel, check its configuration. If the system had been running properly and you are sure it is configured correctly, contact Customer Support.
If a private endpoint has any completed requests queued off of it, it signifies an internal consistency error. Contact Customer Support.
If a private endpoint is not connected, it signifies an internal consistency error. Contact Customer Support.
If a private endpoint has any pending requests queued off of it, it signifies an internal consistency error. Contact Customer Support.
This is an internal software error. Contact Customer Support.
An unexpected value was returned from a p_sema operation. This is an internal consistency error. Contact Customer Support.
This is an internal software error. Contact Customer Support.
Warning messages appear on the console when the kernel encounters an error condition that does not affect the entire system. Typically, warning messages are caused by situations such as a request for space in an operating-system data structure that is filled to capacity, or a hardware error that causes an operation to fail.
Some of the following error messages suggest a corrective action. Typically, if a data structure is filled, you can reconfigure the kernel to increase the size of the data structure. If a filesystem is full, you can delete unneeded files from the filesystem or add additional disks. If you have a hardware problem that you cannot correct, contact Customer Support.
An attempt was made to free a semaphore that was not in the semaphore pool. This is a kernel consistency error. Contact Customer Support.
The device object for an SDI cloneable device entry could not be added to the devsw table. This is a kernel consistency error. Contact Customer Support.
The device object for the SDI master device entry could not be added to the devsw table. This is a kernel consistency error. Contact Customer Support.
The number of maximum permissible nodes has been exceeded. Contact Customer Support.
A critical kernel memory allocation failed during system initialization. Add more memory to the system or reconfigure the system to use less core memory. If corrective action fails, contact Customer Support.
The device object for an SDI cloneable device entry could not be returned to the devsw table. This is a kernel consistency error. Contact Customer Support.
An attempt to allocate devsw resources for the SDI master device has failed. Add memory to the system or reconfigure the kernel to require less memory. Otherwise contact Customer Support.
An attempt to allocate devsw resources for the SDI master device has failed. Add memory to the system or reconfigure the kernel to require less memory. Otherwise contact Customer Support.
An attempt to allocate devsw resources for the SDI driver has failed. Add memory to the system or reconfigure the system to use less memory. Otherwise contact Customer Support.
This is an internal software error. Contact Customer Support.
An attempt was made to free an endpoint that was not linked to a domain. This is a kernel consistency error. Contact Customer Support.
The hardware transport returned an error while completing a callback operation it had set up on a valid request structure. Contact Customer Support.
An attempt was made to free a request that was not linked to an existing endpoint. This is a kernel consistency error. Contact Customer Support.
The size of message requested for the ping operation is not supported. Reduce the size of the ping message to less than 128 bytes.
An invalid state was detected. The hardware layer was not able to find and cancel a request that is waiting for I/O to complete. Contact Customer Support for a resolution to this problem.
A connection is being made to a non-existent node. Bring the remote node online or stop the application from making connections to the remote node.
The connection being made is to a port on a node that is offline. Bring the remote node online or stop the application from making connections to the remote node.
This is an internal software error. Contact Customer Support.
rsdi_port_delete called to move the port to the zombie queue on hash string entry. The port is NULL. This is an internal SDI error. Contact Customer Support.
sdi_r_cancel was called to cancel an unknown request type. Contact Customer Support.
This is an internal software error. Contact Customer Support.
An attempt to allocate additional semaphores to the semaphore table failed. Contact Customer Support.
Notice messages are not really errors. They occur when something happens that is not expected by the kernel. They do not affect system operations; the system continues to run normally after the message appears.
These messages are sometimes due to hardware or software configuration problems. When you see one of these messages on the console, be sure to check your system configuration.
Deletions to the active routing graph have occurred.
This is an informative statement only. SDI has completed initialization and will allocate adapters.
A function was called that has not been implemented yet in ptx/SDI.
This is an informative message as to the structure size of the memory pool. No corrective action is needed.
This is an informative message about the structure size of the memory pool. No corrective action is needed.
This is an informative message as to the structure size of the memory pool. No corrective action is needed.
This is an informative message as to the structure size of the memory pool. No corrective action is needed.
This is an informative message as to the structure size of the memory pool. No corrective action is needed.