Chapter 1
ptx/SDI V2.0.0 Release Notes


Introduction

ptx/SDI provides the distributed message passing (DMP) infrastructure to enable the scaling of applications on IBM NUMA-Q systems. It provides for a high performance, low latency, reliable, and scaleable communication facility within a system.

ptx/SDI provides a solution for environments that require scaling beyond a single instance of an application (for example, a relational database).


Product Compatibility

The following products are required by ptx/SDI:

For information about installing Oracle on a system that is running ptx/SDI, read IBM NUMA-Q's "technical tips" documents for the database that you will be installing. Contact your sales representative for access to the "technical tips."


Software Installation

ptx/SDI can be installed along with other DYNIX/ptx products through the standard ptx/INSTALL installation procedure. See the DYNIX/ptx and Layered Products Software Installation Release Notes for instructions on how to install software with ptx/INSTALL.


Setting Site-Specific ptx/SDI Permissions

By default, the ptx/SDI special device files (/dev/sdi/master and /dev/sdi/clone* are created with root-only access (permissions are set to 600 or rw-------). This setting is adequate for all applications that are executed as the root user.

In order to run a database or other application with ptx/SDI that does not execute as the root user, you must change the access permissions, file ownership, or group ownership of the ptx/SDI special device files. Otherwise, an error condition will result and the database or application attempting to use ptx/SDI may fail.

There are four possible ways to set the permissions on the ptx/SDI special device files:

  1. Change the ownership and group of the file.

  2. Establish a common group for all applications that use ptx/SDI.

  3. Change only the ownership of the file.

  4. Change the access permissions only.

These options are discussed in the following sections.


Changing the Ownership and Group of the ptx/SDI Special Device Files

Changing the ownership and group of the ptx/SDI special device fils gives ptx/SDI access to applications executing as the same unique user ID or belonging to the same group. For example, to have applications using ptx/SDI execute as user rdbms1 or be members of the group dba, set the permissions of the device special files, as follows:

  1. As root, change the permissions on the device special files to allow group access:

    # chmod 660 device_special_file

  2. As root, change the owner and group of the device special files to match the user ID and group of the applications using ptx/SDI. (In this example, the user ID is rdbms1 and the group is dba.)

    # chown rdbms1 device_special_file
    # chgrp dba device_special_file


Creating a Common Group for Applications Using ptx/SDI

Not all applications using ptx/SDI will execute as the same user ID. For example, the applications that access the rdbms1 database will most likely be executed as user rdbms1. Other applications that access the rdbms2 database will probably be executed as user rdbms2. In these cases, establish a common group name for all applications that use ptx/SDI. Create the new group by editing the /etc/group file and adding an entry specifying the group name (or ID) and the user IDs that will have access to the new group. In addition, the group ID and permissions of the ptx/SDI special device files must be modified to reflect the new group name and updated access permissions.

In the following example, all applications using ptx/SDI are members of the group sdi:

  1. As root, edit the /etc/group file and add the new group name and the user IDs that are group members. (The entry should be in the form groupname:userid1, userid2, userid3....) To add group name sdi and user IDs rdbms1 and rdbms2, enter the following in the /etc/group file:

    sdi:rdbms1,rdbms2

  2. As root, change the access permissions to allow group access and change the group ID to sdi on the special device files using the chmod and chgrp commands, respectively:

    # chmod 660 device_special_file
    # chgrp sdi device_special_file


Changing the Ownership of the ptx/SDI Special Device Files


ATTENTION

This option is very restrictive and requires that all applications using ptx/SDI be executed with the same unique user ID.


Changing the ownership ID on the ptx/SDI special device files permits applications executing as the same unique user ID to utilize ptx/SDI. This is sufficient if a particular application or group of applications needing access to ptx/SDI executes as a unique user. For example, if applications using ptx/SDI execute as user rdbms1, set the permissions on the device special files as follows:

# chown rdbms1 device_special_file


Changing the Access Permissions Only


ATTENTION

This option allows access to ptx/SDI by any application and creates a security hole because an application can extract or corrupt data. IBM NUMA-Q does not recommend this option.


By default, the access permissions on the ptx/SDI special device files are set to 600 (root-only access). Changing the access permissions to 666 will allow any application to access the ptx/SDI device.

To change the access permissions to 666, issue the following command:

# chmod 666 device_special_file


Edit /etc/rc2.d/S53sdi

Whichever of the previous strategies you choose to secure the ptx/SDI device special files, the associated commands must be added to the /etc/rc2.d/S53sdi file, in order to make the selected strategy available across reboots. All commands that change owner, group, or file permissions need to be added to the file after the "if" statement following the invocation of the /usr/bin/sdi_devbuild command.


Allocating Additional ptx/SDI Clone Devices

By default, 200 clone devices are created at boot-time. This number may be insufficient if a large number of Oracle instances are run on each node. In that case, create the number of clone devices you wish by running this command:

# /usr/bin/sdi_devbuild number_of_devices

The devices created this way are not persistent across boots. In order to avoid having to run the command on each boot, modify the invocation of the command in /etc/rc2.d/S53sdi to specify the desired number of clone devices.


ptx/SDI Administration Utilities

The following administration utilities are provided with ptx/SDI. See the man page for each utility for more information.

lsdom
Lists all of the currently existing ptx/SDI domains, including the domain handle, the ID of the domain, the process ID of the creator of the domain, and the number of endpoints in the domain.

Syntax of lsdom is the following:

/usr/bin/lsdom [-i interval] [-c count] -me

The -i option causes lsdom to loop and repeat its output every interval seconds.

The -c option allows the user to specify the number of times lsdom will loop. If no interval is specified, an interval of 5 seconds is used.

The -m option lists registered memory segments; the -e option lists endpoints.

lsep
Lists all active endpoints, ports, and connections. It returns the following:

Syntax of lsep is the following:

/usr/bin/lsep [-i interval] [-c count] [-IPCD]

The -i option causes lsep to loop and repeat its output every interval seconds.

The -c option lets the user specify the number of times lsep will loop. If no interval is specified, an interval of 5 seconds is used.

The IPCD arguments specify the type of endpoint to list. -I lists inactive endpoints. -P lists ports. -C lists connections. -D lists disconnected ports. The default is to list all types.

lshst
Lists all the hosts that currently appear in the local host's view of the network.

The syntax of lshst is the following:

/usr/bin/lshst [-i interval] [-c count]

The -i option causes lshst to loop and repeat its output every interval seconds. If no interval is specified, an interval of 5 seconds is used.

The -c option allows the user to specify the number of times lshst will loop.

sdi_online
Creates a ptx/SDI configuration with the local node as the only member.

The syntax of sdi_online is:

/usr/bin/sdi_online [hostname]

If hostname is not supplied, sdi_online checks the /var/tcp/hostname file for a valid entry. If a valid entry exists, that entry is used as the hostname. Otherwise, the result of the uname system call is used. If this also fails, an error is displayed.

sdi_offline
Removes the ptx/SDI configuration of which the local node is a member. All processes on the local node using ptx/SDI should have been killed prior to the invocation of this command; otherwise, the command will fail.

The syntax of sdi_offline is:

/usr/bin/sdi_offline

sdiping
Sends a series of test messages to the named host and prints the results.

The syntax of sdiping is:

/usr/bin/sdiping [-c message_count] [-i time_interval] [-p pattern] [-s size] hostname

The -c option specifies the number of messages to send. If you do not use this option, the default number of message that will be sent is 10.

The -i option specifies the interval to wait between successive messages (in seconds). If you do not use this option, the default interval to wait between successive messages is 1 second.

The -p option specifies a character pattern of up to 16 characters to be used in the message.

The -s option specifies the size of the message to send.

hostname is the name of the host to which the test messages are sent. Only the local host can be pinged.

sdisar
Displays ptx/SDI statistics.

The syntax for sdisar is:

/usr/bin/sdisar [-o outputfile] [-ADHT] interval [count]

/usr/bin/sdisar -f inputfile [-ADHT] [-s hh:mm:ss] [-e hh:mm:ss] [-i ss]

When used with the -o option, sdisar stores time-stamped statistics in outputfile. When the -o option is not specified, utilization statistics are formatted to the standard output.

Prepended to the above fields is the timestamp of the sample. The -f option formats data previously recorded in inputfile. The -s and -e options select which samples are to be printed.

In the first form of the command, if the interval count option is not specified, sdisar is run only once.

In the second form of the command, you can tell sdisar to sample data in inputfile every ss number of seconds. For example, to look at data collected every 30 seconds, specify -i 30. If the -i option is not specified, sdisar by default displays every record between the specified start and end times, if any.

In the second form of the command, the starting and ending times of the report can be bounded with the -s and -e hh:mm:ss arguments.

The -D option displays statistics for the domain management layer.

The -H option outputs statistics for the local host.

The -T option outputs per-adapter statistics for all of the hardware adapters used by ptx/SDI.

The -A option does nothing. It is provided for compatibility with sar(1) and can be used in shell scripts that are executed with either sar or sdisar.


Problem Summary

There are no open software defects.


Kernel Error Messages

This section contains kernel error messages specific to ptx/SDI. Kernel error messages are grouped by type: PANIC, WARNING, and NOTICE.

Variables in the error strings are specified by the following conversion strings:

%x
Hexadecimal value
%d
Signed decimal value
%s
String value
%u
Unsigned integer converted to a decimal value

When such a variable appears in an error message, you will see an actual value in that position when the message is displayed on your screen or console.

For more information about conversion strings, refer to the printf(3S) man page.


Panic Messages

The kernel "panics" whenever it detects an irreparable problem in one of its data structures or encounters a hardware error that causes the entire system to fail catastrophically. A panic usually proceeds as follows:

  1. The CPU that panicked sends NMI (nonmaskable interrupt) signals to the other CPUs to shut them down.

  2. The kernel prints the word PANIC: on the console, along with a message describing the reason for the panic.

  3. The kernel prints a hexadecimal stack dump and a stack trace from the process that the CPU was executing at the time of the panic. This is usually, but not always, the process that caused the panic.

  4. The kernel attempts to sync the disks (that is, to copy all cached disk blocks from memory to disk). Occasionally, it will be impossible to complete the sync operation. For example, a data structure needed for the operation might have been locked by one of the other (not paused) CPUs. In this case, the message sync timeout will appear.

  5. The kernel terminates, returning to the power-up monitor. If you have set the monitor's secondary reboot string (the dump string) and have enabled auto-reboot (by using the AUTO button), the monitor automatically invokes the stand-alone dump utility to save a copy of system memory. If auto-reboot is not set, you will see the power-up monitor prompt. At this point, you should manually obtain a dump of system memory.

Panics can be caused by an incorrect system configuration. If you have just installed a new kernel, check its configuration. If the system had been running properly and you are sure it is configured correctly, contact Customer Support.


PANIC: SDI: Detected completed requests on a private endpoint

If a private endpoint has any completed requests queued off of it, it signifies an internal consistency error. Contact Customer Support.


PANIC: SDI: Detected a private endpoint that is not marked as connected

If a private endpoint is not connected, it signifies an internal consistency error. Contact Customer Support.


PANIC: SDI: Detected pending requests on a private endpoint

If a private endpoint has any pending requests queued off of it, it signifies an internal consistency error. Contact Customer Support.


PANIC: SDI: rsdi_decrement_port_hash_entry_reference:tsdihep ->reference_count< 0 !

This is an internal software error. Contact Customer Support.


PANIC: SDI: sdi_postwait_wait(): unexpected semaphore return value

An unexpected value was returned from a p_sema operation. This is an internal consistency error. Contact Customer Support.


PANIC: sdi_use_cntr < 0!

This is an internal software error. Contact Customer Support.



Warning Messages

Warning messages appear on the console when the kernel encounters an error condition that does not affect the entire system. Typically, warning messages are caused by situations such as a request for space in an operating-system data structure that is filled to capacity, or a hardware error that causes an operation to fail.

Some of the following error messages suggest a corrective action. Typically, if a data structure is filled, you can reconfigure the kernel to increase the size of the data structure. If a filesystem is full, you can delete unneeded files from the filesystem or add additional disks. If you have a hardware problem that you cannot correct, contact Customer Support.


WARNING: SDI: Attempt to free invalid semaphore detected

An attempt was made to free a semaphore that was not in the semaphore pool. This is a kernel consistency error. Contact Customer Support.


WARNING: SDI: Can't add sdi clone device to devsw

The device object for an SDI cloneable device entry could not be added to the devsw table. This is a kernel consistency error. Contact Customer Support.


WARNING: SDI: Can't add sdi master device to devsw

The device object for the SDI master device entry could not be added to the devsw table. This is a kernel consistency error. Contact Customer Support.


WARNING: SDI: DML Host database full. File: %s. Line: %d

The number of maximum permissible nodes has been exceeded. Contact Customer Support.


WARNING: SDI: Failed to allocate memory for semaphore pool

A critical kernel memory allocation failed during system initialization. Add more memory to the system or reconfigure the system to use less core memory. If corrective action fails, contact Customer Support.


WARNING: SDI: Failed to return sdi clone device to devsw

The device object for an SDI cloneable device entry could not be returned to the devsw table. This is a kernel consistency error. Contact Customer Support.


WARNING: SDI: Failure allocating devsw entry

An attempt to allocate devsw resources for the SDI master device has failed. Add memory to the system or reconfigure the kernel to require less memory. Otherwise contact Customer Support.


WARNING: SDI: Failure allocating devsw entry

An attempt to allocate devsw resources for the SDI master device has failed. Add memory to the system or reconfigure the kernel to require less memory. Otherwise contact Customer Support.


WARNING: SDI: Failure allocating devops structure

An attempt to allocate devsw resources for the SDI driver has failed. Add memory to the system or reconfigure the system to use less memory. Otherwise contact Customer Support.


WARNING: SDI: Invalid hostname. File: %s. Line: %d

This is an internal software error. Contact Customer Support.


WARNING: SDI: Non-existent domain used to free endpoint

An attempt was made to free an endpoint that was not linked to a domain. This is a kernel consistency error. Contact Customer Support.


WARNING: SDI: Request completion failed

The hardware transport returned an error while completing a callback operation it had set up on a valid request structure. Contact Customer Support.


WARNING: SDI: Request freed using non-existent endpoint

An attempt was made to free a request that was not linked to an existing endpoint. This is a kernel consistency error. Contact Customer Support.


WARNING: SDI: Requested message size not supported. File: %s. Line: %d

The size of message requested for the ping operation is not supported. Reduce the size of the ping message to less than 128 bytes.


WARNING: SDI: SDI was unable to cancel a pending request

An invalid state was detected. The hardware layer was not able to find and cancel a request that is waiting for I/O to complete. Contact Customer Support for a resolution to this problem.


WARNING: SDI: sdi_find_host: Did not find given hostname. File: %s. Line: %d

A connection is being made to a non-existent node. Bring the remote node online or stop the application from making connections to the remote node.


WARNING: SDI: sdi_find_host: Given hostname marked offline. File: %s. " "Line : %d

The connection being made is to a port on a node that is offline. Bring the remote node online or stop the application from making connections to the remote node.


WARNING: SDI: sdi_ioctl_offline failed. Node has been forced offline. File: % s. Line: " "%d

This is an internal software error. Contact Customer Support.


WARNING: SDI rsdi_port_delete: NULL port

rsdi_port_delete called to move the port to the zombie queue on hash string entry. The port is NULL. This is an internal SDI error. Contact Customer Support.


WARNING: SDI sdi_r_cancel called with bad request type: %x, address=%x

sdi_r_cancel was called to cancel an unknown request type. Contact Customer Support.


WARNING: SDI: sdi_r_del_host: Given hostname not found. File: %s. Line: %d

This is an internal software error. Contact Customer Support.


WARNING: SDI: Semaphore pool overflowed

An attempt to allocate additional semaphores to the semaphore table failed. Contact Customer Support.



Notice Messages

Notice messages are not really errors. They occur when something happens that is not expected by the kernel. They do not affect system operations; the system continues to run normally after the message appears.

These messages are sometimes due to hardware or software configuration problems. When you see one of these messages on the console, be sure to check your system configuration.


NOTICE: SDI: new graph generation.

Deletions to the active routing graph have occurred.


NOTICE: SDI: SDI initialization complete.

This is an informative statement only. SDI has completed initialization and will allocate adapters.


NOTICE: SDI: sdi_r_passthrough called, not implemented.

A function was called that has not been implemented yet in ptx/SDI.


NOTICE: SDI: SDI_CTL_MSG_POOL_STRUCT_SIZE = %d

This is an informative message as to the structure size of the memory pool. No corrective action is needed.


NOTICE: SDI: SDI_LOCAL_PORT_POOL_STRUCT_SIZE =%d

This is an informative message about the structure size of the memory pool. No corrective action is needed.


NOTICE: SDI: sdi_short_msg_size = %d

This is an informative message as to the structure size of the memory pool. No corrective action is needed.


NOTICE: SDI: SDI_VECTOR_POOL_STRUCT_SIZE = %d

This is an informative message as to the structure size of the memory pool. No corrective action is needed.


NOTICE: DSDI: SDI_XCHG_POOL_STRUCT_SIZE = %d

This is an informative message as to the structure size of the memory pool. No corrective action is needed.