ptx®/CLUSTERS V2.1.4 Release Notes


Introduction

These release notes support ptx®/CLUSTERS V2.1.4. Read this document before you install or run this release of ptx/CLUSTERS.


Product Compatibility

This version of ptx/CLUSTERS can be used with the following products:

For product compatibility information on other products, such as ptx/LAN and ptx/TCP/IP, consult the DYNIX/ptx Release Notes.


Supported Cluster Configurations

This release of ptx/CLUSTERS supports the following configurations:

This release of ptx/CLUSTERS does not support ``mixed'' cluster configurations of Symmetry 5000 with IBM NUMA systems. This release also does not support the use of Fibre Channel Arbitrated Loop on IBM NUMA systems.


Features in ptx/CLUSTERS V2.x

ptx/CLUSTERS V2.x is a major revision of ptx/CLUSTERS V1.x and includes the following improvements:


Comparison of ptx/CLUSTERS V2.x with ptx/CLUSTERS V1.1 and V1.3

This section compares ptx/CLUSTERS V2.x with ptx/CLUSTERS V1.1 and V1.3 and details what is new, what has stayed the same, and what has been eliminated.


Major New and Changed Features in ptx/CLUSTERS V2.x

The following major features are new with ptx/CLUSTERS V2.x or have changed since previous releases of ptx/CLUSTERS. The ptx/CLUSTERS Administration Guide provides details about all of the features of ptx/CLUSTERS.


New Cluster Systems Communication Services (CSCS)

The CSCS component of ptx/CLUSTERS is a new feature that provides the basic mechanisms for reliable and coordinated communication among member nodes of a cluster. The CSCS replaces the Network and Cluster Management Area Monitor daemon (ncmd) of ptx/CLUSTERS V1.x with components (modules) inside the kernel. Features of the CSCS are the following:

The mechanisms implemented by the CSCS completely replace those of the V1.x releases of ptx/CLUSTERS for connecting active members of the cluster and protecting against cluster partitioning. The CSCS replaces the disk-based CMA mechanism of ptx/CLUSTERS V1.x for cluster configuration and membership specification with reliable group broadcast communication services.


Integrity Manager Daemon Changed

The Integrity Manager daemon still controls and coordinates the activities of the member nodes in the cluster. Although its role has been greatly reduced because of services provided by the CSCS, the Integrity Manager daemon provides the following functions:

In ptx/CLUSTERS V2.x, the Integrity Manager daemon relies on the communication it receives from the CSCS. Because of the CSCS layer, the architecture of the Integrity Manager has changed significantly. These changes include the following:


immbroker Replaced by clustadm

The immbroker(1M) command has been replaced by a much simpler utility, clustadm(1M), to monitor and control a cluster. The following table compares the immbroker command to the clustadm command. Note that many of the options to immbroker are unnecessary in ptx/CLUSTERS V2.x.

immbroker Option

clustadm Equivalent Option or Other Explanation

-A listname

None. ptx/CLUSTERS V2.x does not maintain shared, master, or LAN lists.

-C listname

None. Same as explanation for -A.

-D listname

None. Same as explanation for -A.

-G listname

None. Same as explanation for -A.

-N nodename

Use -m nodename for short output. Use -vm nodename for verbose output, which includes the node name, node index, number of quorum votes contributed by the node, and when it joined the cluster. Use -l or -vl for short or verbose output about the local node.

-S

None. To remove a node from a cluster, the node must be completely shut down.

-Z

None. In ptx/CLUSTERS V2.x, there is no clusters driver to enable devices to be sharable or shared. Any storage device which is accessible by more than one member node is sharable.

-b

None. See explanation for -Z.

-c

Use -m all for short output. Use -vm all for verbose output, which includes the node names, node indexes, number of quorum votes contributed by each node, and when each node joined the cluster.

-d

None. ptx/CLUSTERS V2.x supports the DYNIX/ptx V4.4.x devctl(1M) command for deconfiguring (spinning down) shareable and nonshareable devices.

-e

None. ptx/CLUSTERS V2.x does not maintain shared, master, or LAN lists.

-f

None. See explanation for -e.

-i

None.

-l

-l for short output. -vl for verbose output, which includes the node name, node index, number of quorum votes contributes by the node, and when it joined the cluster.

-m

None.

-n

-m all for short output. -vm all for verbose output, which includes the node names, node indexes, number of quorum votes contributed by each node, and when each node joined the cluster.

-o objectname

None.

-p plexname

None.

-q

-c (for short output) and -vc (for verbose output) each include ``quorum state'' information. There are no Integrity Manager states, such as Halted, Maintenance, and Normal.

-r diskname

None. In ptx/CLUSTERS V2.x, there is no clusters driver to enable devices to be sharable or shared; any device on the system may be sharable as a direct result of its physical proximity.

-s

None. In ptx/CLUSTERS V2.x, a node joins the cluster while it is booting, usually before it reaches single-user mode.

-t

None. There is no CMA concept in ptx/CLUSTERS V2.x.

-u diskname

None. ptx/CLUSTERS V2.x supports the DYNIX/ptx V4.4 devctl(1M) command for configuring (spinning up) sharable and nonshareable devices.

-v

Same as for immbroker.

-w

None.


The clustadm command also lets you set kernel configuration parameters that provide bootstrap information. (During the installation process, you will be required to set these parameters.) The options are listed as follows:

You can optionally configure a quorum disk. The quorum disk concept is discussed in the next section.


New Concepts: Quorum and Quorum Disk

ptx/CLUSTERS V2.x implements a quorum consensus algorithm that the CSCS uses to control cluster availability. This algorithm requires that a majority of the potential cluster nodes be fully connected to enable cluster operation. If the number of nodes available is one-half or fewer than the expected cluster membership, then none of the nodes will function as cluster members with access to shared storage. The expected cluster membership is specified by the user during initial cluster setup, and increments as new nodes join the cluster.

To avoid cessation of activity in a two-node cluster when a single node fails, a quorum disk can be designated as a virtual cluster member. Its ``vote,'' along with the remaining node's, enables the cluster to remain running (with a single node). The quorum disk, which is simply a single partition on a non-mirrored disk, is designated by the user after the cluster has formed.


Cluster Communications Interconnect (CCI) Replaces LLI

The Low Latency Interconnect (LLI) has been renamed the Cluster Communications Interconnect (CCI). The CCI serves the same basic communication purpose as the LLI. The new name more accurately reflects the use and function of the interconnect.


Changes in Logging and Error Handling

The way errors are handled and logged has changed in ptx/CLUSTERS V2.x. All clusters warnings and errors are logged to ktlog. In addition, changes to the cluster membership are logged to /var/clusters/trans_log and changes to a node's application availability are logged to /var/clusters/avail_log.


Fibre Channel Replaces SCSI on IBM NUMA Systems

ptx/CLUSTERS V2.x supports the use of Fibre Channel as the cluster storage interconnect on IBM NUMA systems, significantly increasing the number of shared storage devices. ptx/CLUSTERS V2.x also supports SCSI-based Symmetry systems.


Changes in Node State Terminology and Concepts

In ptx/CLUSTERS V2.x, a local node can have the following states (or modes):

ACTIVE
The node is an active member of the cluster.
NO_QUORUM
The node does not have communication with more than half the expected votes, and is unable to function as a cluster member with access to shared storage.

Unlike previous versions of ptx/CLUSTERS, V2.x does not have Normal, Halted, Maintenance, and InTransition modes.


Cluster Startup Software Invoked Much Sooner

In ptx/CLUSTERS V2.x, cluster software startup is invoked much earlier in the system boot sequence of any node configured as a member of a cluster. A node on which ptx/CLUSTERS V2.x has been installed is expected to become a cluster member as soon as it boots. Cluster formation is attempted as early as possible in the boot sequence so that the nodes can provide the necessary votes for the cluster to reach and maintain quorum and participate in shared-device coordination protocols and other distributed decisions.

A node will wait until it becomes a cluster member before running the /etc/rc2.d scripts to go to multiuser mode.


Changed Shared Devices Design

The ptx/CLUSTERS V1.x shared-device naming and access mechanism (SVTOC), which was layered over the DYNIX/ptx device names and services, has been eliminated. ptx/CLUSTERS V2.x now has mechanisms for device identification and naming that support device access through Fibre Channel Interconnects. Each shared device is assigned a unique identifier, which forms the basis for access locking on each device. These identifiers are guaranteed to be recognized by all nodes of a cluster. The identifiers are unique and remain consistent if the device is connected to a cluster or to a single node.

Since ptx/CLUSTERS V2.x eliminates the separate shared device namespace (shqd--) and its associated shared object lists, the process of configuring and setting up a cluster is greatly simplified.


Features That Remain Since Previous Releases of ptx/CLUSTERS

The following features were part of ptx/CLUSTERS V1.x and are also part of ptx/CLUSTERS V2.x. Though they may have been modified for use with the new software, their external interfaces are unchanged.


Lock Manager

The Lock Manager is a kernel-level component that supports coordination of application access to shared resources. Lock Manager errors in ptx/CLUSTERS V2.x are reported in the new system-standard format.


Lock Manager Domains

You can assign a unique domain name for each application that you plan to run on a cluster to avoid potential resource-name conflicts.


Features and Concepts That Have Been Eliminated in ptx/CLUSTERS V2.x


Maintenance Mode

In ptx/CLUSTERS V2.x, it is no longer necessary to make all cluster configuration changes on one node while the other cluster members are halted. For the following reasons, ptx/CLUSTERS V2.x reliably maintains synchronization among all cluster members without interruption of service:


Active Monitor

The Active Monitor has been replaced in ptx/CLUSTERS V2.x by a more dynamic mechanism for selecting the node responsible for coordinating each cluster transition. This process is transparent to the administrator and the user.


Cluster Management Area (CMA)

In ptx/CLUSTERS V2.x, all usage of the CMA for communication and maintenance of cluster membership and state has been replaced by the CSCS.


Shared Device Names

Shared device names (sh---) are no longer required for sharable devices. Users who wish to migrate from ptx/CLUSTERS V1.x to V2.x, however, can retain the existing sh--- device names if they wish.


Install ptx/CLUSTERS V2.1.4

For instructions on how to update ptx/CLUSTERS, DYNIX/ptx, and other products running on Symmetry systems, see the DYNIX/ptx and Layered Products Software Installation Release Notes.

Only IBM NUMA personnel are authorized to perform initial cluster installations and to upgrade IBM NUMA 2000 clusters. Customer Support or Professional Services personnel who install new clusters or update IBM NUMA 2000 clusters should follow the procedures in the ptx/CLUSTERS V2.x Installer's Guide and in the DYNIX/ptx and Layered Products Software Installation Release Notes for installation and configuration.


Changing Cluster Node ID

Normally, when changing node IDs in a cluster, you need to reboot only the node whose ID you are changing. However, because of a defect in the software (problem report 235185), after changing the ID of one node or more nodes, you need to reboot all nodes.

To change the node ID, follow these steps:

  1. Issue the clustadm -P nodeid=value command, where value is the new node ID (an integer between 0 and 7, inclusive). Issue this command on each node whose ID you wish to change.

  2. Shut down all cluster nodes. The recommended procedure is to first bring all the nodes to run-level 1, and then bring them to the firmware level.

  3. Start the cluster nodes back up.

Failure to follow this procedure can cause the same node to appear multiple times in clustadm output and may cause the Lock Manager to hang.


Using Disks Containing VTOCs with ptx/SVM

If you wish to place a disk containing a VTOC under ptx/SVM control and use the disk in a cluster, then you must assure that each member node's /etc/devtab file contains a VTOC entry for that disk and then issue the devbuild command to create the virtual devices included in that VTOC on all nodes in the cluster.

If you build the VTOC for a disk on one node (where the disk will be recognized as a ``sliced'' disk), but not on the other node(s) (where the disk will be recognized as a ``simple'' disk), then the ptx/SVM shared disk groups will not match across the cluster and you will not be able to use them.

In ptx/CLUSTERS V1.x, if you built a VTOC on a shared device from one of the nodes, the disk's slices were then available on all of the cluster nodes. In ptx/CLUSTERS V2.x, the remaining node(s) will not be aware of the existence of the VTOC slices if you build a VTOC on a shared device from only one of the nodes.


Forced Node Shutdown During Normal Cluster Operations

There are several situations in which it is necessary for the Integrity Manager to reboot a cluster member node. In these situations, the node has become unable to safely resume access to shared storage. The Integrity Manager invokes the kernel panic mechanism to prevent any further user-level activity that might require access to shared storage and to bring the node most rapidly back into cluster membership. The panic messages used, and their causes, are the following:


Guidelines for Removing a Node from a Cluster

To remove a node from a cluster, follow these steps:

  1. Shut down the node you wish to disconnect from the cluster and power it off.

  2. Disconnect all shared storage from the node to be removed from the cluster.

  3. Disconnect the node from the CCI networks.

  4. Boot the node you wish to remove from the cluster. Go to single-user mode, either with the bootflags or by entering s at the Waiting for cluster membership, enter 's' to go to single-user mode prompt.

  5. Through ptx/ADMIN, deinstall the ptx/CLUSTERS software. For information on how to deinstall software, see the DYNIX/ptx and Layered Products Software Installation Release Notes.


    ATTENTION

    To avoid destroying or corrupting data, do not remove the ptx/CLUSTERS software before detaching the node from all shared storage.


  6. On the remaining nodes, reset the number of expected votes to equal the number of remaining nodes plus the quorum disk, if one is configured.


Product Documentation

The following ptx/CLUSTERS V2.1.4 documentation is available on the online HTML documentation CD-ROM:


Problem Reports

This section lists the following problem report summaries:

The numbers in parentheses identify the problems in the problem-tracking system.


Fixed Problems in ptx/CLUSTERS V2.1.4

This release of ptx/CLUSTERS contains fixes for the following software defects.


Open Problems in ptx/CLUSTERS V2.1.4

This section lists open problems in this release of ptx/CLUSTERS.


Node Index Change Causes IMD to List Same IP Address Twice (235185)

When a node index is changed on one cluster node and the node is restarted, the unchanged node has two entries for the node whose index has changed. One entry is for the original node index, and the other entry is for the new node index.

Workaround. See the section in these release notes entitled "Changing Cluster Node ID" for information on how to change a node's index.


Installation of ptx/CLUSTERS Removes ptx/CTC Menu Entries (237168)

ptx/CTC menus in ptx/ADMIN are removed if an updated version of ptx/CLUSTERS is installed and ptx/CTC is not reinstalled.

Workaround. Always install ptx/CLUSTERS and ptx/CTC together. If you have already installed ptx/CLUSTERS, install ptx/CTC from the CD-ROM so that the menus will reappear.


Node Returns to Firmware Level When Booted (238951)

In a single-node cluster with a quorum disk configured, attempting to boot to single-user mode by changing the initdefault entry in /etc/inittab caused the node to loop with the following message:

Cannot satisfy request to go to run-level 1 because this node
does not have cluster quorum. Going to firmware instead.
To get to single-user mode, reboot from firmware.

Workaround. Do not set is:1:initdefault: in /etc/inittab. Instead, use the bootflag option to specify single-user mode.


Migration From ptx/CLUSTERS V1.x to ptx/CLUSTERS V2.x Produces Parsing Error (239495)

During the upgrade procedure from ptx/CLUSTERS V1.x to ptx/CLUSTERS V2.x, the following error may be returned from the parser:

Creating ptx/CLUSTERS devices ...
Adding cscs to /installmnt/etc/services ...
Bad format for immbroker -G shared output in line 3
Line = disk

Workaround. None. This message can be safely ignored.


ptx/CLUSTERS Does Not Recognize CCI Name Changes Done Through devctl (241128)

When you use devctl to change the name of a CCI device, ptx/CLUSTERS does not know the name has changed.

Workaround. Use clustadm to deconfigure and reconfigure the CCI device.


clustadm -C and clustadm -D Hang When Node Has Lost Quorum (241693)

The clustadm -C (configure quorum disk) and clustadm -D (deconfigure quorum disk) commands may cause a node to hang when the node has lost quorum. The commands cannot be suspended or interrupted.

Workaround. Reboot the node and only make quorum disk configuration changes while the node has quorum.


Memory Allocation Failure Causes System to Panic (242517)

A system panic may result on large systems if a memory allocation failure within ptx/CLUSTERS occurs.

Workaround. Reboot the system.


devdestroy Fails for Device That Hosted the Quorum Disk (243003)

When a quorum disk is configured, the VTOC, if it is not already in place, is built for the device on remote nodes from the kernel. However, this does not update the list of built devices at the user level.

Workaround. Execute the devbuild command on the node where the devdestroy is failing. Doing so will update the list of built devices. Then do the devdestroy.