IBM(R) Total Storage Multipath Subsystem Device Driver Version 1.6.0.1-4 README for Linux ------------------------------------------------------------------------------- CONTENTS 1.0 About this README file 1.1 Who should read this README file 1.2 How to get help 2.0 Prerequisites for SDD 2.1 Supported HBA Driver Levels 2.2 Supported Filesystems 3.0 SDD Change History 3.1 Defects Fixed 3.1.1 Common 3.1.2 ESS/DS8000/DS6000 Defects 3.1.3 SVC Defects 3.1.4 SVCCISCO Defects 3.2 New Features 3.3 Feature Details 3.4 Known Issues 3.5 Correction to User's Guide 4.0 User License Agreement for IBM Device Drivers 5.0 Notices 6.0 Trademarks and Service Marks ------------------------------------------------------------------------------- 1.0 About this README file Welcome to IBM Totalstorage Multipath Subsystem Device Driver (SDD). This README file contains the most recent information about the IBM Totalstorage Multipath Subsystem Device Driver, Version 1.6.0.1-4 for Linux. IBM recommends that you go to the following Web site to get the most current information about this release of SDD: http://www.ibm.com/servers/storage/support/software/sdd.html You should carefully review the following information available through the website: 1. The most current README file. This will contain corrections to this readme file, corrections to the SDD User's Guide, and other documentation updates discovered since this copy of the README was prepared. 2. The Multipath SDD User's Guide. Because SDD can be installed in so many different environments / configurations, detailed information about each environment is placed in the appropriate chapter of the Multipath SDD Users Guide. The 'Summary of Changes' section of the SDD Users' Guide can help you quickly determine if the latest changes affect you. 3. The Flashes. As we become aware of any information that is likely to impact a broad set of our customers, Flashes are prepared and posted on this site. You should review this section periodocally to see any new Flashes that have been posted since your last review. For prerequisites information, be sure to look in the Multipath SDD User's Guide as well as the prerequisites section of this readme file for the latest updates. 1.1 Who should read this README file This README file is intended for storage administrators, system programmers, and performance and capacity analysts. The information in this file only applies to customers who run: 1. ESS, DS8000, DS6000, SAN Volume Controller, or SAN Volume Controller for Cisco MDS 9000 2. ESS and SAN Volume Controller 3. ESS and SAN Volume Controller for Cisco MDS 9000 1.2 How to get help Go to the following Web site for SDD technical support and for the most current SDD documentation and support information: http://www.ibm.com/servers/storage/support/software/sdd.html Go to the following Web site for the IBM ESS Open Systems support: http://www.storage.ibm.com/hardsoft/products/ess/supserver.htm Go to the following Web site for IBM TotalStorage DS8000 support: http://www.ibm.com/servers/storage/disk/ds8000/index.html Go to the following Web site for IBM TotalStorage DS6000 support: http://www.ibm.com/servers/storage/disk/ds6000/index.html Go to the following Web site for IBM TotalStorage SAN Volume Controller support: http://www.ibm.com/servers/storage/support/virtual/2145.html Go to the following Web site for IBM TotalStorage SAN Volume Controller for Cisco MDS 9000 support: http://www.ibm.com/servers/storage/support/virtual/2062-2300.html Call one of the following numbers to obtain nontechnical or administrative support, such as hardware and software orders, hardware maintenance, services contract of entitlement, and invoices: .For commercial or state and local support operations: 1-877-426-6006 (Listen to the voice prompts) .For business partner support operations: 1-800-426-9990 .For federal government support operations: 1-800-333-6705 ------------------------------------------------------------------------------- 2.0 Prerequisites for SDD One of the following combinations of a Linux distribution and associated vendor compiled binary kernel packages: SuSE SLES 9 (x86) with one of the following kernels: kernel-smp-2.6.5-7.139 kernel-bigsmp-2.6.5-7.139 kernel-smp-2.6.5-7.145 kernel-bigsmp-2.6.5-7.145 kernel-smp-2.6.5-7.147 * kernel-bigsmp-2.6.5-7.147 * SuSE SLES 9 (ppc64) with one of the following kernels: kernel-pseries64-2.6.5-7.139 kernel-pseries64-2.6.5-7.145 Red Hat Enterprise Linux 4 (x86) with one of the following kernels: kernel-smp-2.6.9-5.EL * kernel-hugemem-2.6.9-5.EL * kernel-smp-2.6.9-5.0.3.EL * kernel-hugemem-2.6.9-5.0.3.EL * * Newly supported in this release. =============================================================================== 2.1 Supported HBA Driver Levels Intel (x86) o Qlogic qla2x00 8.00.00 pSeries (ppc64) o Emulex lpfc 2.10g =============================================================================== 2.2 Supported Filesystems o ext3 o reiserfs o xfs ------------------------------------------------------------------------------- 3.0 SDD Change History =============================================================================== 3.1 Defects Fixed 3.1.1 Common 1.6.0.1-4 1.6.0.1-3 o 3156 Fix potential system hang during sdd.log trace collection. 1.6.0.1-2 o 3125 Fix incorrect creation and handling of exclusion list used by SDD during configuration. 3.1.2 ESS defects None 3.1.3 SVC defects None 3.1.4 SVCCISCO defects None =============================================================================== 3.2 New Features 1.6.0.1-4 o PCR 2911 Add support for Red Hat Enterprise Linux 4 (Linux 2.6 kernel). 1.6.0.1-3 o PCR 2886 Add "-l" option to datapath query device command to indicate paths to a non-preferred controller. 1.6.0.1-2 o PCR 2447 Add support for SuSE Linux Enterprise Server 9 Service Pack 1 (Linux 2.6 kernel). =============================================================================== 3.3 Feature Details 1.6.0.1-4 o PCR 2911 Add support for Red Hat Enterprise Linux (RHEL) 4 running the Linux 2.6 kernel. See above for supported kernel levels. 1.6.0.1-3 o PCR 2886 A new feature has been introduced to datapath to aid users to in verifying their SAN configurations in a controller environment (such as with the SAN Volume Controller). The datapath query device command now has a new option '-l' to display paths to non-preferred controllers. For example, if users have 4 paths per SDD vpath device and they would like to configure a equal distribution between preferred controller and non-preferred controller, they will configure their environment with 2 paths from a preferred controller and another 2 paths from a non-preferred one. This feature will help users to verify their configuration by indicating which path is from the non-preferred controller before the device starts operation. For 'datapath query device', a new option, '-l', is added to mark the non-preferred paths with an asterisk. This option can be used in addition to the existing datapath query device command. For example: - datapath query device -l - datapath query device -l In the datapath query device output, the non-preferred paths are marked with a *, as below: DEV#: 35 DEVICE NAME: vpathbd TYPE: 2145 POLICY: Optimized Sequential SERIAL: 60050768018b800a800000000000008c LUN IDENTIFIER: 60050768018b800a800000000000008c ============================================================================ Path# Adapter/Hard Disk State Mode Select Errors 0* Host3Channel0/sdcu CLOSE NORMAL 0 0 1 Host3Channel0/sddc CLOSE NORMAL 22985 0 2 Host2Channel0/sds CLOSE NORMAL 26398 0 3* Host2Channel0/sdk CLOSE NORMAL 0 0 1.6.0.1-2 o PCR 2447 Add support for SuSE Linux Enterprise Server (SLES) 9 Service Pack 1 running on the Linux 2.6 kernel. Note that the Linux 2.6 kernel is significantly different from the Linux 2.4 kernel supported on our previous releases. See above for supported kernel levels. =============================================================================== 3.4 Known Issues o SAN Fabric Configuration and Maintenance Limitations A number of maintenance operations in SAN fabrics have been observed to occasionally cause I/O errors for Linux hosts. To avoid these errors, I/O on Linux hosts must be quiesced prior to doing any type of SAN re-configuration activity, switch maintenance or storage maintenance. During Concurrent Code Load (CCL), I/O errors have been occasionally observed. All I/O must be quiesced on these systems before CCL is started and must not be restarted until the code load is complete. All host paths must be online, and the fabric must be fully redundant with no failed paths. o Module loading at boot time When installing with supported fibre-channel Host Bus Adapters (HBAs), SLES 9 will load the fibre-channel adapter driver earlier in the OS boot-up order than the internal SCSI adapter card driver. This may cause problems because the internal disks are usually referenced using a static device entry (such as "/dev/sda3") in the /etc/fstab entry for the OS to load the root filesystem disk. The loading of the HBA driver could cause a disk on the SAN (such as from an ESS) to be loaded as "/dev/sda" and the real local root disk to be moved to something like "/dev/sdb". This static naming convention could cause your system to crash at boot time. One workaround is to reorder the entries in the configuration file /etc/sysconfig/kernel. This file indicates the order in which the drivers are loaded in the initial ramdisk image Linux uses in order to boot (called the initrd). The INITRD_MODULES parameter determines the driver load order at boot time. You might have something that looks like this: INITRD_MODULES="lpfcdd sym53c8xxi" where "lpfcdd" is the Emulex HBA driver and "sym53c8xxi" is the internal SCSI driver. You would want to place the lpfcdd entry after the internal SCSI driver entry, such as this: INITRD_MODULES="sym53c8xxi lpfcdd" After you change this entry, run the command "mkinitrd." This will create a new initial ramdisk image with the driver load order changed. Next time you reboot the system, the new order will go into effect. If you have configured your SAN LUNs already and the system is currently crashing at boot time (i.e. you are seeing the symptom above), you can unplug the cables from the HBA to get the system to boot up normally. Then, follow the above steps to change the driver load order and plug the cables back in during the next reboot. o LVM2 with SDD LVM2 refers to a new userspace toolset that provides logical volume management facilities on Linux. For more information about LVM2, see the LVM2 website at http://sources.redhat.com/lvm2/ or run "man lvm". In order for LVM2 to work on your machine, you must have the LVM2 rpm installed; look on the OS install CD's for the LVM2 package. In order to get LVM2 to run with SDD, several changes must be made to the LVM configuration file /etc/lvm/lvm.conf. In this file, there are several fields that must be modified. As always, back up the file before attempting to modify it. Under the section "devices", there are two values that must be modified. The first value is "filter". In the file, you might see the default as the following: # By default we accept every block device except udev names: filter = [ "r|/dev/.*/by-path/.*|", "r|/dev/.*/by-id/.*|", "a/.*/" ] This filter is too broad for SDD, because it will recognize both SDD vpath devices as well as the underlying paths (i.e. /dev/sdxxx) to that SDD vpath device. We want to narrow this regular expression to just accept vpath devices, and not the underlying scsi disk devices. The easiest way to do this is to modify the regular expression to just accept the name "vpath", and to ignore all other types of devices. Of course, this is the simplest example; your environment might be slightly different, but you can adjust the example accordingly: filter = [ "a/vpath[a-z]*/", "r/.*/" ] This regular expression will accept all vpath devices and reject all other devices under /dev. Another value that you need to change is "types". In the file, you will see that it is commented out: # List of pairs of additional acceptable block device types found # in /proc/devices with maximum (non-zero) number of partitions. # types = [ "fd", 16 ] You want to delete the comment marker, and replace "fd" with "vpath". This allows LVM to add vpath to its list of internally recognized devices. The partition number should stay at 16. For example: types = [ "vpath", 16 ] After making these two changes, save the lvm.conf file. Now, you should be able to run "pvcreate" on vpath devices (i.e. /dev/vpatha) and create volume groups using "vgcreate". o Timeouts with Qlogic qla2x00 8.00.00 Storage Adapter The Qlogic 8.00.00 driver enforces an I/O queue depth limit per path, not per LUN. This value is controlled through Qlogic's ql2xmaxqdepth parameter. Since the Qlogic driver does not enforce a queue depth limit per LUN, multiplying the number of paths to a LUN will also multiply the maximum queue depth per LUN. For example, using SDD with 4 paths to a LUN and a queue depth of 32 will allow up to 32 x 4 = 128 I/O requests to be queued to the LUN at an any instant in time. Thus, using SDD with the Qlogic driver can significantly multiply the I/O load to a LUN versus what is normally generated using a single-pathed solution. In some scenarios, heavy I/O load may cause many I/O requests to timeout because the ESS storage is overloaded with I/O requests and is taking longer to respond. Typically, this is indicated through a series of SCSI errors with a 0x20000 return code. For example, you may see a series of error messages for different paths in /var/log/messages that resemble: kernel: SCSI error : <7 0 0 4> return code = 0x20000 In addition, having saturated queues is another clue that indicates your storage is overloaded. The queues are saturated if the "Pending reqs" queue depth values listed in the "SCSI LUN Information" section of /proc/scsi/qla2xxx/[port_number] are close to or equal to the queue depth limit. For example, a part of an entry for a full path queue may look like this, assuming a queue depth limit of 32 is being used: SCSI LUN Information: (Id:Lun) * - indicates lun is not registered with the OS. ( 0: 4): Total reqs 1817, Pending reqs 32, flags 0x0, 0:0:84 00 ( 0: 5): Total reqs 2555, Pending reqs 32, flags 0x0, 0:0:81 00 ( 0: 6): Total reqs 3003, Pending reqs 32, flags 0x0, 0:0:81 00 ( 0: 7): Total reqs 1971, Pending reqs 32, flags 0x0, 0:0:81 00 Consequently, to avoid overloading a LUN, you can manually enforce a queue depth limit per LUN by adjusting the Qlogic driver's queue depth limit, i.e. the ql2xmaxqdepth parameter. This should be set to the desired per LUN queue depth divided by the number of paths to the LUN. For example, using SDD with 4 paths to the LUN and the Qlogic ql2xmaxqdepth value of 32 as the desired value, the new Qlogic queue depth limit would be set to 32 / 4 = 8. Note, that the appropriate queue depth is not only determined by the number of paths to each LUN, but the number of LUNS per host and the number of hosts connected to the storage should also be considered. You can adjust the queue depth limit by reloading the Qlogic qla2xxx driver with the ql2xmaxqdepth parameter specified. One method is to specify ql2xmaxqdepth at the command line when loading the qla2xxx driver: modprobe qla2xxx ql2xmaxqdepth=[new_queue_depth] The other method is to add the line: options qla2xxx ql2xmaxqdepth=[new_queue_depth] in /etc/modprobe.conf, before reloading the driver with modprobe. =============================================================================== 3.5 Correction to User's Guide o Update of SAN/Remote Boot Procedures for SLES 9 on x86 This document contains the necessary steps to configure SDD for remote boot on SLES 9. This document only lists the differences between what is in the SDD User's Guide (remote boot for 2.4 kernels) and what is necessary for remote boot to work on SLES 9 (remote boot for 2.6 kernels). Note that both SLES 9 ppc64 and RHEL 4 do not use the initrd system, so these notes do not apply to that particular 2.6 kernel based distribution. This document is based on the SDD User's Guide for Linux SDD version 1.6.x.x. Read section 1.0 "About this README file" for the location of the SDD support website and download the latest User's Guide. See the Linux chapter (Chapter 5) under "Booting Linux over the SAN with SDD" section for the original remote boot procedure (p 191-196). This document will reference the original. This document assumes that you have already set up your single path remote boot configuration. This document also assumes that you used a local disk to set up the base system, copied it over to a remote disk, and booted from the remote disk correctly in a single path configuration. Please ensure that your HBA driver is being loaded correctly (lpfcdd or qla2300). To check which modules are being loaded in your initrd, you need to check the file /etc/sysconfig/kernel. If the module is not in there, you will have to add it and run mkinitrd to make a new initial ramdisk before you can run even single path remote boot. The changes below are broken down by steps under the "Booting Linux..." section. Steps 1-4: No change Step 5: Note that I used 50331648 and that was plenty for me. You will need to probably increase this, though. Instead of running a df on /dev/loop0 you want to mount the device under /mnt or /mnt/temp and then run df /mnt to see the amount of space on the filesystem. Steps 6 - 16: No change, but ignore 15 (RHEL). Step 17: Ignore the first two bullets. In the linuxrc script, skipt down to the part where it starts: "echo 'Loading kernel/drivers/...'". This is where the rcscript starts loading modules. At the end of this section, before the "udev_discover_root()" function, add the following lines: echo "Loading sdd-mod.ko" insmod /lib/sdd-mod.ko echo "Running vpath configuration script" /opt/IBMsdd/bin/cfgvpath echo "Mounting and copying some files" /bin/mount -o rw -t ext3 /dev/vpatha3 /sysroot tar cps /dev/IBMsdd /dev/vpath* | (cd /sysroot && tar xps) cp /etc/vpath.conf /sysroot/etc /bin/umount /sysroot Be sure to replace /dev/vpatha3 with whatver vpath root device you are using. Also ignore the section about the "real_root_dev" - this part isn't applicable to SLES 9. Step 18: Edit /etc/fstab and be sure the entries are updated to vpath instead of sd. Also,make sure that the same fstab is put in BOTH the initrd (/mnt/etc) and on the remote disk (/etc), because they need to be consistent! Step 19-21: No change. Step 22: Make sure you edit /boot/grub/menu.lst and make a new entry in the bootloader menu for this new initrd file. MAKE SURE to update the root=/dev/xxx boot option because this is where the initrd searches to determine which device is the root device. ------------------------------------------------------------------------------- 4.0 User License Agreement for IBM Device Drivers See LICENSE file located in /opt/IBMsdd. ------------------------------------------------------------------------------- 5.0 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services,or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: IBM World Trade Asia Corporation Licensing 2-31 Roppongi 3-chome, Minato-ku Tokyo 106, Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND,EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM Corporation Information Enabling Requests Dept. DZWA 5600 Cottle Road San Jose, CA 95193 U.S.A. Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM License Agreement for Non-Warranted Programs. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. =============================================================================== IBM agreement for licensed internal code +---- Read Before Using -----------------------------------------------+ |IMPORTANT | | | |YOU ACCEPT THE TERMS OF THIS IBM LICENSE AGREEMENT FOR MACHINE CODE BY| |YOUR USE OF THE HARDWARE PRODUCT OR MACHINE CODE. PLEASE READ THE | |AGREEMENT CONTAINED IN THIS BOOK BEFORE USING THE HARDWARE PRODUCT.SEE| |IBM agreement for licensed internal code. | | | +----------------------------------------------------------------------+ You accept the terms of this Agreement(3) by your initial use of a machine that contains IBM Licensed Internal Code (called "Code"). These terms apply to Code used by certain machines IBM or your reseller specifies (called "Specific Machines"). International Business Machines Corporation or one of its subsidiaries ("IBM") owns copyrights in Code or has the right to license Code. IBM or a third party owns all copies of Code, including all copies made from them. If you are the rightful possessor of a Specific Machine, IBM grants you a license to use the Code (or any replacement IBM provides) on, or in conjunction with, only the Specific Machine for which the Code is provided. IBM licenses the Code to only one rightful possessor at a time. Under each license, IBM authorizes you to do only the following: 1. execute the Code to enable the Specific Machine to function according to its Official Published Specifications (called "Specifications"); 2. make a backup or archival copy of the Code (unless IBM makes one available for your use), provided you reproduce the copyright notice and any other legend of ownership on the copy. You may use the copy only to replace the original, when necessary; and 3. execute and display the Code as necessary to maintain the Specific Machine. You agree to acquire any replacement for, or additional copy of, Code directly from IBM in accordance with IBM's standard policies and practices. You also agree to use that Code under these terms. You may transfer possession of the Code to another party only with the transfer of the Specific Machine. If you do so, you must 1) destroy all your copies of the Code that were not provided by IBM, 2) either give the other party all your IBM-provided copies of the Code or destroy them, and 3) notify the other party of these terms. IBM licenses the other party when it accepts these terms. These terms apply to all Code you acquire from any source. Your license terminates when you no longer rightfully possess the Specific Machine. Actions you must not take You agree to use the Code only as authorized above. You must not do, for example, any of the following: 1. Otherwise copy, display, transfer, adapt, modify, or distribute the Code (electronically or otherwise), except as IBM may authorize in the Specific Machine's Specifications or in writing to you; 2. Reverse assemble, reverse compile, or otherwise translate the Code unless expressly permitted by applicable law without the possibility of contractual waiver; 3. Sublicense or assign the license for the Code; or 4. Lease the Code or any copy of it. ------------------------------------------------------------------------------- 6.0 Trademarks and service marks The following terms are trademarks of the International Business Machines Corporation in the United States,other countries, or both: AIX AS/400 Enterprise Storage Server HACMP/6000 IBM IBM logo iSeries Netfinity NetVista Operating System/400 pSeries RS/6000 Seascape SP System/360 System/370 System/390 The eServer logo TotalStorage Versatile Storage Server xSeries zSeries z/Architecture z/OS Microsoft, Windows, Windows NT, and the Windows logo are registered trademarks of Microsoft Corporation. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, and service names may be trademarks or service marks of others. ------------------------------------------------------------------------------- (C) Copyright IBM Corporation 2000, 2002, 2003, 2004, 2005. All rights reserved.