		Open Fabrics Enterprise Distribution (OFED)
		      Tips for Working with OFED 1.2

		 	     June 2007

===============================================================================
Table of Contents
===============================================================================
1. OFED Utilities
2. Debug HOWTOs
3. Pinning (Locking) User Memory Pages
4. External Module Compilation Over OFED-1.2
5. Adding/Deleting a patch to OFED package


===============================================================================
1. OFED Utilities
===============================================================================

The OFED package includes utilities under <prefix>/bin, where <prefix> stands
for the OFED installation path. To retrieve this path, run the script 
"/etc/infiniband/info" as explained in Section 2.2 below.

Notes:
------
1. This document includes descriptions for a subset of the existing utilities.
   To learn about other utilities, use their --help flag.

2. The sources for all utilities are not part of the RPM installation. However,
   all sources exist in the openib-1.2.tgz tarball.


1.1 Device Information
----------------------
Device information can be obtained using several utilities:

a. ibv_devinfo

   ibv_devinfo prints the ca attributes.

   usage:
     ibv_devinfo

     Options:
       -d, --ib-dev=<dev>    use IB device <dev> (default: first device found)
       -i, --ib-port=<port>  use port <port> of IB device (default: all ports)
       -l, --list            print only the IB devices names
       -v, --verbose         print all the attributes of the IB device(s)

b. ibstat

	usage:
	ibstat [OPTIONS] <ca_name> [portnum]

	Options:
	-d	debug
	-l	list all IB devices
	-s	print short device summary
	-p	print port GUIDs
	-V	print ibstat version information and exit
	-h	print usage

	Examples:
	ibstat -l 		# list all IB devices
	ibstat mthca0 2 	# stat port 2 of mthca0

c. Using sysfs file system
	The driver supports the sysfs file system under: /sys/class/infiniband

	Examples:

	> ls /sys/class/infiniband/mthca0/
	board_id device fw_ver hca_type hw_rev node_desc node_guid node_type
	ports sys_image_guid

	> cat /sys/class/infiniband/mthca0/board_id
	MT_0200000001

	> ls /sys/class/infiniband/mthca0/ports/1/
	cap_mask counters gids lid lid_mask_count phys_state pkeys rate sm_lid
	sm_sl state

	> cat /sys/class/infiniband/mthca0/ports/1/state
	4: ACTIVE

1.2 Performance Tests
---------------------
	The following performance tests are provided with the OFED release:

	1. Latency tests:
	   - ib_read_lat: RDMA read
	   - ib_write_lat: RDMA write
	   - ib_send_lat: UD, UC and RC (default) send

	2. Bandwidth tests:
	   - ib_read_bw: RDMA read
	   - ib_write_bw: RDMA write
	   - ib_send_bw: UD, UC and RC (default) send

	Usage:
	Server: <test name> <options>
	Client: <test name> <options> <server IP address>
		<server IP address> is an Ethernet or IPoIB address.
	--help	  lists the available <options>. The same options must be
		  passed to both server and client.	
	
	Note: See PERF_TEST_README.txt for more information on the performance
	      tests.
	
	Example: ib_send_bw
	Usage:
	ib_send_bw 			start a server and wait for connection
	ib_send_bw <host>		connect to server at <host>

	options:
	-p, --port=<port> 		listen on/connect to port <port>
						(default: 18515)
	-d, --ib-dev=<dev> 		use IB device <dev>
						(default: first device found)
	-i, --ib-port=<port> 		use port <port> of IB device
						(default: 1)
	-c, --connection=<RC/UC/UD> 	connection type RC/UC/UD (default: RC)
	-m, --mtu=<mtu> 		mtu size (default: 1024)
	-s, --size=<size> 		size of message to exchange
						(default: 65536)
	-a, --all 			run sizes from 2 up to 2^23
	-t, --tx-depth=<dep> 		size of tx queue (default: 300)
	-n, --iters=<iters> 		number of exchanges
						(at least 2, default: 1000)
	-b, --bidirectional 		measure bidirectional bandwidth
						(default: unidirectional)
	-V, --version display version number

1.3 Ping-pong Example Tests
---------------------------
	The ping-pong example tests provide basic connectivity tests. Each test
	has a help message (-h).
	- ibv_ud_pingpong
	- ibv_rc_pingpong
	- ibv_srq_pingpong
	- ibv_uc_pingpong

	Example: ibv_ud_pingpong --h
	Usage:
	ibv_ud_pingpong  	start a server and wait for connection
	ibv_ud_pingpong <host>	connect to server at <host>

	options:
	-p, --port=<port> 	listen on/connect to port <port>
					(default: 18515)
	-d, --ib-dev=<dev> 	use IB device <dev>
					(default: first device found)
	-i, --ib-port=<port> 	use port <port> of IB device (default: 1)
	-s, --size=<size> 	size of message to exchange (default: 2048)
	-r, --rx-depth=<dep> 	number of receives to post at a time
					(default: 500)
	-n, --iters=<iters> 	number of exchanges (default: 1000)
	-e, --events 		sleep on CQ events (default: poll)


===============================================================================
2. Debug HOWTOs
===============================================================================

2.1 OFED Components and Version Information
-------------------------------------------
The text file BUILD_ID provides data on all OFED components (whether installed
or not). This file is a part of the ofed-docs RPM and installed under 
/usr/share/doc/ofed-docs-1.2 on RedHat, and under 
/usr/share/doc/packages/ofed-docs-1.2 on SuSE.

The same information can be obtained by executing the 'ofed_info' command. For
example:

> ofed_info
OFED-1.2-20070429-0610

ofa_kernel-1.2:
Git:
git://git.openfabrics.org/~vlad/ofed_1_2/.git
commit f5829ec5964166081a0004753e9995b5641c168d

ofa_user-1.2:
libibverbs:
git://git.openfabrics.org/ofed_1_2/libibverbs.git ofed_1_2
commit 300810860e4c318b6db71331f62838fda39c7dad
libmthca:
git://git.openfabrics.org/ofed_1_2/libmthca.git ofed_1_2
commit 1539526f180e5b13fbf937e306025ea860b5d5fd
libehca:
git://git.openfabrics.org/ofed_1_2/libehca.git ofed_1_2
commit d4b3dd6c078a85a8e67a480161d1d9138ae66029
libipathverbs:
git://git.openfabrics.org/ofed_1_2/libipathverbs.git ofed_1_2
commit 15f62c3f045295dd2a941ae8d4e0e36035aad5cf
tvflash:
git://git.openfabrics.org/ofed_1_2/tvflash.git ofed_1_2
commit 5049e070afeebedf7916bb04a3363e9af38be6df
libibcm:
git://git.openfabrics.org/ofed_1_2/libibcm.git ofed_1_2
commit 8154d4d57f69789be6d26fdc8f10b552c83a87ec
libsdp:
git://git.openfabrics.org/ofed_1_2/libsdp.git ofed_1_2
commit 77c35bd3258aa2f20878f478b5a467a7fa6f84f4
mstflint:
git://git.openfabrics.org/ofed_1_2/mstflint.git ofed_1_2
commit 896f66fc85825e9cfe24d0583dc69287abc3e535
perftest:
git://git.openfabrics.org/ofed_1_2/perftest.git ofed_1_2
commit 16cf32e9ff226bc92d58844af53b7440cb1c65dd
srptools:
git://git.openfabrics.org/ofed_1_2/srptools.git ofed_1_2
commit 7c84a3ea92eb4e1062b9fce6e0c9d0f9287b879c
ipoibtools:
git://git.openfabrics.org/ofed_1_2/ipoibtools.git ofed_1_2
commit e29da6049cb725b175423fddc80181980ebfa0b4
librdmacm:
git://git.openfabrics.org/ofed_1_2/librdmacm.git ofed_1_2
commit 9450ee3953c5765d276f1310bdd045d0d3b62082
dapl:
git://git.openfabrics.org/ofed_1_2/dapl.git ofed_1_2
commit 04da88bb70ee33b249a4cf2f5a92122eeec83e3c
imgen:
git://git.openfabrics.org/ofed_1_2/imgen.git ofed_1_2
commit 2b5da49987426a3942fa23fe5bc693e16e370643
management:
git://git.openfabrics.org/ofed_1_2/management.git ofed_1_2
commit 9d44348a0c26c9a1c62585f10c1b7bdc3102a4ea
libcxgb3:
git://git.openfabrics.org/ofed_1_2/libcxgb3.git ofed_1_2
commit e889105a95381ae41c0c83716ad8097ed25c8aae
qlvnictools:
git://git.openfabrics.org/ofed_1_2/qlvnictools.git ofed_1_2
commit 1639f124eee359db87467e1fa124aff39737621c
sdpnetstat:
git://git.openfabrics.org/ofed_1_2/sdpnetstat.git ofed_1_2
commit 3329358b7ded50a2b51b436eebb5c9aeeb5e4e5e

# MPI
mvapich-0.9.9-1207.src.rpm
mvapich2-0.9.8-11.src.rpm
openmpi-1.2.1-1.src.rpm
mpitests-2.0-705.src.rpm

2.2 Installed OFED Components
-------------------------------
The script /etc/infiniband/info provides data on the specific OFED installation
on the machine.

For example:

 > /etc/infiniband/info
   prefix=/usr
   Kernel=2.6.9-22.ELsmp

   MODULES:  CONFIG_INFINIBAND=m CONFIG_INFINIBAND_USER_MAD=m
   CONFIG_INFINIBAND_USER_ACCESS=m CONFIG_INFINIBAND_ADDR_TRANS=y
   CONFIG_INFINIBAND_MTHCA=m CONFIG_IPATH_CORE=m CONFIG_INFINIBAND_IPATH=m
   CONFIG_INFINIBAND_IPOIB=m

   User level:  --kernel-version 2.6.9-22.ELsmp --kernel-sources
   /lib/modules/2.6.9-22.ELsmp/build --with-libibcm --with-libibverbs
   --with-libipathverbs --with-libmthca --with-mstflint --with-perftest

2.3 Building/Installing InfiniBand (IB) Modules With Debug Information
----------------------------------------------------------------------
To compile/build/install the IB modules so that they will contain debug
information, set OPENIB_KERNEL_EXTRA_CFLAGS="-g" in your environment
before running OFED's install.sh/build.sh .


===============================================================================
3. Pinning (Locking) User Memory Pages
===============================================================================

Memory locking is managed by the kernel on a per user basis. Regular users (as
opposed to root) have a limited number of pages which they may pin, where
the limit is pre-set by the administrator. Registering memory for IB verbs
requires pinning memory, thus an application cannot register more memory than
it is allowed to pin.

The user can change the system per-process memory lock limit by adding
the following two lines to the file /etc/security/limits.conf:

  *  soft  memlock  <number>
  *  hard  memlock  <number>

  where <number> denotes the number of KBytes that may be locked by a
  user process.

The above change to /etc/security/limits.conf will allow any user process in the
system to lock up to <number> KBytes of memory.

On some systems, it may be possible to use "unlimited" for the size to disable
these limits entirely.
	      
Note: The file /etc/security/limits.conf contains further documentation.


===============================================================================
4. External Module Compilation Over OFED-1.2
===============================================================================

To build kernel modules depending on OFED's modules, take the Modules.symvers
file from <prefix>/src/openib/Module.symvers (part of the kernel-ib-devel RPM)
and copy it to the modules subdir and then compile your module.

If <prefix>/src/openib/Module.symvers does not exist or it is empty, use the
create_Module.symvers.sh (a part of the ofed-docs RPM) script to create the
Module.symvers file.

See "Module versioning & Module.symvers" in the modules.txt from kernel
documentation (e.g. linux-2.6.20/Documentation/kbuild/modules.txt).


===============================================================================
5. Adding/Deleting a patch to OFED package
===============================================================================
If there is a need to add or delete a patch to OFED package one should use the 
ofed_patch.sh script, that is available under the docs directory.

Usage:
Add patch to OFED:
ofed_patch.sh  --add
               --ofed|-o <path_to_ofed>
               --patch|-p <path_to_patch>
               --type|-t <user|kernel|backport <kernel tag>|addons <kernel tag>>

Remove patch from OFED:
ofed_patch.sh  --remove
               --ofed|-o <path_to_ofed>
               --patch|-p <patch name>
               --type|-t <user|kernel|backport <kernel tag>|addons <kernel tag>>

Examples:
ofed_patch.sh --add --ofed /tmp/OFED-1.2/ --patch /tmp/cma_fix.patch --type kernel

ofed_patch.sh --remove --ofed /tmp/OFED-1.2/ --patch cma_fix.patch --type kernel

