IBM Java for AIX MustGather: Data collection procedure for hung Java applications

This document provides step-by-step instructions for collecing data for the IBM Java for AIX for hung Java processes, collecting other required diagnostic files, and instructions for uploading the data to the IBM testcase server.

Collecting and uploading this information at the time the IBM support call is opened will help expedite the resolution of the issue being reported.

The instructions in this document make references to generic terms in Italics that will need to be replaced with information specific to the support call and the environment. It is very important that consistent and accurate values be used in place of the Italicized generic terms when collecting the data to ensure the prompt and correct delivery of the data when uploaded.

Overview

Step-by-Step Instructions

Examples / Tips / Hints / Comments / Descriptions

Step 1:

Prepare

To prepare for these data collection procedures, the pdump.sh script and the perfpmr tool must be downloaded and installed on the AIX system. The process environment also must be configured to enable the creation of complete AIX (process) core dumps.

A. Download the "pdump" shell script

The pdump shell script can be downloaded from the FTP site:

ftp://ftp.software.ibm.com/aix/tools/debug/pdump.sh

Copy the downloaded file to the AIx system using the filename:

/opt/pdump/pdump.sh

From a command prompt, and while logged in as the root user, execute the command:

# chmod 755 /opt/pdump/pdump.sh

B. Download and install "perfpmr" tool

The perfpmr tool can be downloaded from the FTP site:

ftp://ftp.software.ibm.com/aix/tools/perftools/perfpmr/

From the above location, select the directory for the AIX operating system release being used, then select and download the perf##.tar.Z (e.g., perf71.tar.Z) file. Copy the downloaded file to the AIX system using the filename:

/tmp/perf##.tar.Z

From a command prompt, and while logged in as the root user, execute the following commands:

# mkdir -p /opt/perfpmr

# cd /opt/perfpmr

# uncompress -c < /tmp/perf##.tar.Z | tar -xvf -

# ./Install

C. Enable the system and the user environment for full core dumps

From a command prompt, while logged in as the root user, execute the following commands to enable full core dumps for the system and process environment.

# chdev -l sys0 -a fullcore=true

# chuser fsize=-1 data=-1 core=-1 USERID

D. Redirect or save standard error (stderr) messages to a file

Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.

For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".

E. Relogin, then restart your application

Perform the following actions inorder for the changes to take effect:

- Stop the application (and node agent/manager, if applicable)

- Relogin as the USERID used in Step 1.C

- Confirm that full core is enabled and the new ulimits are in effect by executing the commands:

# ulimit -a

# lsattr -El sys0 | grep -i fullcore

- Restart the application (e.g., node agent/manager) from the new login session

A. The "pdump.sh" shell script runs the AIX kernel debugger (kdb), along with some other commands, to obtain kernel level data about the process. For a process with several hundreds threads, it may take a several minutes for the script to complete the inspection of all the threads. Because pdump.sh scans active threads, do not run this script for high CPU situations unless instructed to do so.

B. The "perfpmr" data helps the support specialists understand and identify the hot spots within the different AIX components and subsystem (e.g., kernel, memory, networking, locking, and applications). The process collects both configuration and performance data for the specified duration.

C. When the full core dump options are not enabled and the core dumps are uploaded, in most cases, the core dumps will be incomplete or truncated. Not setting these options will prevent the support specialist from analyzing the data and will also delay the resolution of the reported issue.

When using J2E (or J2EE) application servers such as IBM WebSphere or Oracle WebLogic, for the changes to take effect, both the node agent (manager) and the application (manager) servers have to be stopped and restarted (and relogin before restarting)..

Examples of commands to be executed:

# chdev -l sys0 -a fullcore=true

# chuser fsize=-1 data=-1 core=-1 wasadmin

The AIX core file is generated in the current working directory of the process. Use the AIX environment variables:

IBM_COREDIR=NEW_PATH

to specify an alternate location for the AIX (process) core dump. Likewise, use the AIX environment variable:

IBM_JAVACOREDIR=NEW_PATH

to specify an alternate location for the javacore.*.txt files.

Both the IBM_COREDIR and IBM_JAVACOREDIR variables have to be configured for the process prior to the it being started (i.e., as part of its startup procedure and the process has to be restarted).

Step 2:

Collect

A. Immediately once the hung process can be reproduced, from a command prompt, and while logged in as the root user, execute the following commands to collect the required diagnostic data:

# mkdir -p /TMP_PATH/PMR/MM-DD-HH/data1

# cd /TMP_PATH/PMR/MM-DD-HH/data1

# /opt/pdump/pdump.sh JAVA_PID

# mkdir -p /TMP_PATH/PMR/MM-DD-HH/perfpmr

# cd /TMP_PATH/PMR#/MM-DD-HH/perfpmr

# perfpmr.sh 240

# cd /TMP_PATH/PMR#/MM-DD-HH/data1

# gencore JAVA_PID ../core.dmp

# kill -3 JAVA_PID

# sleep 5

# kill -3 JAVA_PID

# sleep 5

# kill -3 JAVA_PID

# prtconf > prtconf.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# instfix -i > instfix-i.out 2>&1
# emgr -lv3 > emgr-lv3.out 2>&1
# svmon -O segment=category -P > svmon-p.out 2>&1
# ipcs -saPrX > ipcs.out 2>&1
# errpt -a > errpt-a.out 2>&1
# JAVA_COMMAND -version > java-version.out 2>&1

{Copy the javacore.*.txt files to the data1 directory.}
{Copy the *.out files to the data1 directory.}
{Copy the application logs to the data1 directory.}
{Copy the stderr/stdout files to the data1 directory.}

# JAVA_PATH/jre/bin/jextract -J-Xms1024M -J-Xmx1024M \
../core.dmp core.java.dmp.zip

** In situations when the jextract command fails, execute the command:

# snapcore -d ${PWD} ../core.dmp JAVA_COMMAND

A. Examples of commands to be executed:

** Do not copy and paste AS-IS, these are only examples **

# mkdir -p /large_fs/12345.123.000/01-31-10/data1
# cd /large_fs/12345.123.000/01-31-10/data1

# /opt/pdump/pdump.sh 1095

# mkdir -p /large_fs/12345.123.000/01-31-10/perfpmr

# cd /large_fs/12345.123.000/01-31-10/perfpmr

{run perfpmr script for 240 seconds}

# perpmr.sh 240

# cd /large_fs/12345.123.000/01-31-10/data1

{gencore creates an AIX (process) core file for a running process}

# gencore 1095 ../core.dmp

{the kill command generates an Java thread dump}
{in most situations it DOES NOT kill the process}

# kill -3 1095
# sleep 5
# kill -3 1095
# sleep 5
# kill -3 1095

# cp /opt/myapp/javacore*txt ./
# cp /opt/myapp/*.out ./
# cp /opt/myapp/myapp*logs ./

# /usr/java7/jre/bin/jextract -J-Xms1024M -J-Xmx1024M \
../core.dmp core.java.dmp.zip

# snapcore ../core.dmp /usr/java7/bin/java

The jextract utility differs from snapcore by collecting platform-specific details about the application such as word size, endianness, data structure layouts, and symbolic information. This information is captured in an XML file which is packaged with the core dump file.

Step 3:

Confirm

A. Prior to packaging and uploading, confirm that the following files have been saved in the temporary directory:

1. core.java.dmp.zip or snapcore file (mandatory)

2. (three) javacore.*.txt files (mandatory)

3. Application log files (mandatory)

4. All of the *.out files generated in Step 2 (mandatory)

5. perfpmr data (mandatory)

6. pdump data (mandatory)

A. Sending incomplete data or data other than the requested data may delay the resolution of the reported issue.

Examples of commands to execute:
** Do not copy and paste AS-IS, these are only examples **

# cd /large_fs/12345.123.000/01-31-10/data1

# ls core*zip snapcore* javacore**.out

# ls *.logs

# ls *.out

# ls pdump*

# cd /large_fs/12345.123.000/01-31-10/perfpmr

# ls config.sum monitor.int

Step 4:

Package

A. Packaging the files may simplify the upload of the diagnostic data collected. From the command line and while logged in as the root user, execute the commands:

If you do not correctly replace the Generic Terms in the file name with the actual PMR # and date, the will be a delay in both the review of the data and the resolution of this issue.

# cd /TMP_PATH/PMR/MM-DD-HH

# tar -cf - data1 perfpmr | gzip -c > PMR.MM-DD-HH.tgz

A> Examples of commands to execute:

** Do not copy and paste AS-IS, these are only examples **

# cd /large_fs/12345.123.000/01-31-10

# tar -cf - data1 perfpmr | gzip -c > 12345.123.000.01-31-10.tgz

Step 5:

Upload

A. Upload the packaged data to IBM secured servers using one of upload options provided on the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619

Step 6:

Upload

Step 7:

ACTION

Step 8:

ACTION

Step 9:

ACTION

Step 10:

ACTION

Step 11:

ACTION

Step 12:

ACTION

Step 13:

ACTION

Step 14:

ACTION

Step 15:

ACTION

Step 16:

ACTION

Step 17:

ACTION

Step 18:

ACTION

Step 19:

ACTION

Step 20:

ACTION

Step 21:

ACTION