| Generic Term | Replace with |
| WORK_DIRECTORY | The parent directory to be used to store the data collected during the data collection process (e.g., /BIG_FS/ibm) |
| JAVA_PID | The process id of the active Java process (e.g. use "ps" command to check the PID column to identify the process). |
| YOUR_PMR | The full IBM PMR number (e.g. ,12345.678.000). |
Users must complete all of these preparation steps to successfully complete the data collection and upload the required data.
A. Download and install the AIX "pdump.sh" script:
ftp://ftp.software.ibm.com/aix/tools/debug/pdump.sh
Copy the downloaded file to the AIX system using the filename:
/opt/javatools/scripts/pdump.sh
From a command prompt, and while logged in as the root user, execute the command:
# chmod 755 /opt/javatools/scripts/pdump.sh
B. Download and install the AIX "trace_kill9.sh" script:
ftp://ftp.software.ibm.com/aix/tools/java/tools/trace_kill9.sh
Copy the downloaded file to the AIX system using the filename:
/opt/javatools/scripts/trace_kill9.sh
From a command prompt, and while logged in as the root user, execute the command:
# chmod 755 /opt/javatools/scripts/trace_kill9.sh
C. Read and agree to the license before using the "trace_kill9.sh" script:
From a command prompt, and while logged in as the root user, execute the command:
/opt/javatools/scripts/trace_kill9.sh -l
D. Identify a directory (file system) with a minumum of 3 GB of free space:
From a command prompt, and while logged in as the root user, execute the commands:
# df -km | tail +3 | awk '( $3 > 3072) {print $NF" "$3" (fs= "$1")"}'
If no directory (file system) is listed, then space needs to be added to an existing directory or a new (temporary) file system should be created.
If one or more directories (file systems) are listed, choose the one of them to used for the "WORK_DIRECTORY" referenced in the instructions that follow.
E. Understand the data that will be collected:
- pdump.sh: Runs kdb command to collect kernel level information about the process. This is executed twice (before and after).
- ps command: The ps command is executed before and after to confirm the process is still listed in the process table.
- kill command: The kill -9 command will be executed for the hung process
- kernel trace: A brief kernel trace will be collected while the kill -9 command is executed to identify any events preventing the "kill -9" signal processing.
- post processing commands: Additional commands such as gennames, trcnm, trcfmt, etc which are required for post processing of kernel trace
The data collection procedures must be performed while the problem can be reproduced and the process is listed in the process table. To confirm the process is still active, from a command prompt, and while logged in as the root user, run the commands:
# ps -ef | grep JAVA_PID
If the process is no longer listed (the system may have been restarted), the situation must be reproduced prior to performing any data collection procedures. Collecting and uploading data while the situation is not occurring will add no value toward the understanding of the root cause for the situation (the data collection will be requested until it is accurately and successfully collected and uploaded).
A. Reivew and understand the script usage and requirements by executing the following command while logged in as the root user:
# /opt/javatools/scripts/trace_kill9.sh -h
B. With the issue occurring, from a command prompt, and while logged in as the root user, start the "trace_kill9.sh" data collection script. A separate window may be required to run these steps.
Replace the JAVA_PID, WORK_DIRECTORY, and YOUR_PMR parameters with the correct and appropriate values for the situation.
# cd WORK_DIRECOTRY
# /opt/javatools/scripts/trace_kill9.sh -j JAVA_PID -p YOUR_PMR
(e.g., cd /BIG_FS/ibm; /opt/javatools/scripts/trace_kill9.sh -j 123456 -p 12345,123,455)
When prompted, type "Y"(uppercase Y) to confirm the information shown and to continue with the data collection. If "Y" is not specified, the data collection script will halt. Examples of prompts that will be shown are:
{Confirm that Java process id specified on the command line}
[ INFO ] JAVA_PID= 123456
[ INFO ] PROCESS_NAME= /usr/java7/bin/java
[ CONFIRM ] Press 'Y' (uppercase) key, then the 'ENTER' key to continue [Y]
{Confirm to reuse the existing data collection directory - THIS PROMPT IS NOT ALWAYS PRESENTED}
[ CONFIRM ] The temporary work directory (/data/samples/traces/tmp/PMR-12345.1123.455/11-12/data) already exists.
[ CONFIRM ] Press 'Y' (uppercase) key, then the 'ENTER' key to continue to use this directory (files may be overwritten) [Y]
{Confirm the start of the data collection procedures}
[ CONFIRM ] Press 'Y' (uppercase) key, then the 'ENTER' key if you are ready to start the data collection procedures [Y]
Once the data has been successfully collected, it will be pre-packaged, and upload instructions will be provided:
{Follow the instructions to upload the data to your PMR}
[ INFO ] Please upload the file '/data/samples/traces/tmp/PMR-12345.123.455/12345.123.455.11-12.1447358196.tgz' to your IBM PMR.
[ INFO ] Visit the web page: http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619
[ INFO ] for complete upload instructions
Follow the upload instructions provided or continue to the next step for instructions to uploaded the packaged data collection file.
A. Upload the packaged data to an IBM secured server using one of upload options provided on the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:
http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619
| Document Type: | Instruction |
| Content Type: | Mustgather |
| Hardware: | all Power |
| Operating System: | all AIX Versions |
| IBM Java: | all Java Versions |
| Author(s): | Roger Leuckie |
| Reviewer(s): | Rama Tenjarla |