This document provides step-by-step instructions for enabling IBM Java for AIX "Class exceptions" related logging options, collecting the required diagnostic files, then uploading the data to the IBM testcase server.

Collecting and uploading this information at the time the IBM support call is opened will help expedite the resolution of the issue being reported.
Jump to section: (Validate Issue) (Prepare Environment) (Configure Parameters) (Collect Data) (Confirm Data) (Package Data) (Upload Data)
The instructions in this document make references to generic terms in Italics that will need to be replaced with information specific to the support call and the environment. It is very important that consistent and accurate values be used in place of the Italicized generic terms when collecting the data to ensure the prompt and correct delivery of the data when uploaded.
Generic Term
Replace with
USERID
The AIX userid running the Java process (e.g. wasadmin or root).
TMP_PATH
A temporary directory with a minimum of 10 GB of free space (e.g. /large_fs).
MM-DD
The current month and day (e.g. ,01-31).
PMR
The full IBM PMR number (e.g. , PMR12345.b678.c000).
JAVA_PATH
The parent Java installation directory (e.g. /usr/java6 or /usr/java7_64).
JAVA_PID
The process id of the Java process experiencing issues (e.g. use "ps" command to check the PID column to identify the process).
START_PATH
The directory from which the Java process was started (e.g. /usr/IBM/WebSphere).
SPECIFIC_PATH
User specified Directory. (e.g., /tmp)
CLASSNAME
Class for which exception is thrown. (e.g., for java.lang.NoClassDefFoundError: org.apache.log4j.BasicConfigurator -- CLASSNAME=*BasicConfigurator)
EXCEPTION
Exception thrown. (e.g., for "java.lang.NoClassDefFoundError" -- EXCEPTION=java/lang/NoClassDefFoundError)
JAR-FILE
Complete path to the jar file. (e.g., for "java -jar /home/fred/bin/PMRsdi.jar" -- JAR-FILE=/home/fred/bin/PMRsdi.jar)
Overview
Step-by-Step Instructions
Examples / Tips / Hints / Comments / Descriptions

Step 1:

Validate

Exceptions similar to ones listed below will be written to standard output (stdout), standard error (stderr), or to an application log file, depending on the application's configuration:

.......

BundleLoader[com.ibm.ws.runtime.gateway_7.0.0].findLocalClass(bol) BundleClassLoader[/usr/WebSphere85/AppServer/plugins/com.ibm.ws.runtime. gateway.jar].findClass(bol) BundleClassLoader[com.ibm.ws.runtime.gateway_7.0.0].loadClass(bol) failed. java.lang.ClassNotFoundException: bol at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLo ader.java:506) at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.jav a:422) at
......... java.lang.ClassLoader.loadClass(ClassLoader.java:680) at com.ibm.ws.bootstrap.ExtClassLoader.loadClass(ExtClassLoader.java:134) at java.lang.ClassLoader.loadClass(ClassLoader.java:663) at
..........



java.lang.VerifyError: JVMVRFY012 stack shape inconsistent;
class=com/sun/xml/messaging/saaj/soap/SOAPDocumentImpl,
method=createDocumentFragment()Lorg/w3c/dom/DocumentFragment;, pc=5;
Type Mismatch, argument 0 in signature
com/sun/xml/messaging/saaj/soap/SOAPDocumentFragment.:
(Lcom/sun/org/apache/xerces/internal/dom/CoreDocumentImpl;)V does not
match



Nested Exception:
Exception in thread "main" java.lang.NoClassDefFoundError: .aix-java.TestTime Caused by: java.lang.ClassNotFoundException: .aix-java.TestTime at java.net.URLClassLoader.findClass(URLClassLoader.java:434) at java.lang.ClassLoader.loadClass(ClassLoader.java:672) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:358) at java.lang.ClassLoader.loadClass(ClassLoader.java:638) Could not find the main class: /aix-java/TestTime. Program will exit.

.....


java.lang.ClassCastException: java.util.ArrayList incompatible with [Ljp.co.daifuku.dats.xmcs.gui.value.MapPort; java.lang.ClassCastException: java.util.ArrayList incompatible with [Ljp.co.daifuku.dats.xmcs.gui.value.MapPort; at jp.co.daifuku.dats.xmcs.cache.MapPortCache.getArray(MapPortCache.java:16 8)


The above exception stacks are examples only. The exception stack traces and class names may be different depending on the application and situation.




Step 2:

Prepare

Before proceeding with the setup and data collection instructions to collect complete diagnostic data for analysis, click Step 7. Upload Data for instructions to upload any existing generated logs and data collected.

To prepare for these data collection procedures, the process environment needs to be configured to collect complete data and save the additional debug information to a log file.



A. Set the user ulimits:

From a command prompt and while logged in as the root user, execute following commands to:

{ set file and data ulimit sizes to unlimited }

# chuser fsize=-1 data=-1 USERID

B. Redirect or save standard error (stderr) messages to a file

Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.

For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".


C. Perform the following actions in order for the changes to take effect:

- Stop the application

- Relogin as the "USERID" used in Step 1.A

- Confirm the new ulimits are in effect by executing the commands:

# ulimit -a




- Do not restart the application until Step 3. Configure has been completed

A. Enabling debug options will result in additional data being stored in memory buffers and written to application logs. The process file and data sizes should be increased during the data collection to ensure the data is complete.

If there are multiple processes executed by multiple user ids experiencing the issue, then all preparation steps must be repeated for each id and process.

To confirm the process environment is configured correctly, login using the USERID specified in the steps, then run the command:

# ulimit -a

The values for "file" and "data" should show as:
file(blocks) unlimited
data(kbytes) unlimited

B. As an example, to save standard error message to the file /tmp/stderr.log, use a command line syntax similar to:

# java YOUR_APP 2>/tmp/stderr.log

to save both standard out and standard error messages to the file /tmp/out.log, specify:

# java YOUR_APP > /tmp/out.log 2>&1

To confirm the messages are being redirected to the log file, view the contents of the log file.

Step 3:

Configure

A. Add java command line options:

Enable classloader tracing, verbose logging of the class loading, debug output and thread stack:

-verbose:gc,class,dynload,stack,debug -Dibm.cl.verbose=CLASSNAME


Enable shared classes cache data collection:

-Xshareclasses:printAllStats,listAllCaches


Enable manual generation of javacores, heapdumps and snap traces:

-Xdump:java+heap+snap:events=user



Generate Javacore, Heapdump and Snap trace files when the exception is thrown:

-Xdump:java+heap+snap:events=systhrow+throw,filter=EXCEPTION,range=1..3



To designate a directory for the generated logs, instead of the default directory, add:

-Xdump:directory=SPECIFIC_PATH



B. Restart the java application (e.g., node agent/manager) from the USERID new login session.

A. Alternate ways of setting java command line options and non-default directory for generated log files:

Environment variable can be used to set Java command line options. However, options specified on java command line take precedence over environment variable settings:

# export IBM_JAVA_OPTIONS="-verbose:gc -Xverbosegclog:SPECIFIC_PATH/gc.log -Xdump:java+heap+snap:events=systhrow+throw,filter=java/lan/NoClassDefFoundError,range=1..3"



To direct javacore, heapdump and snap trace to specific directories instead of the default directory of START_PATH, set environment variables:

# export IBM_JAVACOREDIR=SPECIFIC_PATH
# export IBM_HEAPDUMPDIR=SPECIFIC_PATH



For exception in step1, replace EXCEPTION as below:

Exception in thread "main" java.lang.NoClassDefFoundError: .aix-java.TestTime Caused by: java.lang.ClassNotFoundException: .aix-java.TestTime at

specify:
-Xdump:java+heap+snap:events=systhrow+throw,filter=java/lang/ClassNotFoundException,range=1..3



B. For application server environments such as WebSphere and WebLogic, both the node manager/agent and the application servers must be restarted after the relogin of "USERID".

Step 4:

Collect

A. Execute the following commands to collect the required diagnostic data.

# mkdir -p /TMP_PATH/MM-DD/class-exceptions/data
# cd /TMP_PATH/MM-DD/class-exceptions/data


B. If the executable is a "jar" file invoked with "-jar" Java command line option, to check CLASSPATH, execute:

# cp JAR-FILE ./


Generated logs are written to the default directory of "START_PATH" or the user specified SPECIFIC_PATH.



C. Collect output of AIX commands from the same AIX LPAR:

# errpt -a > errpt-a.out 2>&1
# oslevel -s > oslevel-s.out 2>&1
# prtconf > prtconf.out 2>&1
# lsps -a > lsps-a.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# ps avwwwg > ps-avwwwg.out 2>&1



D. Copy the files generated to the data directory "/TMP_PATH/MM-DD/class-exceptions/data" directory created in step 4.A:

# cp /START_PATH/javacore*txt ./
# cp /START_PATH/heapdump* ./
# cp /START_PATH/Snap*trc ./

Also copy standard error, standard output, SystemOut, SystemErr, gc.log, application logs and any other logs generated to the /TMP_PATH/MM-DD/class-exceptions/data directory.

A. Examples of commands to be executed:

# mkdir -p /large_fs/01-31/class-exceptions/data

# cd /large_fs/01-31/class-exceptions/data



D. If "START_PATH" can not easily be determined, execute these commands to help determine the correct path:

# ps -ef | grep -i java

rt 3211380 1 0 May 31 - 1109:46 java -Dsrse_property=/rt/pmr/test/classloader/JvmTest-dir/j_test/etc/aib.ini....

# kill -3 3211380

# procwdx 3211380

3211380: /rt/pmr/test/

Check for the generated javacore in the the above directory:

# cd /rt/pmr/test

# ls -l *javacore*

-rw-r--r-- 1 rtstaff 234048 Aug 25 10:54 javacore.20150825.105402.3211380.0010.txt


Confirm all files and directories have been saved to the data directory:

# cp /var/myapp/*.log ./

# ls javacore* heapdump* Snap*

# ls *.log *.out *.txt System*

# ls *.jar gc.log

Step 5:

Confirm

** MANDATORY **

Prior to packaging and uploading the data, confirm that the following files have been saved in the "/TMP_PATH/MM-DD/class-exceptions/data directory:

a. Javacore files
b. Heapdump files
c. Snap traces
d. verbose GC log
e. standard error, standard output, SystemOut, SystemErr, application logs and any other logs generated.
f. AIX commands output

Step 6:

Package

After all of the files have been collected, from a command prompt, execute the following commands to package the requested files:

# cd /TMP_PATH/MM-DD/class-exceptions
# tar -cf - data | gzip -c > PMR.MM-DD.tgz

Examples of commands to be executed:

# cd /large_fs/01-31/class-exceptions
# tar -cf - data | gzip -c > 12345.678.000.01-31.tgz

Step 7:

Upload

Upload the packaged data to IBM secured servers using one of upload options provided on the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619


If this step is reached by clicking "Step 7. Upload Data" in Step 2, at the end of the completion of the existing latest logs/data upload, click Step 2. Prepare Environment to continue with the data collection instructions.

Step 7:

ACTION

Step 8:

ACTION

Step 9:

ACTION

Step 10:

ACTION

Step 11:

ACTION

Step 12:

ACTION

Step 13:

ACTION

Step 14:

ACTION

Step 15:

ACTION

Step 16:

ACTION

Step 17:

ACTION

Step 18:

ACTION

Step 19:

ACTION

Step 20:

ACTION

Step 21:

ACTION

Document Type:
Instruction
Content Type:
Mustgather
Hardware:
all Power
Operating System:
all AIX Versions
IBM Java:
all Java Versions
Author(s):
Rama Tenjarla
Reviewer(s):
Roger Leuckie
Click here to submit feedback for this document.