InfoCenter Home >
8: Problem determination >
8.9: Thread dumps

8.9: Thread dumps

This section introduces the concept of thread dumps in WebSphere Application Server.

What is a thread dump?

A thread represents a work item or task, such as a servlet request. Java processes are usually multi-threaded. This means there can be many tasks occurring simultaneously (that is, multi-threading) within one JVM (Java Virtual Machine) process. Therefore, understanding what is occurring within a JVM process means obtaining information about all the different threads that are defined within the process.

There are two types of thread dumps that could appear when running Java programs:

System thread dumps

System thread dumps provide a system view of a failing JVM (Java Virtual Machine) process. On Unix systems, they usually appear as core files. On Window's systems they appear as drwtsn32.log files.

System dumps do not understand Java classes. Everything in a system dump is C library oriented. The system dump information provided for JVM processes refers to Java's C libraries and not the reference class files.

System dumps should only be interrogated when a Java thread dump is unavailable. Pertinent information can be obtained from system dumps. However, mapping this information back into Java source code is very difficult. The following sections explain how to interrogate the core and drwtsn32.log files. When they are generated by the system, they need to be interrogated.

Unix platforms

Core files
Core files on Unix systems can be interrogated by dbx and gdb. Dbx is a tool that is part of the AIX install. On Sun, dbx can be installed for an additional expense. The gdb (GNU debugger) is freeware that can be downloaded.

Core file tips:

  1. Ensure that the system core file size specification is unlimited.
  2. Ensure that the file system containing the core file has enough space.


The following is a sample of how to use ulimit to verify and set the core dump size. If it is too small, a unusable core file will be generated.

Ulimit sample:

[pwh501]:root> ulimit -a

time(seconds) unlimited

file(blocks) unlimited

data(kbytes) unlimited

stack(kbytes) unlimited

memory(kbytes) unlimited

coredump(blocks) unlimited

nofiles(descriptors) 2000



The following commands will change the coredump (-c) and file (-f) to unlimited:

ulimit -f unlimited

ulimit -c unlimited



The following is an example of using the df command to verify that there is enough room in the file system for the core. The core file is placed in the ./bin directory. On AIX this is in the /usr filesystem. A core file can be 200MB.

Df sample:

[pwh501]:root> df

Filesystem 512-blocks Free %Used Iused %Iused Mounted on

/dev/hd4 131072 80416 39% 2480 8% /

/dev/hd2 8306688 2835096 66% 76320 8% /usr

/dev/hd9var 606208 55176 91% 390 1% /var

/dev/hd3 475136 459808 4% 32 1% /tmp

/dev/hd1 1310720 426120 68% 12453 8% /home

/dev/lv00 65536 47048 29% 96 2% /usr/lpp/netviewdm

/dev/lv01 606208 296504 52% 915 2% /db2

/dev/lv02 4014080 2806320 31% 3328 1% /Projects



Note: These samples were taken from the AIX 4.3.3 system.

DBX command

The purpose of the dbx command is to provide an environment to debug and run programs under the operating system. The dbx command provides a symbolic debug program for C, C++, Pascal, and Fortran programs, allowing you to carry out operations including:

  • Examine object and core files
  • Provide a controlled environment for running a program
  • Set breakpoints at selected statements or run the program one line at a time
  • Debug using symbolic variables and display them in their correct format
DBX syntax

dbx [ -a ProcessID ] [ -c CommandFile ] [ -d NestingDepth ] [ -I Directory ]

[-E DebugEnvironment ] [ -k ] [ -u ] [ -F ] [ -r ] [ -x ] [ ObjectFile

[ CoreFile ] ]

The ObjectFile parameter is an object (executable) file produced by a compiler. Use the -g (generate symbol table) flag when compiling your program to produce the information the dbx command needs.

Note: The -g flag of the cc command should be used when the object file is compiled. If the -g flag is not used or if symbol references are removed from the xcoff file with the strip command, the symbolic capabilities of the dbx command are limited.

If the -c flag is not specified, the dbx command checks for a .dbxinit file in the user's $HOME directory. It then checks for a .dbxinit file in the user's current directory. If a .dbxinit file exists in the current directory, that file overrides the .dbxinit file in the user's $HOME directory. If a .dbxinit file exists in the user's $HOME directory or current directory, that file's subcommands run at the beginning of the debug session. Use an editor to create a .dbxinit file.

If ObjectFile is not specified, then dbx asks for the name of the object file to be examined. The default is a.out. If the core file exists in the current directory or a core file parameter is specified, then dbx reports the location where the program failed. Variables, registers and memory held in the core image may be examined until execution of ObjectFile begins. At that point the dbx debug program prompts for commands.


Note: The commands are referenced in the AIX Version 4.3 Commands Reference, Volume 2.

DBX tips

The common procedure of interrogating a core file is to change the directory to where the core file resides. You can then issue the command with the binary executable file as the parameter. It is important that the binary executable is used. Usually the java command is a shell script that calls the executable. If you enter the shell script, Java, as the parameter a "cannot find" error message is returned.

The following commands show you how to find the binary executable and invoke the dbx command. It also shows an illegal instruction was executed (that is, Invalid opcode):

---------------------------------------------------------------------------------------

[pwh501]:root> cd /usr/jdk_base

[pwh501]:root> find . -name java -print

./bin/aix/native_threads/java

./bin/java

[pwh501]:root> cd /usr/WebSphere/AppServer/bin

[pwh501]:root> ls -l core

-rw-r--r-- 1 root system 191495883 Aug 07 15:08 core

[pwh501]:root> dbx /usr/jdk_base/bin/aix/native_threads/java

Type 'help' for help.

Warning: The core file is truncated. You may need to increase the ulimit for file and core dump, or free some space on the file system.

Reading symbolic information ...Warning: no source compiled with -g [using memory image in core]

Illegal instruction (reserved addressing fault) in . at 0x0 ($t29)0x00000000 00000001 Invalid opcode.

-----------------------------------------------------------------------------------------

If you don't know where the Java binary is located, the following command will display the true Java executable name of the core:

strings core | more

After you enter dbx, the where command provides a stack trace of where the error occurred. The following example shows a:


Stack trace

(dbx) where

warning: could not locate trace table from starting address 0x0

ExecuteJava(??, ??) at 0xd2f9913c

do_execute_java_method_vararg(??, ??, ??, ??, ??, ??, ??, ??) at 0xd2fabd30

execute_java_dynamic_method(0x20e355e0, 0x3002fdb0, 0xd3016aa4, 0xd3016aa8, 0x0, 0x0, 0x0, 0x0) at 0xd2fabef4

ThreadRT0(0x3002fdb0) at 0xd300cd88

sysThread_shell(??) at 0xd2fb50a8

pthread._pthread_body(??) at 0xd010f358

Output of the help command

(dbx) help

Commands:

alias   assign   attribute   call   case   catch

clear   cleari   condition   cont   delete   detach

display(/)   down   dump   edit   file   func

goto   gotoi   help   ignore   list   listi   

map   move   multproc   mutex   next   nexti

print   prompt   quit   registers   rerun   return

run   rwlock   screen   search(/?)   set   sh

skip   source   status   step   stepi   stop

stopi   thread   trace   tracei   unalias   unset

up   use   whatis   where   whereis   which

Topics:

startup   execution   breakpoints   files   data

machine   environment   threads   expressions   scope

set_variables   usage

Type "help" for help on a command or topic.

How to exit dbx

(dbx) quit

[pwh501]:root>

Another useful purpose of the dbx command is to monitor a running process. The -a parameter allows the user to attach to a process. The catch and run commands can be used to walk through the processing of the JVM process and see all signals that are caught. Use of the help xxx command will provide additional information on each of the above commands.

DBXTRACE.SH

There are shell scripts that call the dbx command and format the thread information from the core file. The name of the script is usually dbxtrace.sh. There is an AIX version and a Solaris version.

Here's a description on how to run the shell script:

[pwh501]:root> ./dbxtrace -a

Usage: Automate getting dbx trace information

For core files:

Usage: dbxtrace [executable] [core] or : dbxtrace -c corefile

Example: dbxtrace /usr/jdk_base/bin/aix/native_threads/java core

(Please make sure you use the java executable and not the java script)

To attach to a running or hung process:

Usage: dbxtrace -a PID

Example: dbxtrace -a 1234

The following information describes the beginning of the output when using dbxtrace on AIX:

[pwh501]:root> ./dbxtrace.sh | more
******************************************
* Failure of this script or dbx may *
* overwrite your existing core file. *
* It is recomended that you rename your *
* existing core file and use the -c flag *
* Do you wish to continue (y/n): ******************************************
Creating subcommand file....
Running dbx...
Type 'help' for help.
warning: The core file is truncated. You may need to increasethe ulimit for file and coredump, or free some space on the filesystem.
Reading symbolic information ...warning: no source compiled with -g

Note: The user is prompted for (y/n). Therefore, if the user redirects the output to a file [pwh501]:root> ./dbxtrace.sh > myfile 2>&1 a standalone "y" must be entered before the output is generated.

The output of the dbxtrace.sh provides information about each defined thread. The output has the following sections:

Error condition:
Illegal instruction (reserved addressing fault) in . at 0x0 ($t29)
0x00000000 00000001 Invalid opcode.

One line description for each thread:
$t29 is the current thread
thread state-k wchan state-u k-tid mode held scope function
$t1 run blocked 37671 u no sys _pthread_ksleep
$t2 run blocked 38197 u no sys _pthread_ksleep
..
>$t29 run running 46443 k no sys

Detail thread information
thread state-k wchan state-u k-tid mode held scope function
>$t29 run running 46443 k no sys
general:
pthread addr = 0x20df04e0 size = 0x18c
vp addr = 0x20e376b4 size = 0x284
thread errno = 2
start pc = 0xf0545994
joinable = yes
pthread_t = 1c1d
scheduler:
kernel =
user = 1 (other)
event :
event = 0x0
cancel = enabled, deferred, not pending
stack storage:
base = 0x20df5738 size = 0x40000
limit = 0x20e35738
sp = 0x20e35040

Stack trace of each thread
thread state-k wchan state-u k-tid mode held scope function
*$t29 run running 46443 k no sys
warning: could not locate trace table from starting address 0x0
ExecuteJava(??, ??) at 0xd2f9913c

do_execute_java_method_vararg(??, ??, ??, ??, ??, ??, ??, ??) at 0xd2fabd30
execute_java_dynamic_method(0x20e355e0, 0x3002fdb0, 0xd3016aa4, 0xd3016aa8, 0x0, 0x0, 0x0, 0x0) at 0xd2fabef4
ThreadRT0(0x3002fdb0) at 0xd300cd88
sysThread_shell(??) at 0xd2fb50a8
pthread._pthread_body(??) at 0xd010f358

Windows platform

The drwtsn32.log files are similar to core files on Unix. On Windows 2000, these files are found in the following directory: C:\Documents and Settings\All Users\Documents\DrWatson.

After entering drwtsn32 ?, the "Dr. Watson for Windows 2000" box appears. The DrWatson log file overview option will display a screen which explains the format of the drwtsn32.log files. The output of the dbxtrace.sh provides information about each defined thread. The output has the same section as a Unix platform:

Java thread dumps

Java thread dumps provide a Java view of a failing JVM process. Depending on the platform, Java dumps can appear with different names and at different locations.

A Java dump provides information about the executing Java classes and allows the problem determination process to reference the Java source code.

How to obtain a JAVA Thread Dump

There are two ways to obtain a Java thread dump:

  • DrAdmin function
  • kill -3 command

DrAdmin works on all platforms. On Unix, the kill -3 command serves the same function and is easier to use. Therefore, DrAdmin is discussed in the Windows platform section and kill -3 is discussed in the Unix platforms section.

Unix platforms

Sometimes Java thread dumps will occur due to an error in the JVM. At other times, the user might need to understand what is occurring within a JVM that is currently active. In either case, the Java thread dump is placed at the location described in the locations table. Information on how to manually obtain a thread dump is available in the remainder of this section.

When a process hangs or is working hard (that is, looping), it might be helpful to understand what the individual threads of a JVM process are doing. Obtaining a stack trace of the individual threads will provide this information. The kill -3 process ID command provides this stack trace information. This command should not impact the running process.

Identifying process IDs

WebSphere supports four processes:

  1. Nanny - started with startupServer.sh
  2. Administrative server - started by the nanny process
  3. Administrative client
  4. Application server - started by the administratiave server either automatically or manually via the administration client console

These processes are usually started in the sequence that they are listed. Therefore, their process IDs increase in value. The ps -ef | grep java command will display all the processes that are associated with java.

The process IDs are listed under the second column in the command's output.

Unfortunately, the ps -ef | grep java command does not always allow the user to identify the different processes. The command string to start the processes can be very long and the length of the command string saved by the system may not be adequate for the ps -ef | grep java command.

On AIX, the complete command line is listed in the above ps -ef | grep command output. The user can also enter the following commands to focus on an individual process ID:

  • ps -ef | grep Nanny
  • ps -ef | grep AdminServer
  • ps -ef | grep AdminClient
  • ps -ef | grep ManagedServer

There could be multiple application servers running simultaneously. The application server process ID(s) are also displayed within the ./bin/tracefile with:

Starting Server: "Default Server" (pid number=116032)

Default server is the name of the application server. The process ID is also shown in Application server properties.

As the root user, the kill -3 xxxx can now be entered where xxxx is the process ID of the WebSphere JVM in which you need to see a thread dump.

Location of thread dump

The location of the thread dump depends on the operating system.

Process

AIX 4.3.3

Sun OS 5.7

HP-UX B.11.0.0

Administrative server

./bin/javacore....txt

Appended to ./logs/tracefile

Appended to ./logs/tracefile

Application server

./bin/javacore...txt

./Appended to stderr.file for application server (Note 1)

Appended to stdout file for application server (Note 1)

Administrative client

./bin/javacore...txt

Prompted at window used to enter adminclient.sh

<install_root>/logs/adminclient_audit_messages.log

Nanny

./bin/javacore...txt

Prompted at window used to enter startupServer.sh

Window used to enter startupServer.sh

Note 1: The stderr and stdout files are defined within the application server configuration. See the article Application server properties for more information.

If the user starts a server in the background, the kill command may not dump the thread information. The workaround for this situation is to do the following:

startupServer.sh &

Ctrl+Z

fg

kill -3 xxxx

Windows platform

DrAdmin.bat file

The DrAdmin.bat file is located in the product_installation_root/bin directory. The DrAdmin.bat file will execute the DrAdmin function. In Unix, the DrAdmin.bat file is DrAdmin.sh.

Find the port number of interest

The next step is to identify the port number for either the administrative server or a application server. The port number is different for the administrative server and each of the application server(s). The port number values are contained in the standard out files for each of these processes. Information on how to find these files and the port number are described below.

After starting the administrative server, you should obtain the DrAdmin port number within the .\logs\tracefile file inside the message:

DrAdmin available on port xxxx

After starting the application server (for example, default server), you should obtain the DrAdmin port number provided in the standard output file for the application server. The message within the file containing the port number is:

DrAdmin available on port xxxx

Execute DrAdmin

The DrAdmin.bat file can now be executed providing the port number obtained above. The format of the command to use is:

DrAdmin -serverPort xxxx -dumpThreads

where xxxx is the port number from the above message (without the comma).

Locate the thread dump

The location of the administrative server thread information depends on how the administrative server was started. The javacore...text file is placed in the <Windows_Home>\WINNT\system32 directory or the <WebSphere_Home>\bin directory.

The thread information can be viewed with a text file viewer such as Notepad.

How to interrupt a Java thread dump

A thread dump can be forced or can occur when a Java process error occurs. When a thread dump is not forced, it usually means that an error within a Java process has occurred and it should to be investigated. A thread dump of a Java process needs to be forced when the process has a thread deadlock condition. A thread deadlock condition is defined as:

Thread A currently owns Lock X.
Thread B owns Lock Y.
Thread A is waiting for the release of Lock Y in order to continue processing.
Thread B is waiting for the release of Lock X in order to continue processing.

Because of this stalemate condition, neither thread is able to complete its processing.

Note:The referenced Java thread dump information is taken from a sample AIX dump. Java thread dumps on other platforms have similar information, but they may be formatted different.

Monitors

In order to have a thread safe application, the application may have to ensure that two threads don't execute the same code simultaneously. This can be accomplished with the use of a synchronized()statement or a synchronized modifier of a class method.

Each of the above threads in the thread deadlock condition will create a monitor/lock that will prevent other threads from executing the same code. It is important to understand that threads can be holding multiple monitors/locks while processing a request. Therefore two threads could find themselves in a deadlock condition defined by the following situation:

Thread A owns Lock X.
Thread B owns Lock Y.
Thread A is waiting for the release of Lock Y.
Thread B is waiting for the release of Lock X.

Because of this stalemate condition, neither thread is able to complete its processing.

Example of a deadlock condition
You can recognize a deadlock when looking within the native stack information. For example, when looking at the native stack information of Thread A you can easily recognize that it is blocked by a monitor/lock held by Thread B. This information does not appear in the native stack information of the Thread B. Thread B is currently deleting a connection, (deleteConn), from the ConnectionTable. The deleteConn()is a synchronized method which causes a monitor/lock to occur for the ConnectionTable class. There is only one ConnectionTable instance. The monitor/lock held by Thread B is preventing the Thread A process from completing.

The above diagnosis requires an understanding of the involved source code (that is, which methods are synchronized). However, the Java thread dump does provide the pointers to do this additional investigation.

A summary of the object monitors will provide additional information that identifies Thread A is blocked by Thread B:
com.ibm.CORBA.iiop.IIOPConnection@4fe89740: owner: "Thread B" "Thread A" (0x36951ba8) blocked

Unfortunately, this information does not appear in the summary for the monitor being held by Thread A.

Stack traces

Stack traces represent the current call path of a thread. Call path information explains what functional calls were made to get to the thread's current location.

System dump stack trace

Note that the sysAcceptFD() call is the last function called on the stack. It is a system call that was invoked by java_net_PlainSocketImpl_socketAccept() call. The call indicates that a Java thread did an accept operation on a socket. Question marks appear as parameters. This is because the Java process was not run in debug mode. For Java 1.1 installations (that is, before WebSphere Application Server, version 3.5), debug mode is started by using the java_g command. For Java 1.2 installations (that is, WebSphere Application Server, version 3.5), the -Xdebug options should be used with the Java command. As stated above, no class file information appears in the stack trace. Only the functions with C libraries are referenced.

Java dump stack trace

The reader is able to follow the sequence of calls from the run() method through the read() method of the SocketInputSteam class. The package names of the classes are also present. The "Compiled Code" characters appears as parameters in the call because the Java dump occurred for a JVM that was not running in debug mode. When running in debug mode, the line number of the call within the source replaces the "Compiled Code" characters. For Java 1.1 installations (that is, before WebSphere Application Server, version 3.5), debug mode is started by using the java_g command. For Java 1.2 installations (that is, before WebSphere Application Server, version 3.5), the -Xdebug options should be used with the Java command. Another way of obtaining the source line number is to turn off the JIT (Just In Time) compiler. This can be done by starting the JVM with the -Djava.compiler=NONE parameter. This parameter can also be placed on the application server command line.

WebSphere Application Server thread information

Object Request Broker information

During startup of the different WebSphere Application Server processes, the processes are initialized and placed in a state to accept additional network activity. One of the first steps in initializing a process is to create an ORB instance. This step will create threads that will be used to complete the initialization step and later accept network activity to be processed.

These activities are described within each of the two diagrams of the next two sections. The diagrams describe:

The administrative server has two ORBs defined within it. For each ORB is at least one ORB server listener thread that continually waits for input on a port. When input is received, it is dispatched to an ORB server reader thread so the ORB server listener thread can again wait for input on the port. The ORB server reader thread again dispatches the request to a third thread that completes the work activity. The reply to the work activity is sent from the third thread to an ORB client reader thread that receives replies from the ORB reader thread. An ORB request has four steps/threads involved:

  1. ORB server listener thread receives input on port X.
  2. ORB server reader thread is given a request.
  3. Pooled/instantiated thread handles the request and sends a reply.
  4. ORB client reader thread handles the reply.

The application server has one ORB defined within it. There are two ORB server listener threads and multiple ORB client threads. The servlet traffic does not use the ORB for communications. It is done with the plug-in interface. This interface supports a pool of worker threads (Worker#_) that complete the HTTP requests.

The following port numbers are preset:

  • 9000 is used for obtaining naming services (that is, data source names, enterprise bean names)
  • 900 is used by the administrative server to listen for administrative client requests

Other port numbers are randomly chosen for ORB communications.

Thread names

ORB threads

The ORB instance creates reader and listener threads. The names of these threads get changed after they begin processing (that is, during run() method processing). The name is constructed with the following parameters separated by a colon (:):
  • ORB information
    • P = unique for this process and algorithmically constructed from a time stamp
    • O = number of ORBs within this process
  • Thread type
    • StandardRT = identifies which reader thread is within the ORB
    • CT = client thread
    • LT = listener thread
  • Connection values
    • LocalPort = Local port that thread is dealing with
    • RemoteHost = Hostname for ORB server reader thread, or IP address for ORB client reader thread
    • RemotePort = Port number on the remote host for the connection
Worker#__ (SERVLET ENGINE THREADS)

These thread names begin with Worker# and process HTTP requests.
Thread-x

These thread names are the default thread name for Windows 2000 and AIX. Because no thread name is provided this name is used. X is incremented as each new thread is created.
Pooled ORB request dispatch WorkerThread

These threads are created by the main thread (that is, P=479481:O=0:CT)and handle the request/replies that are sent across IIOP connections.
Web server plug-in configuration thread

Thread used for setting the Web server configuration.
Alarm manager

This thread manages the creation of alarm thread x's.
Alarm thread 1

The alarm thread 1 reclaims unused connections.

BackgroundLruEvictionStrategy

This thread sweeps a cache, reclaiming the least recently used objects.
Refresh

This thread insures that any changes to a model get propagated to clones.

Thread stack traces

When a thread is created, the start() method is used to invoke the run() method. The start() method is executed on one thread and the run() method is executed on the newly created thread. Depending on when the stack trace is obtained, an activity could have different stack traces. Therefore, thread names have two base method calls. The following text describes these base method calls for the common thread names used for both the administrative server and the application server. Two stack trace examples of base method calls are also provided:

Base method calls

  1. Main or P=xx:O=0:CT
    • run ---> com.ibm.ejs.sm.server.AdminServer.main()


  2. ORB server listener thread (JavaIDL Listener or P=xx:O=0:LT=0:port=9000)
    • start ---> com.ibm.ejs.sm.server.AdminServer.main()
    • run ---> com.ibm.CORBA.iiop.ListenerThread.run()
  3. ORB server reader thread (JavaIDL Reader for hostname:port# or P=xx:O=0:StandardRT=0:LocalPort=port#:RemoteHost=hostname:RemotePort=port#:)
    • start ---> com.ibm.CORBA.iiop.ListenerThread.run()
    • run ---> com.ibm.CORBA.iiop.StandardReaderThread.run()
  4. ORB client reader thread (JavaIDL Reader for ipaddr:port# or P=xx:O=1:StandardRT=1:LocalPort=port#:RemoteHost=ipaddr:RemotePort=port#:)
    • start ---> com.ibm.ejs.sm.server.AdminServer.main()
    • run ---> com.ibm.CORBA.iiop.StandardReaderThread.run()
  5. Pooled ORB request dispatch WorkerThread
    • start ---> com.ibm.CORBA.iiop.StandardReaderThread.run()
    • run ---> com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run()
  6. Worker#__
    • start ---> com.ibm.ejs.sm.server.AdminServer.main()
    • run ---> com.ibm.servlet.engine.oselistener.outofproc.OutOfProcThread$CtlRunnable.run()java.lang.Thread.run()
  7. Web server plug-in configuration thread
    • start ---> com.ibm.ejs.sm.server.AdminServer.main()
    • run ---> com.ibm.servlet.engine.oselistener.outofproc.OutOfProcThread$CtlRunnable.run()java.lang.Thread.run()
  8. Alarm manager
    • start ---> com.ibm.ejs.sm.server.AdminServer.main() <--AdminServer
    • com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run() <--AppServer

    • run ---> com.ibm.ejs.util.am.AlarmManagerThread.run() java.lang.Thread.run()
  9. Alarm thread 1
    • start ---> com.ibm.ejs.util.am.AlarmManagerThread.run() java.lang.Thread.run() <--AdminServer
    • com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run()
    • <--AppServer
    • run ---> com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run() <--AdminServer
    • com.ibm.ejs.util.am.AlarmThread.run() <--AppServer
  10. BackgroundLruEvictionStrategy
    • start ---> com.ibm.ejs.sm.server.AdminServer.main()
    • run ---> com.ibm.ejs.util.cache.BackgroundLruEvictionStrategy.run()
  11. RefreshThread
    • start ---> com.ibm.ejs.sm.server.AdminServer.main()
    • run ---> com.ibm.ejs.wlm.server.config.ServerGroupRefresh$RefreshThread.run()

Examples

Thread dump of a standard reader thread:

"P=863240:O=1:StandardRT=16:LocalPort=10502:RemoteHost=gofast:RemotePort=2619:"
(TID:0x11ccef0, sys_thread_t:0xcdd81d0, state:R, native ID:0x128) prio=5
> at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
at com.ibm.rmi.iiop.Message.readFully(Message.java(Compiled Code))
at com.ibm.rmi.iiop.Message.createFromStream(Message.java:173)
at com.ibm.CORBA.iiop.IIOPConnection.createInputStream(Unknown Source)
at com.ibm.CORBA.iiop.StandardReaderThread.run(Unknown Source)

The base method, com.ibm.CORBA.iiop.StandardReaderThread.run(), is identified as the run base method for JavaIDL Reader for hostname:port# threads. Also, the thread is waiting for input because it is in the java.net.SocketInputStream.socketRead() method.

Thread dump of a worker thread:

"Worker#49" (TID:0x10793660, sys_thread_t:0xab25b0, state:R, native ID:0x19a) prio=5
at com.ibm.servlet.engine.oselistener.outofproc.NativeServerQueueImp.nativeGetSeviceMessageId()
at com.ibm.servlet.engine.oselistener.outofproc.NativeServerQueueImp.getSeviceMessageId()
at com.ibm.servlet.engine.oselistener.serverqueue.SQWrapperEventSource$SelectRunnable.getNewConnectionFromQueue()
at com.ibm.servlet.engine.oselistener.serverqueue.SQWrapperEventSource$SelectRunnable.run()
at com.ibm.servlet.engine.oselistener.outofproc.OutOfProcThread$CtlRunnable.run()
at java.lang.Thread.run()

The base method, java.lang.Thread.run(), is identified as the run base method for Worker#__ threads. Also, the thread is waiting for input from the Web server plug-in (native code) because it is in the com.ibm.servlet.engine.oselistener.outofproc.NativeServerQueueImp.nativeGetSeviceMessageId() method.

Administrative server startup with immediate takedown diagram

The following diagram has highlighted request flows that start with a SendReqXXX where XXX is the port number of the send request. The steps in the flow changes between different threads. The sequence of the steps are identified with, for example, 1A,1B, 1C and 1D. It also shows how the port that the request is sent to determines which thread the processing has completed.

Diagram Legend

In each diagram, every continuous line (-------) is a thread. The name of the thread always appears between (...). The letters in the diagram have the following meanings:

C = Thread name changed to (.....)
S = Start method called on this thread
R = Run method called on this thread
W = Thread is in wait state waiting for notify
WM = Thread is waiting for message from plug-in (Worker# threads only)
SendReq____ = Request sent to port number (____)
SendReply___ = Reply sent to port number (____)

For example:

C(P=479481:O=0:CT) = thread name is changed to P=479481:O=0:CT
R = thread is placed in a running state

Diagram

ORB 0 Threads(that is, O=0)

main

|

|

C(P=479481:O=0:CT)

|

|S(JavaIDL Listener) R C(P=479481:O=0:LT=0:port=9000)

|-----------------------------------------------------------------------------------------> | |

| S(JavaIDL Reader for rbostick:1294)

| |

| |

| R

| |

| |

| C(P=479481:O=0:StandardRT=0:LocalPort=9000:

| RemoteHost=rbostick:RemotePort=1294:)

| |

| 1B

| |S(Thread-1) R

SendReq9000(1A) |-1C--------------------->

| |

| 2B

| |S(Thread-2) R

SednReq9000(2A) |-2C--------------------->

| |

| 5B

| |

| |

| |

| V

|

|S(JavaIDL Reader for 9.27.63.245:9000) R C(P=479481:O=1:StandardRT=1:LocalPort=1294:

| RemoteHost=9.27.63.245:RemotePort=9000:)

|------1D--2D--5D-------------------------------------------------------------------------> |

|

|

|

|

|

|

|

|

|

|

|

V

ORB 1 Threads (that is, O=1)

|

|

|

|S(JavaIDL Listener) R C(P=479481:O=1:LT=1:port=1295)

|--------------------------------------------------------------------------------------------> |

|

|

|

|

|S(JavaIDL Listener) R C(P=479481:O=1:LT=2:port=1296)

|-------------------------------------------------------------------------------------------->

| |

| S(JavaIDL Reader for rbostick:1299)

| |

| R

| |

| C(P=479481:O=1:StandardRT=5:LocalPort=1296:

| RemoteHost=rbostick:RemotePort=1299:)

| |

| 6B

| |

| V

|

|

|

|S(JavaIDL Reader for 9.27.63.245:1299) R C(P=479481:O=1:StandardRT=4:LocalPort=1299:

| RemoteHost=9.27.63.245:RemotePort=1296:)

|------6D--------------------------------------------------------------------------------->

|

|

|

|S(JavaIDL Listener) R C(P=479481:O=1:LT=3:port=900)

|--------------------------------------------------------------------------------------------> | |

| S(JavaIDL Reader for rbostick:1297)

| |

| R

| |

| C(P=479481:O=1:StandardRT=3:LocalPort=900

| :RemoteHost=rbostick:RemotePort=1297:)

| |

| 3B

| |

| 4B

| |

| V

|

|

|S(JavaIDL Reader for 9.27.63.245:900) R C(P=479481:O=1:StandardRT=2:LocalPort=1297:

| RemoteHost=9.27.63.245:RemotePort=900:)

|-----3D--4D--------------------------------------------------------------------------------->

|

|

|

SendReq900(3A)

|

SendReq900(4A)

|

SendReq9000(5A)

|

SendReq1296(6A)

|

SednReq1296(7A)

|

Other Threads

|

|

|

|

|

|S(Pooled ORB request dispatch WorkerThread) W R

|------3C--5C----------------------------------------------------------------------------->

|

|

|

|S(Pooled ORB request dispatch WorkerThread) W R |------4C--6C----------------------------------------------------------------------------->

|

|

|

|

|S(Alarm Manager) R

|----------------------------------------------------------------------------------------->

| S(Alarm Thread 1)

| |

| R

| |

| V

|

|S(Thread-3) R

|---------------------------------------------------------------------------------------->

|

|

|S(Thread-4) R

|---------------------------------------------------------------------------------------->

| | | | | |

| S(Thread-8) S(Thread-9) S(Thread-10) S(Thread-11) S(Thread-12) | | | | | |

| R R R R R

| | | | | |

| V V V V V

|

|

|S(Worker#0) R S(Worker#0) R

|-------------------------------------------------------------------------------------------->

|

|

|S(WebServer-Plugin-Cfg-Thread) R

|-------------------------------------------------------------------------------------------->

|

|S(BackgroundLruEvictionStrategy) R

|-------------------------------------------------------------------------------------------->

|

|S(RefreshThread) R

|--------------------------------------------------------------------------------------->

V

Thread-1(2) (worker threads)

  • start ---> com.ibm.CORBA.iiop.StandardReaderThread.run()
  • run ---> com.ibm.CORBA.iiop.WorkerThread.run()

Thread-3 (transaction timeout)

  • start ---> com.ibm.ejs.sm.server.AdminServer.main()
  • run ---> com.ibm.ejs.jts.tran.JavaClock.run()

Thread-4 (used for administrative server takedown)

  • start ---> com.ibm.ejs.sm.server.AdminServer.main()
  • run ---> com.ibm.ejs.sm.server.ManagedServer$DiagonisticThread.run()

Thread-8,9,10,11,12 (threads for takedown process)

  • start ---> com.ibm.ejs.sm.server.ManagedServer$DiagonisticThread.run()
  • run ---> com.ibm.ejs.sm.util.task.AsyncTaskEngine$WorkerThread.run()

Note: Thread-x are default names of threads. The above numbers may be different depending on the system that the administrative server runs on.

Application server startup with servlet traffic diagram

The Worker#_threads are the threads on which servlet requests are processed. The threads start during the application server startup and wait on input from the Web server plug-in interface.

main

|

|

C(P=905990:O=0:CT)

|

|

|S(Thread-0) R

|--------------------------------------------------------------------------------------->

|

|

|S(Pooled ORB request dispatch WorkerThread) W R

|-------------------------------------------------------------------------------------------->

| | | | | | | |

| S | | S(Worker#0)S(Worker#1).......S(Worker#24) S(Thread-6) (BackgroundLruEvictionStrategy) | | | |

| | | R WM WM WM R

| R | | | | | |

| | S(AlarmManager) | S(Worker#0) | Servlet |

| | | | | | Request |

| | | S(pluginRegenScheduler) | | | |

| | R | WM | WM |

| | | V V V V V

| | |

| | |

| V |S(AlarmThread1) R

| |---------------------------->

| V

|

|

|S(Pooled ORB request dispatch WorkerThread) W R W |------------------------------------------------------------------------------------>

|

|S(Thread-1) R

|------------------------------------------------------------------------------------>

|

|

|

|S(Thread-3) R

|------------------------------------------------------------------------------------>

|

|

|

|

ORB 0 Threads (that is, O=0)

|

|S(JavaIDL Reader for 9.27.63.129:9000) R C(P=905990:O=0:StandardRT=0:LocalPort=1480:

| RemoteHost=9.27.63.129:RemotePort=9000:

|------------------------------------------------------------------------------------>

|

|

|

|S(JavaIDL Listener) R C(P=905990:O=0:LT=0:port=1481)

|------------------------------------------------------------------------------------->

|

|

|S(JavaIDL Reader for 9.27.63.129:1434) R C(P=905990:O=0:StandardRT=1:LocalPort=1482:

| RemoteHost=9.27.63.129:RemotePort=1434:)

|------------------------------------------------------------------------------------->

|

|

|

|

|S(JavaIDL Reader for 9.27.63.129:900) R C(P=905990:O=0:StandardRT=2:LocalPort=1483:

| RemoteHost=9.27.63.129:RemotePort=900:)

|------------------------------------------------------------------------------------->

|

|

|

|S(JavaIDL Reader for 9.27.63.129:1433) R C(P=905990:O=0:StandardRT=3:LocalPort=1484:

| RemoteHost=9.27.63.129:RemotePort=1433:)

|------------------------------------------------------------------------------------->

|

|

|

|

|S(JavaIDL Listener) R C(P=905990:O=0:LT=1:port=1485)

|------------------------------------------------------------------------------------------>

| |

| S(JavaIDL Reader for rbostick:1487)

| |

| R

| |

| C(P=905990:O=0:StandardRT=4:LocalPort=1485:

| RemoteHost=rbostick:RemotePort=1487:)

| |

| |

| V

V

Thread-0 (transaction timeout)

  • start ---> com.ibm.ejs.sm.server.ManagedServer.main()
  • run ---> com.ibm.ejs.jts.tran.JavaClock.run()

Thread-1 (Used for logging messages)

  • start ---> com.ibm.ejs.sm.server.ManagedServer.main()
  • run ---> com.ibm.ejs.sm.server.SeriousEventListener$DeliveryThread.run()

Thread-3

>
  • start ---> com.ibm.ejs.sm.server.ManagedServer.main()
  • run ---> com.ibm.ejs.sm.server.ManagedServer.main()

Thread-6 (administrative server ping)

  • start ---> com.ibm.ejs.oa.pool.ThreadPool$PooledThread.run()
  • run ---> com.ibm.ejs.sm.server.ManagedServer$PingThread.run()

Note: Thread-x are default names of threads. The above numbers may be different depending on the system the application server runs on.

Summary

In multi-processing and multi-thread environments, problem determination can require analysis of actively running threads. This thread information can be obtained with system thread dumps and Java thread dumps. When doing problem determination in a WebSphere Application Server environment, Java thread dumps provide much more information and are recommended. However, sometimes system thread dumps are the only information obtained and should be interrogated.

When dealing with thread deadlock problems, Java thread dumps can be forced using kill -3 on Unix platforms and DrAdmin on all platforms.

The output of these commands provides thread information necessary to diagnose the problem.

Go to previous article: Log Analyzer Go to next article: Applying e-fixes

 

 
Go to previous article: Log Analyzer Go to next article: Applying e-fixes