Once the issue occurs, it can be easily and repeatedly reproduced by executing the following command from a command prompt:


# JAVA_PATH/jre/bin/java -version (e.g., /usr/java6/jre/bin/java -version)


Also, this issue occurs with IBM Java for AIX versions 6 and higher (e.g., IBM Java 6, IBM Java 7, and IBM Java 8).

Issue:

An AIX system was not rebooted after applying an AIX updated requiring that system to bre rebooted.


Root Cause:

When an AIX update modifies the critical kernel and device components, a reboot may be required to ensure the critical kernel components are correctly initialized during the boot up process. If the system is not rebooted, these kernel components remain in an invalid or corrupted stated, resulting in various issues.


Solution:

The solution is to simply reboot the system as required by the updates.

If this is the root cause, the issue will not reoccur once the system is rebooted.

If, after rebooting the system, the issue eventually reoccurs, then continue to next scenarios.

Issue:

A known AIX defect with AIX 6.1 and AIX 7.1.


Root Cause:

AIX systems with the following configuration:

- AIX 6.1 without APAR IV03746 installed
- AIX 7.1 without APAR IV09585 installed

are susceptible to a known AIX kernel defect that can cause this Java issue. To confirm if the appropriate APAR has been installed, from a command prompt, execute the following commands:

# instfix -i | grep -i "iv03746|iv09585"

{One of the following lines should be displayed}

(AIX 6.1) All filesets for IV03746 were found.
(AIX 7.1) All filesets for IV09585 were found.

Another method that may also be used to confirm this issue would be to executed the following commands from a command prompt:

{Use the command below that reproduces the "Port Library" error}

# truss -dealfo /tmp/java_truss.out JAVA_PATH/jre/bin/java -version
or
# truss -dealfo /tmp/java_truss.out FAILING_COMMAND

# grep -i "proc.*attr.*Err" /tmp/java_truss.out

6619232: 24051823: 49.2677: proc_setattr(-1, 0x000001001013EDD0, 5) Err#22 EINVAL

If the output shows a reference to "proc_setattr" and "Err", the issue is due to the missing APAR.
If the output is not similar to the above line, it DOES NOT mean its not the issue (it could be with some other call).

This issue does not occur for all systems that do not have the APARs applied. Some customers have reported that they have mutliple systems at the same AIX release level, however the issue only occurs with one or two of their systems.

Solution:

If, after running the instfix command above, the APAR is listed as "found", then continue to the next scenarios.


If, after running the instfix command above, the APAR is NOT listed, the APAR for that AIX version must be downloaded and installed, and the system must be rebooted. APAR (fix) details are available on this web page:

https://www-304.ibm.com/support/docview.wss?uid=isg1fixinfo134913

The description of the problem on the AIX APAR web pages do not make a direct reference to the IBM Java for AIX messages.

Also, for some customers, the impacted system may be at a higher AIX release level that reported in the APAR web pages. Customers must still confirm that the appropriate APAR has been installed using the instfix command above, and install the APAR if it not installed.

Due to system and application uptime requirements, applying the APAR and rebooting the system may be difficult to schedule. In most cases, simply rebooting the system (without applying the APAR) will provide relief (and prevent the issue from reoccurring).


For IBM WebSphere and IBM HTTP Server customers, similar information is available on this web page:

http://www-01.ibm.com/support/docview.wss?uid=swg21575120

If this is the root cause, once the correct APAR has been installed and the system has been reboot, the issue will not reoccur.

If, after applying the APAR, the issue eventually reoccurs for the same system, then continue to the next scenarios.

Issue:

An application does not cleanup semaphores (a.k.a., semaphore leak).



Root Cause:

AIX has a maximum limit of 65535 semaphores per ID (click here for more details). There have been reported situations when applications (Java or non-Java) are not correctly deleting semaphores, resulting in a gradual semaphore leak. When the number of semaphores reaches the maximum limit for the user id, the JVM is unable to create the required semaphores on startup, resulting in the "Port Library" error message.

To confirm if the semaphore leak is occurring, from a command prompt, repeatedly execute the commands:

# ipcs -saPrX | wc -l

over a period of time. If the number of semaphores continue to increase into the 10000-30000 range, then there is a semaphore leak.


Solution:

a. Rebooting the system will provide temporary relief for the issue. However, over time, the issue is expected to reoccur.

b. If IBM InfoSphere or IBM DataStage is running on the same system, then follow the instructions on the web page:

http://www-01.ibm.com/support/docview.wss?uid=swg21654008

c. Open a support call to the vendor / team for the application (not the JVM) creating, but not deleting the semaphores. That vendor / team will be responsible to resolving the issues of the semaphores not being released.

Issue:

The Java Attach API needs to be disabled.


Root Cause:


The Java Attach API feature allows a remote process to attach to the Java (JVM) process. This startup of this feature is part of the initialization of the Java process and if the resources required are not available, the "Port Library" issue can occur. In some cases, disabling the Attach API feature can prevent the issue from occurring.

Solution:

For most applications, this feature is not required and can be disabled by adding the IBM Java command line option:

-Dcom.ibm.tools.attach.enable=no

when the application is started (this will require an application restart).

To confirm if this option is safe for the application(s), contact the application vendor (not the IBM AIX Java team) for assistance.

For additional information about the Java Attach API implementation when using IBM Java for AIX, visit the web page:

https://www-01.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/com.ibm.java.aix.70.doc/user/attachapi.html

If, after disabling the Attach API, the problem still occurs, continue to the next section.

Issue:

The AIX system does not meet the minimum installed level to support the version of IBM Java for AIX.


Root Cause:

When the AIX system does not have the minimum required installed levels for the version of Java, unpredictable errors may occur such as the "Port Library" message or other types Java process (JVM) crashes.


Solution:

Users should visit the web pages listed below to confirm their system's meet the minimum operating system requirements:


Minimum AIX levels: http://www-01.ibm.com/support/docview.wss?uid=isg3T1022644#requirements

Minimum Operating Levels for Power 7 Support: http://www.ibm.com/developerworks/java/jdk/power7/index.html

Minimum Operating Levels for Power 8 Support: http://www.ibm.com/developerworks/java/jdk/power8/index.html


Then, users must upgrade AIX and/or Java version that do not meet the minimum requirements.

Below is a list of additional solutions that may resolve the issue, but these solutions have not been confirmed.


A. The Java shared libraries need to be cleared

1. Stop all applications using that version of Java
2. From a command prompt, and while logged in as the root user, execute the command:

# slibclean -a

3. Retry the application.


B. There is a permissions conflict with /tmp and/or the Attach API directory

1. Stop all applications using that version of Java
2. Add the IBM Java command line option:

-Dcom.ibm.tools.attach.directory=ATTACH_PATH

where ATTACH_PATH is a different directory for each application (e.g., /tmp/attach/app1, /tmp/attach/app2, etc). Each path must be created, with the correct permissions, prior to starting the application.

3. Retry the application


C. The Java (JVM) process memory size is being restricted

1. Stop the application (and if they are running, any node agents or node manager processes for the application server)
2. From a command prompt, and while logged in as the root user, execute the command:

# chuser fsize=-1 data=-1 USERID

3. Relogin as USERID, then from that new session, restart all application(s).
4. Retry the application


D. Not enough free disk space

1. From a command prompt, and in the start up directory of the application, execute the command:

# df -kP ./ /tmp

2. If the amount of free space for the start up directory or the /tmp file system is very low (99% used), then increase the file system size.
3. Retry the application.


E. Corrupted IBM Java for AIX installation

1. It may be that some IBM Java for AIX shared library files have become corrupted or overwritten which is contributing to the issue.
2. Delete, then re-install that version of IBM Java for AIX
3. Retry the application.

If none of the above actions have resolved the issue, then complete the following steps:


1. From a command prompt, and while logged in as the root user, execute the commands:


# mkdir -p /TMP_PATH/PMR/MM-DD/port_library

# cd /TMP_PATH/PMR/MM-DD/port_library

# snap -cad ${PWD}

# ipcs -saPrX > ipcs.out 2>&1

{The following 'java -version' command should generated the "Port Library" message}
{If this command does not generate the message, then it will be important to use a}
{command that will generate the message.}

# truss -dealfo java_truss.out JAVA_PATH/jre/bin/java -version 2>&1 | tee java_version.out

# cd /TMP_PATH/PMR/MM-DD

# tar -cf - port_library | gzip -c PMR.MM-DD.tgz


2. Open a new IBM support call with the "IBM AIX Java" team. The "IBM AIX Java" team will confirm and require that Scenarios #1 - #5 have been eliminated as potential causes for this issue, since these are known contributors to the issue. Therefore, to expedite the resolution of the issue, please confirm that all of the above scenarios have been completed.

When the IBM support call is created, the answers to the following questions must be provided:

a. Has the system been rebooted? If not, when was the last time the system was rebooted? (see "Scenario #1" section)

b. Do the impacted systems have IV03746 (AIX 6) or IV09585 (AIX 7) installed? (see "Scenario #2" section)

c. Do the impacted systems meet the minimum operating requirements? (see "Scenario #5" section)

d. Did the ipcs command report 10,000+ semaphores created? (see "Scenario #3" section)

e. Did disabling the Attach API resolve the issue? (see "Scenario #4" section)

f. What "Other Considerations" were tested and did or did not work? (see "Other Considerions" section)


3. Upload the packaged data using one of the IBM secure upload methods listed on the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619