Solution:

The most common scenario of core dumps generated during LPM on Power 8, can be fixed by applying the fixes for the following two known issues for specific AIX releases.

The first known issue with core dumps on Power 8 is addressed in the following APAR:

IV88479: RANDOM PROCESSES CORE DUMP ON POWER8 RUNNING IBM JAVA 8 (AIX 7100-03)
IV86126: RANDOM PROCESSES CORE DUMP ON POWER8 RUNNING IBM JAVA 8 (AIX 7100-04)
IV88175: RANDOM PROCESSES CORE DUMP ON POWER8 RUNNING IBM JAVA 8 (AIX 7200-00)

The second known issue is addressed in the following APARs for specific AIX releases and needs to be applied along with the above APAR:

IV89524: COREDUMP OF PROCESSES WITH EBB CONTEXT AFTER PARTITION MIGRATION (AIX 7.1)
IV89475: COREDUMP OF PROCESSES WITH EBB CONTEXT AFTER PARTITION MIGRATION (AIX 7.2)

There are no fixes available from fix central to address these APARs at the time of publishing this document. Hence, a combo ifix to address both the above known issues, has to be requested from the PTOOLs team (Request queue PTOOLS,165). When requesting and before applying any ifix, customers must assess the risk of applying the ifix since they go through a minimum amount of testing and are temporarily supported until the official update is available for download.

In the common scenario of core dumps during LPM, installing the above ifixes for specific AIX releases and rebooting the LPAR, will prevent the issue from reoccurring. There are other possible scenarios where the core dumps can occur on Power 8, though not using LPM. To identify these situations, please contact IBM support after completing the data collection.

NOTE: The IBM Java APAR "IV65519: JAVA 7.1SR1 JIT SIGILL CRASH ON AIX WHEN RUNNING ON POWER8 " was released in an effort to work around the issue when using IBM Java 7.1 SR1 and prior releases. Until the AIX APARs can be applied, the issue can continue to occur even if the APAR IV65519 is applied.

The instructions that follow provide both AIX and Java temporary solutions to work around the issue if the APAR can not be applied or the customer chooses to wait till the availability of official fix.

Workarounds:

Below is a list of workarounds that may resolve the issue as an alternative to applying an ifix:

AIX Workarounds:

1) Run in Power 7 Compatibility Mode as a temporary fix.
2) If migrating using LPM, stop the processes before the migration and restart them after the migration, to avoid processes from crashing.

Workarounds:

Below is a list of workarounds that may resolve the issue as an alternative to applying an ifix:

1) AIX Workarounds:

a. Run in Power 7 Compatibility Mode as a temporary fix.

b. If migrating using LPM, stop the processes before the migration and restart them after the migration, to avoid processes from crashing.

2) IBM Java Workarounds:

a. Disable the RI facility by adding the JVM command line option to the java command of the crashing process(es):

-XX:-RuntimeInstrumentation

b. Disable hardware profiling by adding the JVM command line option to the java command of the crashing process(es):

-Xjit:disableHardwareProfiling

c. If using Java 7.1(prior to SR1), upgrade Java to SR2 or above to fix the issue as per:

IV65519: JAVA 7.1SR1 JIT SIGILL CRASH ON AIX WHEN RUNNING ON POWER8

Work-arounds:

Below is a list of workarounds that may resolve the issue as an alternative to applying an ifix:

AIX Work-arounds:

a. Run in Power 7 Compatibility Mode as a temporary fix.

To determine the processor mode, at the command prompt, execute the following command:

# prtconf | grep "Processor Implementation Mode"

CPU Compatibility Mopde can be changed to POWER 7 from the HMC through the LPAR properties, following the steps below:

i. Select the LPAR and deactivate it.
ii. Go to Configuration -> Manage Profiles.
iii. Select the profile that needs to be activated.
iv. Go to Processors, change the Processor compatibility mode to Power 7 and boot it using the same profile.

b. If migrating using LPM, stop the processes before the migration and restart them after the migration, to avoid processes from crashing.

For example, from a command line prompt, run the following commands to identify the Java applications using Java 7.1 or Java 8:

# ps -ef | grep java

For each command listed, work with the appropriate system's or application's administrator to temporarily shutdown these processes using Java 7.1 or Java 8, before using LPM and restart once LPM has completed.

IBM Java Work-arounds:

a. For IBM Java 8, disable the Runtime Instrumentation facility by adding the JVM command line option to the java command of the crashing process(es):

-XX:-RuntimeInstrumentation

b. For IBM Java 7.1, disable hardware profiling by adding the JVM command line option to the java command of the crashing process(es):

-Xjit:disableHardwareProfiling

Troubleshoot:

If, after following the above instructions, the application continues to generate core dumps, please complete the following steps:

1. Confirm that you have completed all of the above steps.
2. Contact IBM and open a new IBM service request (new PMR).
3. Collect and upload data as per the data collection procedure for automatically generated process core dumps.

Step 7:

ACTION

Step 8:

ACTION

Step 9:

ACTION

Step 10:

ACTION

Step 11:

ACTION

Step 12:

ACTION

Step 13:

ACTION

Step 14:

ACTION

Step 15:

ACTION

Step 16:

ACTION

Step 17:

ACTION

Step 18:

ACTION

Step 19:

ACTION

Step 20:

ACTION

Step 21:

ACTION