Tool Mentor: Analyzing Test Failures using IBM Rational Quality Manager
This tool mentor describes how to analyze the test results using RQM.
Tool: IBM Rational Quality Manager
Relationships
Related Elements
Main Description

Introduction

After running a test, all the test failures need to be located, isolated and diagnosed. RQM proposes multiple ways of conducting this analysis. Once understood, those test failures need to be appropriately documented and communicated to the development team.

Tool Steps

1. Open the Log files

In order to analyze any test failure using RQM, the first step that you must perform is to open a test log attached to a specific test execution record. Rational Quality Manager offers multiple possible paths to open a test log for a test execution record.

From the Planning section:

  1. Start by opening the test plan containing the test execution record that you want to open the log for: point to the Planning icon and click All Test Plans (or My Test Plans). A list of test plan(s) is then displayed on the screen.
  2. Click on the test plan's name containing the targeted failed test case(s). At this point, you can see the test plan displayed on the screen.
  3. In the Table of Contents section, click on Test Cases. It will bring you to the list of test cases associated with that test plan.
  4. Click on the test case for which you want to investigate the failure for. At this point, the test case is displayed on the screen.
  5. In the Table of Contents section, click Test Execution Records. This action will bring you to the list of executions for that item.
  6. In the Test Execution Records section, click the Status link attached to one of the executions. By clicking on that link, you will be directed to the result history for that specific item. The status that is displayed in that column is related to the last test run. Thus, you may have to click on a status link different than Failed to access the result history for that specific item.
  7. You can now click log icon of the targeted failed execution.

The second way of accessing a test log is after a test execution, as RQM brings you by default to the log result of that run.

From the Execution section:

  1. Start by opening the Test Execution Records list: point to the Execution icon and then click All Test Execution Records (or My Test Execution Records).
  2. Please refer to steps 6 and 7 of the following above section.

After following one of those two paths, you will now have a displayed test log containing specific information, such as the test verdict, test environment and test result details.

2. Investigate each Failure

You now need to investigate each failure. It is important to look for similarities in the failures (Did they happened under the same test configuration? Did they have similar symptoms? Etc ...). It's important to understand that a single failure may have cause several subsequent failures (maybe all of the subsequent failures!). Make sure to investigate the failure(s) in a sequential order: from the beginning to the end of the execution.

Depending on which type of test script is attached to the test (except for manual script type), you will have to open the external tool log file by clicking on the log file link in the Result Details section.

For more information about how to interpret the different external log types, please refer to the online help for their respective product.

Example of investigating a test failure coming from a manual test script

For a test implemented with a manual script, you can start the investigation by looking at the Result Details section at the bottom of the test log:

  1. Look for Failed status in the steps and verification points listing section.
  2. For each of those Failed verification point(s), in the Result Details section, look for the comment icon and the attachment icon. Those icons will be your main source of information to investigate the failure. If you point on those icons, any comment or file attachment that was created during the execution for a specific step will be displayed. Those comment(s) and attachment(s) may contain a description of the failure with possible screenshots of the problem. Use this information to diagnose the problem.
  3. For any of the found failure(s), you may enter a defect, from RQM, in a supported defect tracking tool.