Results / Test Reports and Metrics Overview

controls

1. Active Build

The only report which is effected by the "Active Build" control is the "Metrics of Active Build" report.

2. Report format

All test reports (except charts) can be generated in 3 ways :
  • 1.normal - report is displayed in web page
  • 2.MS Excel - report exported to Microsoft Excel
  • 3.HTML Email - report is emailed to user's email address
  • reports

    There are currently 10 seperate reports to choose from, their purpose and function are explained below.

    1)Metrics of Active Build

    This report shows the detailed results for a particular build defined by the "Active Build" control. It only takes into account executions performed on the selected build. The following tables are displayed:

    Results by top level Test Suites

    Lists the results of each top level suite. Total cases, passed, failed, blocked, not run, and percent completed are listed. A "completed" test case is one that has been marked pass, fail, or block. Results for top level suites include all children suites.

    Results by Test Suite

    Lists metrics for all suites in test plan, not just top level suites.

    Results By Keyword

    Lists all keywords that are assigned to cases in the current test plan, and the results associated with them.

    2)General Test Plan Metrics

    This page shows you only the most current status of a test plan by test suite, owner, and keyword. The most "current status" is determined by the most recent build test cases were executed on. For instance, if a test case was executed over multiple builds, only the latest result is taken into account.

    "Last Test Result" is a concept used in many reports, and is determined as follows :
  • 1)The order in which builds are added to a Test Plan determines which build is most recent. The results from the most recent build will take precendence over older builds. For example, if you mark a test as "fail" in build 1, and mark it as "pass" in build 2, it's latest result will be "pass".
  • 2)If a test case is executed mulitple times on the same build, the most recent execution will take precedence. For example, if build 3 is released to your team and tester 1 marks it as "pass" at 2PM, and tester 2 marks it as "fail" at 3PM - it will appear as "fail".
  • 3)Test cases listed as "not run" against a build are not taken into account. For example, if you mark a case as "pass" in build 1, and don't execute it in build 2, it's last result will be considered as "pass".

  • The following tables are displayed :

    Results by top level Test Suites

    Lists the results of each top level suite. Total cases, passed, failed, blocked, not run, and percent completed are listed. A "completed" test case is one that has been marked pass, fail, or block. Results for top level suites include all children suites.

    Results By Keyword

    Lists all keywords that are assigned to cases in the current test plan, and the results associated with them.

    Results by owner

    Lists each owner that has test cases assigned to them in the current test plan. Test cases which are not assigned are tallied under the "unassigned" heading.

    3.)The Overall Build Status

    Lists the execution results for every build. For each build, the total test cases, total pass, % pass, total fail, % fail, blocked, % blocked, not run, %not run. If a test case has been executed twice on the same build, the most recent execution will be taken into account.

    4.)Query Metrics

    This report consists of a query form page, and a query results page which contains the queried data.

    Query Form Page:

    User is presented with a query page with 4 controls. Each control is set to a default which maximizes the number of test cases and builds the query should be performed against. Altering the controls allows the user to filter the results and generate specific reports for specific owner, keyword, suite, and build combinations.

    keyword

    0->1 keywords can be selected. By default - no keyword is selected. If a keyword is not selected, then all test cases will be considered regardless of keyword assignments. Keywords are assigned in the test specification or Keyword Management pages. Keywords assigned to test cases span all test plans, and span across all versions of a test case. If you are interested in the results for a specific keyword you would alter this control.

    owner

    0->1 owners can be selected. By default - no owner is selected. If an owner is not selected, then all test cases will be considered regardless of owner assignment. Currently there is no functionality to search for "unassigned" test cases. Ownership is assigned through the "Assign Test Case execution" page, and is done on a per test plan basis. If you are interested in the work done by a specific tester you would alter this control.

    top level suite

    0-> n top level suites can be selected. By default - all suites are selected. Only suites that are selected will be queried for result metrics. If you are only intested in the results for a specific suite you would alter this control.

    Builds

    1->n builds can be selected. By default - all builds are selected. Only executions performed on builds you select will be taken into account when producing metrics. For example - if you wanted to see how many test cases were executed on the last 3 builds - you would alter this control. Keyword, owner, and top level suite selections will dictate the number of test cases from your test plan are used to computate per suite and per test plan metrics. For example, if you select owner = "Greg", Keyword="Priority 1", and all available test suites - only Priority 1 test cases assigned to Greg will be considered. The "# of Test Cases" totals you will see on the report will be influenced by these 3 controls. Build selections will influence if a case is considered "pass", "fail", "blocked", or "not run". Please refer to "Last Test Result" rules as they appear above.

    Press the "submit" button to proceed with the query and display the output page.

    Query Report Page:

    The report page will display:
  • 1.the query parameters used to create report
  • 2. totals for the entire test plan
  • 3.a per suite breakdown of totals (sum / pass / fail / blocked / not run) and all executions performed on that suite. If a test case has been executed more than once on multiple builds - all executions will be displayed that were recorded against the selected builds. However, the summary for that suite will only include the "Last Test Result" for the selected builds.
  • 5-7.)Blocked, Failed, and Not Run Test Case Reports

    These reports show all of the currently blocked, failing, or not run test cases. "Last test Result" logic (which is described above under General Test Plan Metrics) is again employed to determine if a test case should be considered blocked, failed, or not run. Blocked and failed test case reports will display the associated bugs if the user is using an integrated bug tracking system.

    8.)Test Report

    View status of every test case on every build. If a test case was executed multiple times on the same build - the most recent execution result will be used. In this report, "?" signifies a test case that was not run on a build. It is recommened to export this report to Excel format for easier browsing if a large data set is being used.

    9.)Charts

    This report page requires your browser have a flash plugin. It uses flash technology provided by http://www.maani.us to display results in a graphical format. "Last test Result" logic is used for all 4 charts that you will see. The graphs are animated to help the user visualize the metrics from the current test plan.

    The four charts provide are :

    1.Pie chart of overall pass / fail / blocked / and not run test cases

    2.Bar chart of Results by Keyword

    3.Bar chart of Results By Owner

    4.Bar chart of Results By Top Level Suite

    The bars in the bar charts are colored such that the user can identify the approximate number of pass, fail, blocked, and not run cases.

    10.)Total Bugs For Each Test Case

    This report shows each test case with all of the bugs filed against it for the entire project. This report is only available if a Bug Tracking System is connected.