What is a Test Automation Report and Why is it important?
Test automation enables engineers to validate and test features and bug fixes at a faster rate. Test automation is primarily useful for regression testing. But is it enough to develop the automation scripts alone? Test automation is incomplete without reporting. Test Automation reports help QA engineers to analyse the test results in a cumulative manner, specifically failures and validate if they are actual issues or flaky tests. In this blog, the focus will be on automation reports and dashboards and the key metrics that will help simplify and expedite the report analysis.
Key components in a test automation report
Summary
A high level summary of the complete test suite execution providing details like -
- No of tests executed
- Failed and passed tests count and percentage
- Test execution time
- Test execution environment details
- Browsers and devices
Test Case List and Filters
Every report will contain a list of test cases executed along with the status - Pass/Fail/Skipped. Additionally there will also be filters available for filtering the test cases based on status, tags etc.
Test Steps
Each test case will contain details like browser, execution time and resolution. Each test case will also contain the test steps in detail and will clearly depict at which step the test case has failed for failed test cases. In some reports, the execution time for each test step will also be mentioned. This is greatly useful to debug tests that take a longer time to execute and which can be used to eventually reduce the test execution time and overall execution time of the test automation suite.
Screenshots and Screen Recordings
For failed test cases, a proof in the form of screenshots or screen recordings should be made available which will further enhance the failed test case analysis and help debug the failures and identify the root cause of the failure.
Reasons for Failed Tests
This is the most important parameter in a test automation report. The exceptions for the failed tests help identify the reason for the test failure and there-by help detect bugs in the application or feature.
Network Logs and Console Logs for UI Tests
This is another useful parameter in a test automation report that helps debug test failures and also detect any performance issues. Network and console logs can also be used to identify the root cause of the failure and will help developers fix the code faster. This is primarily useful when the bugs are not easily reproducible, in which case such network and console traces will help narrow down the root cause of the bug.
Key Data to Track in a Test Automation Report
Here are some of the key parameters that can be used for analysis of test automation jobs -
- No of tests - Total no of tests executed within a single automation job
- No of jobs - Total no of automation jobs added and executed
- Automation job status - Automation job status can be any one of Pass/Fail/Running/Cancelled.
- Passed tests - Can be count or percentage
- Failed tests - Can be count or percentage
- Skipped tests - Can be count or percentage
- Exceptions - Reasons for test case failures. Can be used to group failed test cases based on similar exceptions, thereby reducing the overall time spent to analyse individual tests that have failed because of the same reasons.
- Execution time - Can be computed for independent test steps, test cases, entire job
- Test case stability - helps understand the overall stability of the test case across different jobs and environments. This will help fix flaky tests.
- Environments - Depicts which environments the tests were executed across different automation jobs. Eg) DevBox, Beta, Sandbox, Production etc
- Tags - Help group tests that belong to the same functionalities
- Priority - Help group tests based on priorities
Building Analysis using Test Automation Results
Using the above mentioned parameters, some of the key metrics that can be built are -
- No of tests executed over a period of time and depiction of the failure rate in each execution
- Categorise test cases and status by tags and features and priorities
- Test case stability rate
- Average Test execution time
- Group test cases by exceptions
- Total number of test cases added over a period of time
- Test case failure and pass rate trends
- Clusters to help identify which code branch caused which test to fail when tests are executed across multiple branches.
Open Source Test Automation Reports
Here are a few sample test automation reports being widely used across teams -
TestNG Reports
TestNG is an open source framework for test automation. TestNG annotations are used along with Selenium automation in Java. TestNG generates html and xml based reports that cover basic details about the tests executed. Here’s a good resource to help understand more about testng reporting: https://www.tutorialspoint.com/testng/testng_html_xml_report.htm.
Allure reports
Allure reports is an open source test automation reporting tool. It can be integrated with open source test automation tools like Selenium and playwright. One of the key advantages of allure reports is that it helps analyse historical data as well. Learn more about allure reports: https://allurereport.org/
Report Portal
Report portal is a test automation analytics dashboard - https://reportportal.io/. It provides real time analysis and also helps in data visualizations. One of the major advantages of ReportPortal is the usage of machine learning algorithms that help identify patterns and detect potential issues.
Extent Reports
Extent reports is an open source test automation reporting tool that integrates well with open source tools like TestNG - https://extentreports.com/.
It is widely used in Selenium Test Automation frameworks.
DevAssure's Test Automation Reports
DevAssure is a no code test automation tool that has very powerful reporting and customizable analytical dashboards available. With DevAssure static html reports will be generated after every test suite execution while executing locally and on the CI machines. The tool also provides test results data as json files, which enables seamless integration into tools like PowerBI to create test automation dashboards.
Some of key aspects of a DevAssure test automation report include
- Pass and failure rate
- Test execution time at job, independent test and independent test steps level
- Test Failure reasons
- Screenshots and accessibility reports for failed tests
- Visual validation data
- Details of test steps executed and validations made
- Ability to add custom logs
- Other test execution details like branch, environment and job id.
- API responses
- Network and console logs
- Browsers and devices
Here's a Sample Report from DevAssure -
Learn more about DevAssure's test automation report.
Reports across multiple jobs or executions will be available for tracking at a single place in the DevAssure web application. When tests are executed on the CI machine, all the tests being executed can be tracked in a single place irrespective of the devops tool being used.
The reports page provides the ability to filter test execution results based on jobs, environments and branches.
DevAssure’s Analytics Dashboards for Test Automation
Having a reporting tool alone will not be enough for a test automation framework. While reports provide details on the test executions, dashboards help with the analysis of test executions over a period of time.
Test Analytics dashboards are important to
- Understand failure patterns and predict issues and root causes
- Understand failure trends and fix flaky tests
- Understand test executions and remove stale tests
- Faster report validations for regression test automation executions
DevAssure's Analytics Dashboard for Test Automation helps build these analyses using customizable visualizations.
To experience DevAssure's reporting and dashboard, sign up for a free trial.
Furthermore, to have a personalized demo on how the DevAssure test automation platform can leverage your Organization's testing capabilities, please click the button to request a demo session with our team of experts.