Test Automation From the Ground Up By Vasily Shishkin and Yaron Kottler

Introduction

Test automation promises many attractive benefits such as running multiple tests overnight at a click of a button, eliminating mindless work, increasing test coverage, and reducing the cost of testing. So the question is, how to get started?

Deliberation of Merit

The first order of business then is to be sure that test automation is the right decision in your particular case. The initial automation effort can be expensive, especially if the systems have not been designed with test automation in mind (very few of them are…). Before considering automation make sure you can positively respond to ALL the following per-requisites:

• The systems or applications are relatively stable with no significant redesigns expected in the foreseeable future.
• The organization has a solid test process in place.
• The Organization has the right type of person or team in place.
• The Organization has a defined scope for an initial automation effort.
• There is an understanding by senior management that test automation is an investment, not a short term solution.
• There is a return on investment (ROI) analysis demonstrating at least a 50% ROI for a set period.

For an effort to be considered for test automation, it should result in a positive ROI. Since the expected lifespan of testing a system is not always known or can be hard to predict, an attempt at a comprehensive ROI calculation at an early stage will likely miscalculate an actual ROI. A short term ROI estimate (between one to three years) is a safer and easier calculation to make, typically utilizing at least some of the following factors:

• Length of time to run a predefined test set manually.
• Cost and availability of support resources necessary for manual test execution.
• Number of times such test sets are expected to run during the agreed upon period.
• Expected effort for establishing automation infrastructure (mapping objects, reusable functions…).
• Expected effort for establishing test automation coverage utilizing above infrastructure.
• Expected maintenance effort factor between each version or cycles.
• Expected effort for executing automated tests and investigating results.
• The value of enabling manual testers to focus mostly on testing new functionality.
• Estimated value of shortening a test cycle by X days.
• Estimated value of increased test coverage consistency.
• Estimated value of knowledge retention.
• Estimated value of executing additional test types which couldn’t be executed earlier.

A ROI calculation is not enough though; as indicated above you must also ensure that you have a solid test process in place including things like scope, strategy, test types and test levels. Simply put, automated garbage is still garbage, so if you haven’t been able to positively answer the entire automation per-requisites list above, start there. Also, keep in mind that test automation will almost never replace the first round of manual testing of new functionality, but is rather targeted at reducing the cost and risks associated with regression testing. There are cases in which a progressive automation approach enables utilizing test automation for both regression testing and testing of new functionality but it is not recommended for organizations just getting started with test automation.
Once your foundation and preliminary ROI calculations are in place, it is time to run a test automation proof of concept (POC) project.

Test Automaton Proof of Concept (POC)

Phase	Duration (small project)	Customer Involvement
Determining Scenarios and Learning the Application	1 Day	2-3 Hours
Writing Up the Test Plan	2-3 Days (depends on revisions)	1-4 Hours (Depends on the infrastructure of the customer)
Scripting	1-3 Weeks	5 Hours – to answer whatever questions come up during scripting
Testing	1-3 Weeks (depends on issues found)	Best Case scenario: (no problems, testers have access to server hardware) – 3 Hours Worst Case scenario: (lots of problems, server hardware has to be monitored by customer): practically same as tester
Final Report	1 Day (compile and Present)	1-2 Hours for the presentation

A POC project should start with defining the immediate goals it should meet. While doing so, keep in mind that the POC’s main purpose is not actual automation implementation, but is rather intended to help determine what kind of methods and resources would be required for a successful automation implementation. It is recommended that the POC project take no longer than three to four weeks. It should have a balance between the need to discover the main automation challenges and demonstrating feasibility with the need to support relatively quick decision making. In addition to organization-specific goals, your list of POC goals should include at least some of the following:

• Verifying test automation feasibility.
• Exposing technical blocks and challenges.
• Experimenting with potential solutions and workarounds.
• Further refining cost estimates and ROI calculations.
• Experimenting with a number of test automation tools
• Finalizing a tool selection or at least supporting a selection.
• Experimenting with a number of test automation approaches (KDT, DDT, TDD, BDD, Progressive automation…) and finalizing an approach or supporting a selection.
• Helping establish test automation context and an appropriate state of mind at your organization.
• Defining a detailed 6 month implementation plan with 30, 60, 90 and 180 day targets.

Once your POC goals are in place, start the POC project by educating yourself on available automation approaches and their advantages. Selection of test automation approaches should include reviewing at least some of the following factors:

• Proficiency of current personnel with each approach and technical tools associated with it.
• Cost of training for each approach.
• Depth of knowledge of the system under test required for each approach.
• Number of personnel who have BOTH the skills required for the approach and the knowledge of the system under test.

When an approach is selected, next consider the tools you will need. Some of the important factors are:

• Compatibility with the technologies present in the system under test.
• Compatibility with the technologies planned to be utilized.
• Compatibility with the test management systems in place.
• Market prevalence of the tool ( a wider user base facilitates finding experienced users, support groups, and other network effect factors).
• Quality of tool vendor support.
• Tool usability.
• Tool features that simplify script maintenance.
• Cost of the tool.

Once the tools have been chosen, the next step in a POC project is to establish a detailed Test Automation Implementation Plan. The plan should include a list of reusable business and utility functions to be tested, as well as a detailed automation breakdown. The plan should allow for the distribution of workload, estimation of the timeline, and tracking of progress.
The last topic to consider is technical guidelines, such as: object mapping structures, creation and maintenance processes, and coding conventions.
When all of the preliminary work is finished, it is time to actually implement the selected scenarios on the tools of choice. From these scenarios, a list of encountered and predicted challenges should be generated and solutions for said challenges evaluated. If the challenges prove to be insurmountable, the POC conclusion is that there is no test automation feasibility. Otherwise, the information gathered should enable meeting the POC goals and enable more accurate recalculation of the ROI. If a revised ROI is unacceptable, another POC with an alternative approach or toolset can be performed. Alternatively, this could indicate that test automation might not be a good idea for this system.

Test automation maintenance

If the POC is completed successfully, and the decision to pursue automation is reached, the concern of automation maintenance should become the leading factor throughout your test automation implementation as it will be the major factor in delivering positive ROI. To insure efficiencies, it is important to build a framework that supports minimal and easy maintenance.
A good maintenance process follows this general outline:

1. Review the list of known changes in the system and conduct a gap analysis.
2. Create an action list of things to update such as:
   • New objects to map.
   • Existing object maps to update.
   • Object maps to remove.
   • API calls to revise.
   • Changes in system logic to accommodate.
3. Inform the personnel of the workflow and data set changes.
4. Upgrade scripts and infrastructure as necessary.
5. Conduct a manual sanity test to verify that the environment is operational.
6. Run a full suite of tests and investigate failures to understand if issues are system related or automation related.
7. In case issues are automation related, fix the automation as needed and add such issues to future gap analysis.

Once maintenance has been completed successfully, automation infrastructure and automation scenarios should be developed to support additional test coverage.

Summary

In this paper, we outline the basic steps of getting started with test automation at your organization. While each organization has different needs and technologies we find that just about every organization can benefit from utilizing test automation and that the above step by step guide will provide a good framework for getting started.

Examples of good candidates

One last thing to keep in mind. When looking at test automation, the following list of areas typically benefit from test automation:

• Regression tests that are repeated often – a sanity test is a good example of that.
• Tests of stable functionality that is not expected to change much.
• Tests that can be completed automatically with no human intervention.
• Tests which are expensive to run manually.
• Tests which require multiple user roles to execute.

Contact the Authors

Yaron Kottler: yaronk@qualitestgroup.com
Vasily Shishkin: vasilys@qualitestgroup.com