Choosing the Right Automation Testing Strategy: Do's and Don’ts

Are you planning to develop your own software product, yet you lack the necessary experience? We will help you avoid a number of time-consuming and costly traps by choosing the right software test automation strategy. In this article, we will outline the criteria on how to choose the right testing suite, name the worst testing strategies, and provide recommendations.

What is the test automation?

In a nutshell, testing automation is a software testing option that implies no manual intervention. To perform tests like that, QA engineers write special scripts that go through code with the help of an automation tool. Together, they test and compare the result they got with the forecasted expectations.

Advantages of testing automation

From the business perspective, automated tests have several advantages:

Cost-efficient

In software testing, there are operations that manual tests simply cannot perform. Especially if it’s a repetitive task that has to be completed over and over again. Automation saves the day here. Once you write the script, it can be used multiple times without any additional costs. Even though the first adoption may be expensive, it will cost less in the long run.

Faster delivery

Automated testing runs faster than the manual one. Instead of waiting for days and weeks for a series of manual tests to complete, you will need only a few hours. It will positively affect the overall delivery time of the project.

Instant feedback

Automated testing provides the QA team with immediate feedback on the results. When there is a bug in the code, they will find it out and fix it as soon as possible. It’s extremely important when the product is already released since updates and fixes will be made faster.

Check out the article to read about more benefits of automated QA.

What are the types of automated tests?

There are a lot of ways to group automated testing methods but the most popular would be to divide them depending on what they check: functional and non-functional.

Functional tests are meant to check if the code meets all the business requirements. These requirements are set even before the development starts, so it’s good for QA specialists to be present at this stage. That way, they can create perfect scripts to test the functions and confirm that they function properly. Such testing includes:

• Acceptance tests
• Integration tests
• Smoke tests
• Unit tests

Non-functional tests are not connected to functions directly but scan the general performance of the app. They test metrics like load time, overall speed, usability, and survivability. The predicted results of these tests should be clear so when testing is finished you will see if the app needs improvement. Non-functional tests consist of:

• Security tests
• Performance tests
• Usability tests
• And many more

Automated testing strategy pyramid

Why test software? When you make a software product and change some of it, e.g. features or code, you need to make sure that the whole system’s behavior stays the same. To do this, you must validate the system’s features once more by running tests with a specific outcome. Currently, such tests can be easily automated.

The test automation pyramid (originally coined by Mike Cohn in his book "Succeeding with Agile") depicts three levels of automating tests. Basically, the pyramid says that unit tests at the bottom of the pyramid are quick, simple, and cheap, while UI tests at the top are slow, complicated, and expensive. Developers use all three levels of automating tests depending on their particular needs.

Unit test level

Unit tests at the bottom of the pyramid are applied to test small pieces of code. For example, when you test a calculator program, you may test one single function—addition or multiplication with a unit test. A unit test is quick to execute and, therefore, developers receive immediate feedback about the performance of a particular feature or code.

As unit tests are simple and specific, it's easy to detect trouble with code at an early stage. Some teams even write unit tests before the code in order to improve the overall coding design. At the same time, unit tests are rather useful for developers rather than product owners, as they do not carry any business value.

Service test level

Service tests in the middle of the pyramid are applied to test different services without involving the user interface (UI). With the help of service tests, you may test various components and not encounter all the complications related to the user interface. Following the example of a calculator program, a service test would test the simultaneous performance of two functions—addition and multiplication.

Service tests are slower compared with unit tests and, obviously, more complicated to perform. Still, service tests are faster and way more reliable compared with tests at the UI level. At the same time, with the help of component testing, it's possible to validate the business value of a product.

UI test level

UI tests at the top of the pyramid are applied to test the correct work of the user interface. Referring to the above-mentioned example of a calculator program, UI tests are used when a product owner decides to change the layout of a calculator.

Overall, UI tests are complicated and slow to maintain. Typically, UI tests are performed with test automation tools such as Selenium, which go through multiple layers of a browser or a database. This makes the testing operations incredibly slow. Indeed, it takes too many factors to consider when you run a test emulating a screen touch. In addition, any time the user interface changes, UI tests require maintenance.

In addition, UI tests are hard to write due to browsers being asynchronous and parts of web pages loading at a different speed. Therefore, UI tests remain the most expensive among the rest of the pyramid and they require a careful approach.

Features of automation testing

Do you have a software product that you want to test? Here are a few tips to determine the right testing combination. Consider the following criteria: speed, cost, reliability, and focus.

Speed

A single unit test runs about 0.01 seconds, while a single UI test may typically run 3 to 15 minutes. Therefore, you should consider how much time you are able to allocate to slow tests. Many developers run such tests only prior to their products’ release. In addition, running slow tests is rather pointless in case the software hasn’t changed. Moreover, be prepared to stop slow tests in case they disrupt the overall performance.

Apart from that, you should consider how much time all your tests run together. Ideally, you would want a test suite running 10–15 minutes. If this takes longer and affects the overall productivity, think about changing the testing strategy. In addition, be prepared to change the testing strategy if your testing suite fails to detect the issue fast enough.

Cost

Speaking of cost, you should consider the budget allocation for tests’ building and maintenance. Going back to the test automation pyramid, unit tests are simple, cheap, and fast, so you need a lot of them to test the product. Frequently, unit tests are already included in the software development cost. Service tests are equally important to make sure that independent units run together properly and bring the essential business value; therefore, you typically need lots of those as well.

UI tests are the slowest and the most expensive, so you should be careful before planning such tests. UI tests are relevant for already deployed products or for products that are unlikely to change in the upcoming years. Otherwise, they will pointlessly consume your budget. In fact, you only need a handful of such tests throughout the development process.

Reliability

Think about your tests’ reliability in terms of their trustworthiness. Can you trust the selected tests in case the system is broken? According to the test pyramid, unit tests are the most reliable, while UI tests, on the contrary, are the least reliable. At the same time, unit tests cannot be viewed as a solution to everything as they often change due to their classes’ change. Service tests do not change that often, yet remain more reliable compared with UI tests. Using them might be the right option in many cases. Still, selecting the right testing suite in terms of reliability depends on a particular situation.

Focus

In fact, tests are not being made for the sake of testing: each test should be focused and should respond to a particular problem. In addition, not only should testing quickly detect trouble, but it should also provide detailed feedback for developers. The more correct the feedback is, the faster developers will fix the problem.

It's important to acknowledge that different levels of testing are suitable in different situations. Any of them might fail in case they are applied inappropriately. Your major challenge here is to precisely identify the problem and apply a correct and targeted combination of tests.

Automation testing steps

Automated testing doesn’t come at once. There are several automation testing process steps that will structure the workflow and minimize the number of possible pitfalls.

Step 1: Choose the cases

Since not every single case is suitable for automation, you should pick a sample of cases for it. The fittest cases are those that involve a lot of input data, apply for various environments, are repetitive, and carry a lot of business value.

Step 2: Hire an expert

If you have your own QA specialists, you can skip this stage. But if you need a testing engineer with the relevant experience, pay attention to it. Specify the requirements for your project and be sure to hire someone who can work with multiple tools.

Step 3: Pick the testing tools

Testing tools should correspond with the complexity of testing cases. The list of proper testing tools is described in the next block.

Step 4: Write scripts

When everything is done, it’s time to write testing scripts. To make sure that scripts are perfect, break the scenarios down to simple flows, stick to one purpose, and try to steer clear of if-statements.

Step 5: Maintain scripts

Building a test suite with low maintenance needs is quite a task for QA teams. There is a chance that this goal can lead to the underuse of scripts. To avoid it, opt for small test cases and simple scenarios and limit the environments to those with high usage rates.

Best automation testing tools

There are plenty of solutions to provide you with testing automation. Here are the most used and trustworthy tools:

• Selenium
• Kobiton
• Cucumber
• Eggplant
• SoapUI
• TestComplete
• ZeuZ
• Telerik Test Studio

Most of these tools have free licenses and allow you to create a comfortable testing environment.

The worst test automation strategies

Here we go with the most interesting part: a couple of the worst test automation strategies you should avoid at all costs. These strategies are known as the "ice cream cone" and "testing hourglass". Both strategies generally presume the lack of proper communication between QAs and developers.

"Ice cream cone"

An "ice cream cone" testing strategy presumes rather few unit tests, yet a substantial number of UI tests. Manual tests are also most probably present. In the case of "ice cream cone", tests are usually inefficient and long-running. With a high number of UI tests, it will take longer for developers to receive proper feedback on the changes that they made. In many cases, "ice cream cone" testing strategy appears when QAs have not agreed upon the testing suite with developers.

"Testing hourglass"

"Testing hourglass" strategy presumes lots of unit and UI tests, yet a low number of service tests. Although "testing hourglass" is essentially better than "ice cream cone", it has its own shortcomings. First, a low number of service tests fail to validate an overall business logic properly. Consequently, developers have to do double work. In addition, the "testing hourglass" strategy might drain your budget due to the use of expensive UI tests instead of cheaper service tests.

How to avoid mistakes?

How to avoid selecting inappropriate test automation strategies?

First, try following the test automation pyramid: Unit test the majority of the features, pay attention to service tests, and carefully plan UI tests only if they are truly needed.

Second, align your development and testing goals and make sure your QAs have proper communication with developers. After all, quality is their equal responsibility.

Third, remember that there is no magic recipe for choosing the right testing strategy. Multiple tests should be applied depending on each particular situation. The best solution is to make a proper analysis of your testing case and target all the challenges in detail.

Conclusion

Choosing the right testing strategy might be challenging. It's easy to fall into numerous traps resulting in a loss of time and budget. The test automation pyramid helps you avoid numerous mistakes related to the selection of an appropriate testing suite. The pyramid classifies automation tests according to three levels: from cheap and simple to slow and expensive. Based on this classification, you may select the right combination of automation tests.

In addition, take into consideration testing speed, costs, reliability, and focus. Always remember that a proper testing suite should provide targeted and detailed feedback.