Choosing the Right Automation Testing Strategy: Do's and Don’ts
Choosing the right testing strategy might be truly challenging. Find out how to avoid risks, manage the development team, and save your time and money – all in this article!
Are you planning to develop your own software product, yet you lack the necessary experience? We will help you avoid a number of time-consuming and costly traps by choosing the right testing strategy. In this article, we will outline the criteria on how to choose the right testing suite, name the worst testing strategies, and provide recommendations.
What is the test automation pyramid?
Why test software? When you make a software product and change some of it, e.g. features or code, you need to make sure that the whole system’s behavior stays the same. To do this, you must validate the system’s features once more by running tests with a specific outcome. Currently, such tests can be easily automated.
The test automation pyramid (originally coined by Mike Cohn in his book “Succeeding with Agile”) depicts three levels of automating tests. Basically, the pyramid says that unit tests at the bottom of the pyramid are quick, simple, and cheap, while UI tests at the top are slow, complicated, and expensive. Developers use all three levels of automating tests depending on their particular needs.
Unit test level
Unit tests at the bottom of the pyramid are applied to test small pieces of code. For example, when you test a calculator program, you may test one single function – addition or multiplication with a unit test. A unit test is quick to execute and, therefore, developers receive immediate feedback about the performance of a particular feature or code.
As unit tests are simple and specific, it's easy to detect trouble with code at an early stage. Some teams even write unit tests before the code in order to improve the overall coding design. At the same time, unit tests are rather useful for developers rather than product owners, as they do not carry any business value.
Service test level
Service tests in the middle of the pyramid are applied to test different services without involving the user interface (UI). With the help of service tests, you may test various components and not encounter all the complications related to the user interface. Following the example of a calculator program, a service test would test the simultaneous performance of two functions – addition and multiplication.
Service tests are slower compared with unit tests and, obviously, more complicated to perform. Still, service tests are faster and way more reliable compared with tests at the UI level. At the same time, with the help of component testing, it's possible to validate the business value of a product.
UI test level
UI tests at the top of the pyramid are applied to test the correct work of the user interface. Referring to the above-mentioned example of a calculator program, UI tests are used when a product owner decides to change the layout of a calculator.
Overall, UI tests are complicated and slow to maintain. Typically, UI tests are performed with test automation tools such as Selenium, which go through multiple layers of a browser or a database. This makes the testing operations incredibly slow. Indeed, it takes too many factors to consider when you run a test emulating a screen touch. In addition, any time the user interface changes, UI tests require maintenance.
In addition, UI tests are hard to write due to browsers being asynchronous and parts of web pages loading at a different speed. Therefore, UI tests remain the most expensive among the rest of the pyramid and they require a careful approach.
How to choose the right testing combination?
Do you have a software product that you want to test? Here are a few tips to determine the right testing combination. Consider the following criteria: speed, cost, reliability, and focus.
A single unit test runs about 0.01 seconds, while a single UI test may typically run 3 to 15 minutes. Therefore, you should consider how much time you are able to allocate to slow tests. Many developers run such tests only prior to their products’ release. In addition, running slow tests is rather pointless in case the software hasn’t changed. Moreover, be prepared to stop slow tests in case they disrupt the overall performance.
Apart from that, you should consider how much time all your tests run together. Ideally, you would want a test suite running 10–15 minutes. If this takes longer and affects the overall productivity, think about changing the testing strategy. In addition, be prepared to change the testing strategy if your testing suite fails to detect the issue fast enough.
Speaking of cost, you should consider the budget allocation for tests’ building and maintenance. Going back to the test automation pyramid, unit tests are simple, cheap, and fast, so you need a lot of them to test the product. Frequently, unit tests are already included in the software development cost. Service tests are equally important to make sure that independent units run together properly and bring the essential business value; therefore, you typically need lots of those as well.
UI tests are the slowest and the most expensive, so you should be careful prior to planning such tests. UI tests are relevant for already deployed products or for products that are unlikely to change in the upcoming years. Otherwise, they will pointlessly consume your budget. In fact, you only need a handful of such tests throughout the development process.
Think about your tests’ reliability in terms of their trustworthiness. Can you trust the selected tests in case the system is broken? According to the test pyramid, unit tests are the most reliable, while UI tests, on the contrary, are the least reliable. At the same time, unit tests cannot be viewed as a solution to everything as they often change due to their classes’ change. Service tests do not change that often, yet remain more reliable compared with UI tests. Using them might be the right option in many cases. Still, selecting the right testing suite in terms of reliability depends on a particular situation.
In fact, tests are not being made for the sake of testing: each test should be focused and should respond to a particular problem. In addition, not only should testing quickly detect trouble, but it should also provide detailed feedback for developers. The more correct the feedback is, the faster developers will fix the problem.
It's important to acknowledge that different levels of testing are suitable in different situations. Any of them might fail in case they are applied inappropriately. Your major challenge here is to precisely identify the problem and apply a correct and targeted combination of tests.
The worst test automation strategies
Here we go with the most interesting part: a couple of the worst test automation strategies you should avoid at all costs. These strategies are known as the "ice cream cone" and "testing hourglass". Both strategies generally presume the lack of proper communication between QAs and developers.
"Ice cream cone"
An "ice cream cone" testing strategy presumes rather few unit tests, yet a substantial number of UI tests. Manual tests are also most probably present. In the case of "ice cream cone", tests are usually inefficient and long-running. With a high number of UI tests, it will take longer for developers to receive proper feedback on the changes that they made. In many cases, "ice cream cone" testing strategy appears when QAs have not agreed upon the testing suite with developers.
"Testing hourglass" strategy presumes lots of unit and UI tests, yet a low number of service tests. Although "testing hourglass" is essentially better than "ice cream cone", it has its own shortcomings. First, a low number of service tests fail to validate an overall business logic properly. Consequently, developers have to do double work. In addition, the "testing hourglass" strategy might drain your budget due to the use of expensive UI tests instead of cheaper service tests.
How to Avoid Mistakes?
How to avoid selecting inappropriate test automation strategies?
First, try following the test automation pyramid: unit test the majority of the features, pay attention to service tests, and carefully plan UI tests only if they are truly needed.
Second, align your development and testing goals and make sure your QAs have proper communication with developers. After all, quality is their equal responsibility.
Third, remember that there is no magic recipe for choosing the right testing strategy. Multiple tests should be applied depending on each particular situation. The best solution is to make a proper analysis of your testing case and target all the challenges in detail.
Choosing the right testing strategy might be challenging. It's easy to fall into numerous traps resulting in a loss of time and budget. The test automation pyramid helps you avoid numerous mistakes related to the selection of an appropriate testing suite. The pyramid classifies automation tests according to three levels: from cheap and simple to slow and expensive. Based on this classification, you may select the right combination of automation tests.
In addition, take into consideration testing speed, costs, reliability, and focus. Always remember that a proper testing suite should provide targeted and detailed feedback.
Get weekly updates on the newest design stories, case studies and tips right in your mailbox.
With this article, we are launching a series of stories casting light on the ups and downs behind StickerBox. Spoiler: with all the downs, we made it!
Selenium is truly amazing, and there’s a video in this article proving that. Join us to learn about how you can save time and money (and make your testers happier!) on your next project.
Based on real cases: check out our tips on how to stay focused, confident, and cooperative when working with your engineering team