How testing automation can improve software delivery

“Testing leads to failure and failure leads to understanding”– Burt Rutan

GitLab’s worldwide 2021 survey of DevOps teams revealed that inefficient testing was the main reason for software release delays in 2021. More and more teams have fully automated software tests, but testing still remains a big bottleneck in software delivery.

Understandably, many businesses haven’t automated their software testing workflows yet. Automating manual tests is an extremely challenging process that requires careful planning, standardizing code, and designing repeatable test cases. Some businesses might not see any value in switching to automated testing.

Therefore, automated software testing can be extremely beneficial if done right. Testing automation can help teams build higher-quality products that better meet the needs of end users (Khan, et al., 2020). And more importantly, automated testing can help businesses achieve more in less time.

But how exactly can testing automation help the team achieve these outcomes? To answer this question, we need to understand what testing automation is.

What is Testing Automation?

Testing automation is using specialized software or scripts to test code without the manual involvement of QA engineers. An important component in CI/CD, test automation encompasses pretty much any type of test that QA engineers might carry out to verify the quality of feature updates. Tests that can be automated include:

• Unit and integration tests.
• Security tests.
• Scalability and performance tests.
• API tests.
• GUI tests.

At a high-level, software delivery teams can use test automation to:

• Test at earlier stages of the CI/CD pipeline. By testing and detecting issues early, teams can keep bugs and errors from piling up as feature updates progress through the CI/CD workflow. This approach can allow developers to fix bugs more easily because they won’t have to significantly rework code nearly ready to go to staging or production.
• Test more often before release. Like earlier testing, more frequent tests enable teams to catch and fix issues early. Not only that, but automation allows hundreds of tests to be performed back-to-back, reducing the time necessary for running comprehensive testing pipelines.
• Test across different environments, platforms, and devices. Automated tests can be made platform-agnostic – that is, they can be made to work on any device or operating system. This allows teams to more comprehensively test the quality of their app within the designed range of the environments.

“The problem is not that testing is the bottleneck. The problem is that you don’t know what’s in the bottle. That’s a problem that testing addresses.”— Michael Bolton

If we were to view test automation at a more granular level, its benefits would be as follows:

• Reduced time to market. CI/CD pipelines are only as fast as their slowest component. Automation could significantly boost the time to market because manual testing can be a big bottleneck in software delivery. QA engineers might need to spend several days completing their testing routines with manual testing. Automation could reduce the testing time to just minutes.
• Increased test coverage. Test coverage is the percentage of code or application components that are covered by manual or automated tests. With automated tests, software delivery teams can cover more ground and ensure that no piece of code is left out. Additionally, by freeing up the time of QA engineers, testing automation can allow them to explore new avenues for improving the quality of their work.
• Improved software quality. Test automation brings improvements to software quality in several ways. First, with increased test coverage, test automation leaves few to no blind spots in an application and can ensure that all of its critical areas are covered. Secondly, properly implemented test automation eliminates the risk of human error and enables consistent, reliable testing of new code. Finally, test automation instills confidence in frequently changing products and allows teams to deliver polished, high-quality updates to their applications consistently.

What Are the Challenges of Manual Software Testing?

According to GitLab’s 2021 Survey of DevOps Teams from around the world, most software delivery teams are not doing automated software tests. While manual testing can be successful at small scales, manual work becomes extremely inefficient whenever we start talking about the need to run dozens and hundreds of tests several times per day.

Nowadays, high-performing software delivery teams commit new code several times per day. Without any automation, QA engineers would need to test new code every time it came in manually. With several commits per day and dozens of tests per commit, this would mean that QA engineers would need to spend many of their working hours on running tests.

Doing the same tests over and over manually can be extremely inefficient – so much so that manual testing can completely negate the benefits of frequent code commits. No matter how fast and often developers commit new features, these features cannot go live until they pass their test routines.

To confirm the importance of testing efficiency, GitLab’s 2021 survey revealed that testing was the primary reason for release delays in 2021. Moreover, 2021 marked the third year in a row when testing was highlighted as the main reason for delayed releases.

Security testing was a particularly worrying issue for teams. Over 42% of respondents said that security testing was happening too late in the process, and roughly the same people said they struggled to unpack, process, and fix vulnerabilities. For about 37%, tracking the status of bug fixes was a challenge, and 33% struggled to prioritize remediations. And perhaps most amazingly, 32% reported finding someone to fix the issues was difficult.

Test automation can make tracking, identifying, and remediating issues much easier. Despite this, most teams don’t have full test automation, according to GitLab’s survey. In 2021, the organizations indicated as estimated:

• 25% of the surveyed teams reported full test automation, which was more than double the previous year’s figures,
• 28% reported that they were at least halfway toward test automation, and
• 25% had no test automation or had just started thinking about it.

What Are the Business Benefits of Testing Automation?

“Test strategies are like getting a new dress…you can get a regular store size and model, but if you get it tailor made it will fit like a glove! Test strategies should be made specific to the project!”– Joao Proenca

On paper, the benefits of testing automation appear to be clear. However, what are its business benefits in the real world? In 2021, Forrester Consulting together with automation software provider BMC released a study that detailed the savings enabled by BMC Compuware Topaz For Total Test. BMC Compuware Topaz For Total Test is a mainframe software testing solution that allows the team to run automated unit, integration, system, and other forms of tests.

According to the study, four users of BMC Compuware Topaz For Total Test were able to achieve the following outcomes over three years:

• $4.5 million in time savings and increased developer output,
• $376,000 in cost avoidance through bug remediation,
• $2.9 million saved with testing automation, and
• $2.6 million saved by reducing major mainframe defects.

According to the study, automated testing with BMC Compuware Topaz For Total Test took only 2-3 hours, versus 2-3 days when done manually. Developers saved 90% of testing time and had 20% fewer bugs in mainframe code.

The GitLab survey we’ve already mentioned several times also provides some insight into the usefulness of test automation and CI/CD. Thanks to optimizations in their development processes:

• 84% of developers could release code faster than before,
• 57% reported releasing code twice as fast (compared to 35% in 2020), and
• 19% managed to deliver code to production 10 times faster.

In 2021, 10.38% of teams added test automation to their arsenals. With that in mind, automated testing likely contributed to software delivery acceleration. However, it wasn’t the only factor in increasing software delivery speeds. Aside from test automation,

• 21.02% of teams also implemented source code management,
• 17.74% continuous integration,
• 13.59% continuous delivery, and
• 11.65% adopted a DevOps platform.

Coupled with the fact that testing is a major bottleneck in software delivery, test automation must be combined with DevOps culture and CI/CD pipelines to deliver results. Sharma and Angmo (2014, p. 910) documented a range of benefits from web automation testing:

• Time and financial savings
• Improved accuracy,
• Increased testing coverage,
• Achievements by automation that manual testing could not accomplish,
• Automated support for QA testing developers and testers, and
• Improved team moral.

NASA’s Jet Propulsion Laboratory (Cervantes, 2009, March) assessed the value proposition of test automation in their Mission Data Processing and Control Subsystem (MPCS) and the Multi-mission Automated Task Invocation Subsystem (MATIS). The STAF framework provided a collection of general test services that could be used to develop automated test solutions. A critical, additional component was the STAF Execution Engine (STAX). It furnished a test automation programming language. STAX facilitated the development of a test execution workflow. Testers could:

• distribute test data,
• configure the test environment,
• execute the test, and
• analyze the test results.

Optionally, STAX provided an optional GUI that permitted a tester to conduct, monitor, and interact with test activities. The outcome was expressed as:

“Using STAF and STAX to implement the automated testcase showcased the benefits of using a test automation framework. The biggest benefit was that there was no need to develop test infrastructure to support the implementation of the automated test. There seemed to be a STAF service or STAX capability to facilitate the development of every part of the test.” (Ibid, p. 5)“There is definite overhead associated with utilizing a test framework. Testers must invest a considerable amount of time to learn how to use and adapt to using a test automation framework. Also, since test automation is a development project, testers need a considerable amount of knowledge in the area of software engineering. For these reasons, a test automation framework may not be suitable for small software projects with small team of developers and testers.” (p. 7)

Types of Tests That Can and Should Be Automated

The arsenals of QA engineers can include a vast range of tests, each aimed at checking a specific area or feature of their product. Test automation can be applied to applications and code at various levels, from individual functions to full applications. Sharma and Angmo (2014) proposed three types of testing:

• Static and dynamic testing,
• Box approach method (white-, grey- and black-box testing), and
• Manual and automation testing.

Below are the four layers of tests that can be automated:

• Unit tests. Unit tests check individual pieces of code or components in an application. QA engineers can automate unit tests to check the behavior of functions, classes, or API endpoints. Unit tests are very granular and can help engineers check the low-level functionality of their products.
• Integration tests. Integration tests are run on a larger group of components that aim to ensure that the application units work independently and when combined with each other. In essence, integration tests combine several related unit tests and check their operation. Examples of functionalities that could be tested with integration tests include payment processing, user registration, or messaging.
• System tests. System tests check the operation of full systems, with all the components that have passed unit and integration testing working together. QA engineers can automate system tests to ensure, for example, that their application allows users to seamlessly add products to their cart, choose a shipping method, and then make a payment to finalize their order.
• Acceptance tests. Acceptance testing aims to determine if the application meets the specifications and requirements of the end users. Unlike the previous three types of tests that check if a product technically works, acceptance testing aims to ensure that the product meets the end users’ expectations. Acceptance tests can be functional or non-functional. Functional tests check if the product behaves as the user would expect, e.g., if data submitted by the user gets passed to correct APIs. In contrast, non-functional tests check application security, scalability, and performance.

How can a software delivery team decide which tests to automate? A good candidate for testing automation will be:

• Repeatable. If a specific test needs to be run frequently across different feature updates or different applications, it’s a great candidate for automation. Repeating manual tests over and over can be a tremendous waste of time, so automating them can help increase the productivity of software delivery teams.
• Determinant. Software delivery teams should be able to clearly outline what counts as a test pass and what should be considered a failure. Otherwise, the automated testing pipeline will not be able to take proper action as code passes or fails its tests.
• Tedious. Repetitive tests are unsatisfying, boring, and can easily cause QA engineers to get distracted. Automating monotonous tests is a good way of making testing more fulfilling. Automation can also free up the time of QA engineers for more important tasks.
• Business-critical. The investment in test automation should make business sense. If the automation of certain tests will incur more costs than delivering value, the team should just keep them manual.

Types of Tests That Should Not Be Automated

While the team could automate most types of tests, a few classes of tests are difficult to automate or should not be automated. These include:

• Tests that are run only once. From a business perspective, automating a one-off test might not make sense because the costs would likely outweigh the benefits. With that said, automating the test might be worth it if it is complex and will take a long time.
• Exploratory tests. Because exploratory tests are done on poorly understood components, they are often impossible to automate – at least, fully. Besides, exploratory tests can be somewhat random as QA engineers might need to explore new aspects of the application that aren’t covered by any existing test.
• Tests of new functions that are under development. Work-in-progress features and code can change frequently, making their automation impractical. The team might be able to successfully develop an automated test suite for a feature that’s under development, but if this feature suddenly changes because it didn’t meet performance or security requirements, the automated tests might need to be rewritten entirely. New code and features should be written with both manual and automated testing in mind whenever possible, but the automation itself should be done only after the feature is finalized and deployed.
• UX tests. UX tests might be very difficult to automate because they heavily rely on the end user’s feedback. Quantifying user experience is one of the major challenges in UX testing automation. After an app or website goes live, measuring UX can be easy because the team can use metrics like bounce rate, conversion rate, or average order value to measure customer satisfaction. But while the app is in development, predicting how good the user experience will be might be extremely tough. However, predictive methods like machine learning could be effective at grading UX based on the past data of real users, making UX test automation more achievable.

Next Steps

“A group of managers were given the assignment of measuring the height of a flagpole. So they go out to the flagpole with ladders and tape measures and they’re struggling to get the correct measurement; dropping the tape measures and falling off the ladders.A tester comes along and sees what they’re trying to do, walks over, pulls down the flagpole, lays it flat, measures it from end to end, gives the measurement to one of the managers and walks away.After the tester is gone, one manager turns to another and laughs, ‘Isn’t that just like a tester? We’re looking for the height and he gives us the length.’ ”– Joke: The ‘Height’ of a Flagpole

In a DevOps-oriented CI/CD pipeline, no code is delivered without its corresponding suite of automated tests. Testing as early as possible is the key to keeping software bug-free, secure, efficient, and attractive to end users.

Software delivery teams can write their own automated tests from scratch or use third-party automation software to save time. The options are numerous, but no matter which route a software delivery team chooses, standardization and modularization are the most important prerequisites for testing automation.

Microservices can easily be reused across multiple projects because of standardized code and application components. Similarly, standardization makes software tests repeatable and transferable. If different application products are built on the same standardized components, developers and QA engineers can build automated test suites and easily reuse them across a wide range of products (Garousi and Elberzhager, 2017; Polo, et al., 2013).

However, automated testing is only part of the story. Testing isn’t the only thing that should be automated in a software delivery pipeline – every step from commit to deployment should ideally be automated. And the key to automating software delivery is orchestration [TA2] or, more specifically, CI/CD.

Team members may be interested in becoming familiar with a test automation course:

• DevOps CI/CD Pipeline: Automation from Development to Deployment
• Selenium Automation Testing for Web Applications
• Create Your First Test Automation Script - Selenium and C#

[TA2] The article “The Business Value of Orchestration in Software Delivery” can be linked from here.