Why Your Test Automation is Failing
[This blog was originally published on June 5, 2019, and updated on July 28, 2021]
One of the most common issues is that the test part of test automation is rather straightforward to automate, especially with today’s abundance of great automation tools. But jumping straight into automated testing leaves you with a big heap of automation scripts, and often not the best ROI. Understanding why requires looking into the automation process.
When an engineer builds an automated test, they do it in a test environment that they (or someone in the organization) set up. Usually, this is the same way manual testers access test environments. Before the engineers start automating, they have a working copy of the product they are testing, which includes all its relevant data. They have a good idea of the test they want to write, and everything looks promising. They write the test and give it a few runs. Everything appears stable. Now, to provide the ROI we were hoping for, the test must run a million times. Or, at least, a few thousand times, right?
To reap the benefits of automation, testing needs to happen frequently. However, the bottleneck is often discovered after creating many tests. Because the tests must run on a test environment that simulates the one used to create them, with the new versions of the product or with any introduced changes that are being tested, you need access to hundreds of environments. This can slow software testing and reduce its effectiveness.
Three test environment approaches that inhibit test automation
Manually Setting Up an Environment Each Time to Run Automated Tests
For tests that don’t need to run as frequently, such as security testing and performance testing, it’s common to set up an environment for automated tests every time the test needs to be executed. As a result, testers must wait for the environment to be ready, perhaps do some additional configurations and then run a suite of tests. And, if they’re trying to optimize utilization, the environment is left to run overnight or throughout the weekend.
The problem with this approach is it takes the “auto” out of automated testing. When automation depends on a manual step, the test is only as efficient as this manual step.
If the manual step isn’t very efficient, automation may not be helping at all. I’ve seen companies take so long to get an environment to work that running automated tests on it didn’t make any improvement in the speed at which testing occurs.
Setting Up a Static Environment That Automation Always Runs On
If environment configuration and provisioning take a long time, many people assume it makes sense to leave an environment running, so test automation can run on it. The preparations were made, the infrastructure is all in place, the product is configured, and the data is ready, so let the automation party begin.
While complex, static test environments may seem ideal, but the party doesn’t last long. First, since it usually takes some time to put the environment together (same as in the previous case), it often becomes high maintenance. It can be scary to make changes that could ruin things, and you can end up needing someone to manage change full-time. Much like chocolate cake in the office kitchen, an environment that mimics production with good data on it tends to attract everyone. Soon, these nice environments become shared assets and tests start conflicting for the same resources. The natural next step is to set up another static environment. This loop usually ends with an angry manager saying something about cutting CAPEX.
Automating the Environment and Cutting Ourselves Some Slack
With the rise of infrastructure as code (IaC) and immutable infrastructure, there are more ways to spin up test environments. Resourceful test teams will try to automate the environment creation or use built-in test tool or CI system capabilities to run tests on a public cloud instance or a container, if the product they test allows that. This is cheap and useful—and could work for some simple tests. The challenge usually starts with everything this approach doesn’t cover. It can be difficult to use IaC to spin up a test environment and handle all aspects of it—from security and isolation to dependencies, data, third-party integration and more. This is where this approach is often blocked and sends us back to the previous two approaches for a big part of the testing load—which again makes test automation investment less fruitful.
The Right Way to Set Up Test Environments: Test Automation on Dynamic Environments
The test automation world needs a game changer. You can make your car faster, but if the road is full of potholes, you won’t be able to go that fast or get very far. Test automation is similarly limited by the pace of environment creation, with its infrastructure, application, data and other requirements.
This challenge becomes critical when adopting DevOps, and testing stops being a stage in the pipeline, instead becoming an overall pipeline service. QA is no longer about being the gate between development to operations. It’s about overseeing the quality of the product during its entire life cycle. This means automated tests should always be able to run, and the environments they run on must be dynamic.
The solution to this challenge is Environments as a Service (EaaS) that provide infrastructure automation capabilities. EaaS is all about packaging the environment, including the application, infrastructure and data, as an entity that can be provided as a service to a business process. Once defined, it can be provided on-demand for manual or automated users. Since it’s a well-defined entity, it also can be automatically decommissioned when the process finishes using it. Playing an important role in making infrastructure horizontal, EaaS is also the key for paving the road for brilliant test automation.
If you think you should get more from your test automation initiative, click below to learn about how Quali’s CloudShell can benefit your test automation processes.