How to Ensure Effective Automation Strategy

The concept of “Test Pyramid” has been around for a while; however, time and time again, we find the automation put in place by teams is not effective or does not produce the ROI as expected by stakeholders.

The diagram below is the Test Pyramid introduced by Mike Cohn in his book, “Succeeding with Agile.” Test Pyramid serves as a guideline as you embark on the journey of test automation. It helps you identify various layers in your test strategy and how many tests you should have in each layer such that you can stick to the shape of “Test Pyramid”.

Let’s dig a bit deeper to understand more about the Test Pyramid.

1.  Unit tests are isolated and faster to execute. Since they are isolated, the unit tests execute predictably with a lesser chance of false failure. Hence, they are less fragile.
2. However, as Unit tests are isolated, they provide lesser confidence. We would want to have thousands of unit tests as they can execute faster and can shorten the feedback cycle to developers.
3. Integration tests fall somewhere between unit tests and UI tests, thus, providing the balance between the two. The Integration tests should be in the hundreds.
4. UI tests are integrated, thus slowing the execution process. Since they are dependent, the chances of false failure are higher. Hence, UI tests are highly fragile in nature.
5. However, since they are integrated, they provide higher confidence in terms of ensuring the functionality is working as expected. Due to the increased execution time and fragility, we would want to limit the number of UI tests typically in dozens.

As a general rule, in order to ensure effectiveness of the Automation strategy, we would want to make sure the Test Strategy follows the “Test Pyramid” wherein you have thousands of unit tests, hundreds of integration tests, and dozens of UI tests. 

The Use Case

We will use Cartos (our own product) as the use case to look at the challenges and examine various approaches to improve the effectiveness of automation.

Let us first look at the way Cartos stood in terms of the “Test Pyramid.” The diagram below shows a picture representing the test coverage across “UI tests,” “Integration test,” and “Unit tests.”

After several years of development efforts, Cartos ended up with several thousand unit tests, hundreds of UI tests, and no API Integration tests. Despite years of investment in automation, the ROI was not significant.

UI tests took many hours to execute and we experienced false failures too often. Two consecutive runs would never produce the same result; hence, the test team lost confidence in the tests and they ended up testing everything manually. Over a period of time, maintaining these tests became a lot of overhead and the team had to spend a significant amount of time to ensure the tests could run smoothly.

Integration tests were not available.

Unit Tests coverage was more than 60%; despite that, they would hardly catch any issues. The outcomes did not provide confidence to the test team and they ran all tests manually.

The Path Forward: Introduction of API Tests

Evidently, the Cartos automation strategy was not aligned with the Test Pyramid. In order to align the automation strategy with the Test Pyramid, we needed a significant number of API integration tests and perform the minimal amount of UI tests. The first step we took was to stop any further development of UI tests and begin the development of API tests. An API test framework was put in place with an objective to ensure that the majority of integration tests can be done at the API level while leaving a minimal number of tests to be covered at the UI level.

Focus on Quality Unit Tests Instead of Quantity

The next step was to look at the quality of unit tests. Although the unit tests were significant in numbers, we found that the unit tests did not provide practical business value. The unit tests were written using the traditional approach, where for every method there was a corresponding test method and any Dependency would be mocked. This ensured that the unit tests were isolated and they passed the majority of the time. Yet, when integrated, the system did not work.

In order to ensure the effectiveness of the unit tests, we decided to increase the granularity of the unit we wanted our tests to cover. Instead of making the unit tests focus on a class and its method, we wanted the unit tests to focus on a module and functionality/feature provided by the module.

To make it more evident, let’s take an example of AES Encryption implementation provided by Java (https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/com/sun/crypto/provider/AESCipher.java)

The AESCipher class has several methods that perform various routines that are needed to implement the encryption. Following the traditional unit test implementation approach, we would test each method of the AESCipher class and mock any dependencies. Another approach would be to look at it from functional perspective and define it as a functional unit as follows:

String AESEncrypt (String input, String key)

If we focus our effort on testing this method, it is guaranteed that if any of the other methods of AESCipher class did not behave properly, the tests targeted for AESEncrypt would fail. Hence, we do not have to test each method specifically. By following this approach, we can reduce the quantity of unit tests and have better quality tests in place.

Following this approach, we started to look into our system as a set of functional units rather than a bunch of classes and methods. In addition, we designed the system so that each functional unit had minimal to no dependencies on other functional units. This way, each of the functional units can be tested independently without having to mock the dependencies. This process resulted in very high quality unit tests.

Here’s the rulebook one can follow to ensure high quality unit tests.

Minimize Logic in UI Code

Being an Angular application, Cartos UI had a significant amount of business logic due to the nature of the application. Also, since it is an Angular app, there is a tendency from the developers to introduce business logic in the UI as opposed to traditional web apps where most of the business logic will be in the backend.

We refactored the the UI layer in Cartos to ensure it is very thin and had almost no business logic, thereby reducing the need of UI tests to absolute minimum.

We created a separate TypeScript layer that would handle all the business logic which is agnostic to the framework such as Angular. This layer can be used to perform all the operations that can be implemented in the UI. Afterwards, we targeted all our tests to the TypeScript layer instead of the UI. This strategy allowed us to have the benefit of UI tests (higher integration) without actually testing the UI itself.

Summary

We were able to make the Automation effective by doing the following:

1. Align the Test Strategy with the Test Pyramid - We eliminated UI tests and introduced API test to ensure the Test Strategy is aligned with Test Pyramid. 
2. Focus on Quality Unit tests - We focused on building functional unit tests instead of pure unit tests to produce high quality unit tests.
3. Eliminated Business logic from UI - We eliminated Business logic from the UI layer and segregated in a separate layer that could be easily unit tested; this also helped in reducing the need of automation at the UI level.

With the above changes in the strategy, we were able to eliminate manual efforts of the team while incrementally building the quality automated tests.