Let us discuss some good practices in the test stage of the build pipeline now.
I remember a story that I have heard in my childhood.
A king tried to attack one of his neighboring kingdoms and annex their territories to his. However, each of his attack was belligerently defied and each time he had to retreat. Once on his retreat, he happened to chance up on a mother feeding a child. The child had his tongue stung by the hot food and was crying. The mother asked the child to start eating from periphery, where the food was less hot and work his way to the middle of his plate. The king got inspired by this idea and the rest, as the say, is legend.
Improving your test automation is nothing short of a battle. There are formidable challenges in the form of evolving code, the sheer number of test cases to be automated, dependencies between the test cases and some rather temperamental test environments.
I recommend the following four guidelines which could give a good head-start for your test automation.
1. Take small bites
Test automation is best achieved incrementally. You can start with automating the smallest set of test cases that matter the most. Usually, this will be the User Acceptance Test (UAT) cases. Each small set of test cases that are automated should be time tested as suggested in the next guideline.
Once most of your UAT are automated, you can extend your automation to other happy path tests. This can be followed by including more core module tests. The sequence of your automation can follow the figure below:
|Sequence for improving Test Automation|
Once you have automated most of your UAT, you can identify your smoke test suite from these tests. This suite characterized by a very short feedback time, usually less than 5 to 10 minutes. The smoke test suite can be integrated to your Version Build which is triggered with each code merge to the master branch.
2. Time-test your automation
Each set of test cases that are automated should be time tested for its viability. The automated test cases should be incrementally added to your CI and should run at least once daily.
Developing your test code as carefully as your code can improve your pass rate during time testing. Refactoring the redundant test setup code among test cases can be considered first. Good design patterns and (test) coding guidelines are also essential.
3. Faster feedback first
There are different types of tests. Functional tests, performance tests, security testing, load tests and other validating tools are some of them.
The thumb rule in sequencing their automation is to achieve the faster feedback first. When you start automating, your CI, adaptations to test frame work, test cases and code (usually!) are all evolving. Automating the test types with faster feedback times will enable you to achieve your early wins faster. You can contrast between debugging the test failures in two test setups; a functional smoke test suite that runs in 10 minutes and a load test that takes 12 hours to run. Be mindful of the impact of these feedback loops on your test automation progress.
4. Maintain green builds
What is your progress metrics for test automation?
Punters may argue between test automation rate, code coverage through automation and any other. In my experience, the One-Metrics-That-Matters in the opening game of test automation is the green build rate from your CI.
You can start with a goal of at least one green build per day and focus on reducing the red builds by debugging and avoiding random test failures in your CI.
Rushing your test automation improvements is like running up slippery stairs. Being careful and taking one step at a time can make the difference.