We live in a day and age where consumers cannot access medication for their ailment because of cost. We are aware of the anecdotes related to delaying treatment. In the same way, some engineering teams are stuck with projects which are in an unhealthy testing state. The team goes on sprint after sprint with the situation getting worse, not better. In this article, I will introduce a low-cost tool to enable you to diagnose your situation and put together a treatment plan.
It’s a sad situation, but I think we’ve all been in the place where testing on our project is a pain.
It’s a project where no one took the time to set up unit testing. Or the unit test framework exists and no one bothered to add unit tests. There are unit tests, but no integration tests. Or there is no support for the types of integration tests you want to write. The tests are flaky. They take too long. The tests are difficult to maintain or understand. The tests don’t provide value and were added to hit a random test coverage number. After all the automated tests pass, you still have this nagging doubt about releasing the latest version to the public.
The pain points I have called out above are not exhaustive. We can relate to these pain points, but why don’t we fix them?
We know ignoring a medical problem is not a good idea. It’s not a good idea to ignore your testing pain. But choosing to tackle your testing problems can seem like a daunting task. A tool that I have found helpful to start dealing with my testing pain is to develop a test strategy.
Develop your Test Strategy
A strategy is “A plan of action or policy designed to achieve a long-term or overall aim.” (Definition of strategy in English. https://www.lexico.com/en/definition/strategy) A test strategy is a plan of action or policy designed to achieve a long-term testing aim.
To develop a test strategy, start answering the following questions.
What are the testing pain points?
If you have never taken the time to identify the testing pain that you are experiencing, that is a great place to start. A prescription is very focused on the treatment of a specific ailment. Similarly, develop a test strategy to deal with your immediate testing pain. The first step is being clear about what is painful. In this exercise, make sure you go beyond the symptoms. To identify test strategies that last and are effective, we need to get to the root of the problem.
Maybe your issues have to do with being clear about what needs to be tested. Identify your key features that need testing.
- What are the things you want to have confidence before you release your project? User success scenarios? User error scenarios? Likely error states? Unlikely error states?
- What is your risk tolerance?
- What is in the scope of testing?
- What is out of scope?
Maybe your issues have to do with being clear about who is responsible for the testing. Identify who is responsible for different parts of the testing.
- Who tests?
- Who develops the tests?
- Who runs the tests?
- Who looks at the test failures?
Maybe your issues have to do with how your project is tested. Figure out how you want to test the project.
- How many layers of tests that you will have? Unit tests? Integration tests? End to end tests? Smoke tests?
- How will you test your analytics?
- How will you test your integration with that upstream API?
- How will you test your dependency on the database?
- How will you test that the user interface conforms to user experience designs?
- How will you test different error scenarios?
- Do you use hardcoded data or generated data?
- Do you mock or depend on live services?
Maybe your issues have to do with where your project is tested. Find out where the tests are running and if that is where you want them to run.
- Do they run locally?
- Do they run in a test environment?
- A staging environment?
- A QA environment?
- In production?
Maybe your issues have to do with where your tests are located. Identify the best location for the different types of tests.
- Do they live with the production code?
- Do they live within the QA test repo?
- Do they have their own separate repo?
- Is it ok that someone has them stored in a directory on their workstation?
Maybe your issues have to do with when the tests are developed. Identify when different tests are expected to be developed.
- Do tests need to be written before you develop code?
- Do you write tests after you develop the code being tested?
- Do you need to push unit tests with your code?
- Do you need to push integration tests with your code?
- Do you add tests after you push your code?
- When do the end to end tests get updated?
- When do the smoke tests get updated?
Maybe your issues have to do with when tests are run. Identify the times for them to run.
- Do you block pushing a commit until the tests run?
- Do you run the tests before deploying to an environment?
Discuss your test strategy
After you answer the questions you think need answering, have a discussion as a team and get agreement on the test strategy. We found the process of debate and discussion is very beneficial to the strategy, the team and the project.
Implement your test strategy
Once you have your first version of the test strategy, start applying it. There are many techniques for deciding what to do first. Focus on addressing the biggest pain point. Starting immediately, operate in light of the test strategy. Build up a backlog, prioritize it and add the stories and tasks to your board. Whatever approach you use, remember that just because you came up with all these points and goals does not mean you are committing yourself to fix everything right now.
Benefits of developing a test strategy.
I like the definition of strategy because it contains the concept of the long-term view. What you are doing is setting the direction of where you want to go.
Another benefit of developing a test strategy is the conversation it drives. We all have hidden biases and assumptions that we operate on. Get the specifics out of our minds and write them down where we can see and evaluate them. Consensus and alignment are essential to the long-term testing goal of a whole team approach to quality and testing.
A test strategy also reduces ambiguity. Being clear about expectations related to testing helps people move faster.
A test strategy guides building testing into your planning. Testing can easily become an afterthought. A test strategy brings testing to the forefront. It facilitates being intentional rather than passive.
A test strategy assists you in identifying and prioritizing your top testing concerns. It helps identify the implications of those choices.
Remember a test strategy should be a living document. It should not be something that you come up with one time and then forget about it. As you apply your test strategy you will learn from your successes and mistakes. Adapt your test strategy accordingly.
Happy and healthy testing.
Treat Testing Pain with a Healthy Dose of Test Strategy was originally published in RetailMeNot Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.