Transform your Automation Suite into a testing product — Part 1
When we talk about the test product that is owned and managed by the testing team, what comes to your mind immediately. Is it a test suite, test management tool, or any defect tracking tool?
I guess the most obvious one that comes to your mind is the test suite, in this automation era its counterpart can be easily termed as an automation suite, but just adding the new test cases to the existing automation suite and running it once or twice by the testing team at the time of the release doesn’t necessarily make it fall under the category of a product.
If we go by the definition of a product, a product is an object or system made available for consumer use.
In our case, the consumer is the one that is using our automation suite for regression of their application or just to validate that everything is working fine as expected and not breaking something.
Let’s explore the pillars of a great product in the context of an automated test suite.
Key Attributes of Product Revision are Maintainability, Flexibility, and Testability
Maintainability: In the context of the automation suite it means you can modify the existing test cases or add new ones as per the changes in the product with time and the ease of making such modifications in the test suite. How much time does it take? Lesser the time and effort you need to put in, the more maintainable is your test suite.
Flexibility: Flexibility means if your test suite is flexible enough to automate test cases of all the systems under test e.g microservices, app, web.
Testability: This answers the question that do you have any mechanism to test your test suite? Do we have any PR checks, linter checks to test the automated test suite? they are essential to ensure the testability of the product.
Attributes that ensures the Product Transition are Portability, Reusability and Interoperability
Portability: It means that you should be able to run the test suite on any machine whether its a dev machine or any CI setup. It can be achieved by dockerizing your test suite or exporting your test suite as an executable package.
Reusability: Reusability means whether you can reuse the module that you have created earlier e.g reporting modules or common utilities (logging, data fixtures, mocks, logging, etc) to build a new module to test another system under test. Are your existing functions written in a way that it can be used with other modules to automate new functionalities and scenarios?
Interoperability: Interoperability means if your test suite is capable enough to communicate with the other tools for exchanging information e.g any alert and monitoring tool like Grafana.
Key Attributes of Product Operation are Correctness, Usability Efficiency and Reliability, Integrity
Correctness: Correctness means how correctly you have automated your test Scenarios. Do we have the necessary assertion/checks in each automated scenarios to ensure that we have covered it's all critical functionality and a green build gives us the confidence to push that changes in production env.
Usability: Usability means, how easy to run/trigger these tests and to how many systems it has been integrated to, as these factors increase the usability of your test suite. Is your automation suite is integrated with any CI that can automatically trigger these test cases with each changes happening in your system under test, having CI setup or functionality to run the test suite from the developer code increases the chances of the test suite to get used by the devs
Efficiency: Efficiency means how efficient is your test suite to find the bugs, Does the passing of your automated test suite means that all the critical flows of the system under test will work as expected.
Reliability: Reliability means how stable are your automated tests? Does every time the test fails is it due to the actual issue and not the false positive? Any flakiness in the test suite hampers the reliability.
Integrity: Integrity in the automation suite means following the principles honestly whether it’s related to the testing principles while adding assertion or following the standard coding practices.
Until now, all of us are on the same page as to what to expect from a product in context to the automation suite
Evolution of Automation Suite at Grofers
Phase 1: Era of UI Test Suite
UI + BDD (cucumber)
UI + BDD + local device farm
UI + BDD + local device farm +CI
UI+ BDD + local device farm + CI + parallel execution
UI+ BDD + CI + parallel + Dashboard + Running on the cloud
UI+ BDD + CI + parallel + Dashboard + Running on the cloud + optimisation (Debuggability + Speed)
Phase 2: Transition From UI to API e2e Tests
API +BDD+ parallel + CI
API + BDD + parallel + CI + optimisation (stability (own data fixtures and mocks ))
Phase 3: Transition From Automation Suite to a Testing Product
Use them as PR Checks and in the production Deployment pipeline
API + parallel + CI + CD + optimisation (Speed (reduce the execution time))
API + parallel + CI + CD + optimisation +Analytics and Dashboard
API + parallel + CI + CD + optimisation +Analytics + Learnings ( User Feedback + RCA)
API + parallel + CI + CD + Analytics + optimisation (debuggability (based on learnings ))
API + parallel + CI + CD + Analytics + optimisation + Health Checks
Phase 4: Transition From API e2e to Contract Tests (shift left strategy)
Contract test + CI + CD + code to be part of Service Repo
This is just an overview of the series — Transform your Automation Suite into a product. Stay tuned in for subsequent parts where we will be discussing each transition stage in detail and how we have achieved the same in Grofers.
Pramod Kumar is the Engineering Manager (Test Engineering) at Grofers, a Quality Advocate, and loves to be called Tester/Automation Engineer.
And yes Grofers is hiring! If you are interested in doing great work, we’d love to hear from you.
Transform your Automation Suite Into a Testing Product — Part 1 was originally published in Lambda on Medium, where people are continuing the conversation by highlighting and responding to this story.