I am seriously pro test automation. After working on several projects with scant, if any automated test coverage I really appreciate the benefits of having fast, repeatable tests running as part of the build pipeline.

Unfortunately (or fortunately depending on your viewpoint) test automation is not a replacement for manual testing. You really don’t want to be duplicating your testing by manually testing the same things as your automated tests but you do need to make sure you’re placing you trust in a sound automation suite.

When you implement a new feature you’ve probably had some conversations with developers, designers and product people. Hopefully you have a good idea of the risks around this feature. In an ideal world the developer would have added lots of unit tests and maybe a few integration tests to test this feature. In a perfect world you, as the tester, know exactly what has and hasn’t been tested.

Problem number 1 comes from automated tests needing exact expected results to be completely defined up front. Quite a lot of testing is about finding out more information, using your tests to answer questions. Automation doesn’t give you any help with this.

Problem number 2 comes from trusting your tests. How do you know that your automated tests will find problems? Will they fail when something goes wrong? Does that green build really means things are working? The only way to answer these questions is to manually test. Finding bugs in high risk aspects of the system indicates missing or incorrect automated tests. Keep reviewing your risk categories and make sure you have then tested with reliable, complete automation.

Next time you sit down to carry out some manual testing use it to test the tests. The outcome of the testing should shape your opinion of your automated test suite. Bugs found should be fixed and new tests added. Confirm that tests passed have suitable test scenarios covered by automation and then relax.