Testing the Tests

I am seriously pro test automation. After working on several projects with scant, if any automated test coverage I really appreciate the benefits of having fast, repeatable tests running as part of the build pipeline.

Unfortunately (or fortunately depending on your viewpoint) test automation is not a replacement for manual testing. You really don’t want to be duplicating your testing by manually testing the same things as your automated tests but you do need to make sure you’re placing you trust in a sound automation suite.

When you implement a new feature you’ve probably had some conversations with developers, designers and product people. Hopefully you have a good idea of the risks around this feature. In an ideal world the developer would have added lots of unit tests and maybe a few integration tests to test this feature. In a perfect world you, as the tester, know exactly what has and hasn’t been tested.

Problem number 1 comes from automated tests needing exact expected results to be completely defined up front. Quite a lot of testing is about finding out more information, using your tests to answer questions. Automation doesn’t give you any help with this.

Problem number 2 comes from trusting your tests. How do you know that your automated tests will find problems? Will they fail when something goes wrong? Does that green build really means things are working? The only way to answer these questions is to manually test. Finding bugs in high risk aspects of the system indicates missing or incorrect automated tests. Keep reviewing your risk categories and make sure you have then tested with reliable, complete automation.

Next time you sit down to carry out some manual testing use it to test the tests. The outcome of the testing should shape your opinion of your automated test suite. Bugs found should be fixed and new tests added. Confirm that tests passed have suitable test scenarios covered by automation and then relax.


One comment

  1. Good post – it is indeed hard to verify automation scripts are reliable and even harder to verify the keep on being reliable over time.
    Serious debugging should be done, verifying not only that the tests pass, but also that they will fail when needed.
    As to manual tests – these can be eased by using the already existing automation infrastructure building stones,
    If only you supply your testers with a simple GUI which allows executing these functions.
    This will save time in running the tedious tasks of manual tests, and increase their power.

    Kobi Halperin (@halperinko)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s