Partial Automation: Keeping humans in the loop
Tuesday, November 04, 2008
Posted by Patricia Legaspi, Test Engineering Manager
One of the challenges of automation is achieving complete automation. Ideally, complete or total automation would not require any human intervention or verification yet this is a difficult level to achieve. Investing Engineering time to completely automate tests is expensive and, many times, has diminishing returns. Rather than trying to achieve complete automation, investing in ways to make the most out of the automated test and the human time is time better spent.
Effective test report
Consider an automated UI test... Routinely, automated UI tests result in false negatives due to timing issues, the complexity of the steps taken, and other factors. These have to be investigated which can be time consuming. A thorough report can be created for automated UI tests to present log information, error reports, screen shots of the state of the application when the test failed, and an easy way to re-run the test in question. A human can make use of the information provided and effectively investigate the possible issue. If we can reduce the amount of work that someone has to do to investigate a test failure, the UI tests become more valuable and the human plays a much more important role as a "verifier." Had the report not provided any information for the failed test, the human would have spent, in some cases, hours investigating the issue rather than continuing to automate or run exploratory tests which would be of more value. Is there a way to maximize what people do well, while also maximizing the use of automation?
Applying human intelligence
What about tests that require a human eye? There are many arguments about tests that require a human make a judgment about the appearance of rendered UI objects. Machines are great at running tests that return in a firm pass or fail but for tests that require opinion or judgment, the human has an advantage. We've been experimenting with image comparison tests. Screen shots of a golden version of the application under test are compared to screen shots of the current release candidate to verify that the application continues to render properly and that the UI "looks good". Although image comparison techniques can determine if the screen shots are different, they cannot help determine if the difference is "important". The human comes back into the picture to complement the automated test. To make effective use of a person's time, the test results should be organized and well presented. For image comparison tests, a report which shows the golden and release candidate screen shots side-by-side with clearly highlighted differences or allows you to replace the golden screen shot is key. A tester can quickly navigate the reported differences and determine if they are actual failures or acceptable differences. Frequently, an application's UI will change which will result in expected image differences. If the differences are expected changes, the tester should be able to replace the golden with the new image with the click of a button for future comparison.
Aside from test automation, efficiency can also be improved by streamlining the various components of the process. This includes test environment setup, test data retrieval, and test execution. Release cycles tighten and so should the testing processes. Automating the environment setup can be a major time saver and allow for added time to run in depth tests. Creating continuous builds to run automated tests ahead of time and preparing environments overnight results in added testing time.
Automated tests are rarely bullet proof, they are prone to errors and false failures. In some cases, creating an infrastructure to make test analysis easy for humans, is much more effective than trying to engineer tests to automatically "understand" the UI. We coined this, "Partial Automation." We found that by having a person in the loop dramatically reduces the time spent trying to over engineer the automated tests. Obviously, there are trade-offs to this approach and one size doesn't fit all. We believe that automation does not always need to mean complete automation; we automate as much as you can and consider how human judgment can be used efficiently.
One of the challenges of automation is achieving complete automation. Ideally, complete or total automation would not require any human intervention or verification yet this is a difficult level to achieve. Investing Engineering time to completely automate tests is expensive and, many times, has diminishing returns. Rather than trying to achieve complete automation, investing in ways to make the most out of the automated test and the human time is time better spent.
Effective test report
Consider an automated UI test... Routinely, automated UI tests result in false negatives due to timing issues, the complexity of the steps taken, and other factors. These have to be investigated which can be time consuming. A thorough report can be created for automated UI tests to present log information, error reports, screen shots of the state of the application when the test failed, and an easy way to re-run the test in question. A human can make use of the information provided and effectively investigate the possible issue. If we can reduce the amount of work that someone has to do to investigate a test failure, the UI tests become more valuable and the human plays a much more important role as a "verifier." Had the report not provided any information for the failed test, the human would have spent, in some cases, hours investigating the issue rather than continuing to automate or run exploratory tests which would be of more value. Is there a way to maximize what people do well, while also maximizing the use of automation?
Applying human intelligence
What about tests that require a human eye? There are many arguments about tests that require a human make a judgment about the appearance of rendered UI objects. Machines are great at running tests that return in a firm pass or fail but for tests that require opinion or judgment, the human has an advantage. We've been experimenting with image comparison tests. Screen shots of a golden version of the application under test are compared to screen shots of the current release candidate to verify that the application continues to render properly and that the UI "looks good". Although image comparison techniques can determine if the screen shots are different, they cannot help determine if the difference is "important". The human comes back into the picture to complement the automated test. To make effective use of a person's time, the test results should be organized and well presented. For image comparison tests, a report which shows the golden and release candidate screen shots side-by-side with clearly highlighted differences or allows you to replace the golden screen shot is key. A tester can quickly navigate the reported differences and determine if they are actual failures or acceptable differences. Frequently, an application's UI will change which will result in expected image differences. If the differences are expected changes, the tester should be able to replace the golden with the new image with the click of a button for future comparison.
Aside from test automation, efficiency can also be improved by streamlining the various components of the process. This includes test environment setup, test data retrieval, and test execution. Release cycles tighten and so should the testing processes. Automating the environment setup can be a major time saver and allow for added time to run in depth tests. Creating continuous builds to run automated tests ahead of time and preparing environments overnight results in added testing time.
Automated tests are rarely bullet proof, they are prone to errors and false failures. In some cases, creating an infrastructure to make test analysis easy for humans, is much more effective than trying to engineer tests to automatically "understand" the UI. We coined this, "Partial Automation." We found that by having a person in the loop dramatically reduces the time spent trying to over engineer the automated tests. Obviously, there are trade-offs to this approach and one size doesn't fit all. We believe that automation does not always need to mean complete automation; we automate as much as you can and consider how human judgment can be used efficiently.
This comment has been removed by the author.
ReplyDeleteIn reply to this openion:
ReplyDelete>>>> Aside from test automation, efficiency can also be improved by streamlining the various components of the process. This includes test environment setup, test data retrieval, and test execution. Release cycles tighten and so should the testing processes.
I feel Partial Automation is very effective way of carrying out the QA process in an organization. With rapidly changing UI many a times it just wastes engineers time in automating some component which becomes obsolete faster than the time it consumed to get implemented.
Looking into streamlining the QA process and automation I found, getting sanity test automated in first stab makes life lil easier. For both the engineer who is automating it and for the testers, who can off load some work from their back.
Also doing the formal code review with QA automation engineers helps them in understanding what part qualifies for automation. Which might help in building an effective partial automation.
I very much appreciate the general message.
ReplyDeleteI could give a lot of examples when only the combination of human interaction and automation yields economic results.
(a testmanager)
Yes, Partial Automation definitely is an effective way of carrying out the QA Process. The Automation Tools can Automate the process and make things faster and manytimes easier but humen interaction and intelligence always needed to yield the correct and useful results.
ReplyDeleteWell, I recently built a decent software-based solution to partial automation (I think!) that would allow to minimize the impact of human intervention into the automated process. I can't call this a framework, but it manages the capability to run automated scripts, while allowing to seamlessly switch between automated and manual modes of execution. I will soon start posting information on this into my personal website - http://www.my2pie.com/blog (I'll need some revamping of my site, before I can do this, and might sometime). If you can't hold you questions, please look up my profile and write to me directly.
ReplyDeleteOn the topic of partial automation, I agree its helpful, and done well can offer a first pass/stage of testing to alleviate the amount of manual testing required (reduce manual testing scope to areas of importances based on partial automation test results, etc.)
ReplyDeleteBut that also requires management buy in. In the past, I had suggested to an organization adding partial automation to our automation suite to gain additional test coverage, even though it would not be full coverage with automation, it didn't get buy in, because to them it was either go big or go home. They don't want incremental improvements. :(