Comments on: UI Automation Out of Control http://www.testingmentor.com/imtesty/2009/11/18/ui-automation-out-of-control/ Treatises on the practice of software testing Tue, 07 Feb 2012 09:15:11 -0500 hourly 1 http://wordpress.org/?v=3.1.3 By: testingmentor http://www.testingmentor.com/imtesty/2009/11/18/ui-automation-out-of-control/comment-page-1/#comment-309 testingmentor Thu, 19 Nov 2009 05:38:19 +0000 http://testingmentor.com/imtesty/2009/11/18/ui-automation-out-of-control/#comment-309 Thanks Wesley. These are some great examples of things that could cause problems with visual comparison in automation. There is some interesting work being done with visual comparison such as histograms, color and pixel tolerance maps, etc. This is all interesting work, but the simple fact is that sometimes it is easier, more reliable, and cheaper to validate UI visuals via usability testing, end-to-end or user scenario testing, exploratory testing, dog-fooding, self-hosting, beta feedback, etc. Many of us have been down that hard way path...but sometimes the school of hard knocks provides the best education. Thursday, August 13, 2009 1:52 AM by I.M.Testy Thanks Wesley. These are some great examples of things that could cause problems with visual comparison in automation.

There is some interesting work being done with visual comparison such as histograms, color and pixel tolerance maps, etc.

This is all interesting work, but the simple fact is that sometimes it is easier, more reliable, and cheaper to validate UI visuals via usability testing, end-to-end or user scenario testing, exploratory testing, dog-fooding, self-hosting, beta feedback, etc.

Many of us have been down that hard way path…but sometimes the school of hard knocks provides the best education.

Thursday, August 13, 2009 1:52 AM by I.M.Testy

]]> By: WesleyB77 http://www.testingmentor.com/imtesty/2009/11/18/ui-automation-out-of-control/comment-page-1/#comment-308 WesleyB77 Thu, 19 Nov 2009 05:38:10 +0000 http://testingmentor.com/imtesty/2009/11/18/ui-automation-out-of-control/#comment-308 I couldn’t agree with you more. We had tests in a previous team of mine that tested parts of MSTIME (Microsoft Timed Interactive Multimedia Extensions) by comparing a known good screenshot to a screenshot taken using the binary being tested. I arrived a few years after the code was written, and the nightmare was not over. There were a lot of problems. Due to the sheer magnitude of environments and platforms and machines being tested, there were always a high number of false positives. Sometimes certain video cards on the machine would colour a pixel differently (FAIL), or a frame would be skipped (FAIL), or it would render slowly (FAIL), or ClearType was enabled on the machine (FAIL), or (FAIL) or (FAIL) or (FAIL) or (FAIL). It was very frustrating. At the very least though, I learned the hard way that automating everything does not solve all your problems. Wednesday, August 12, 2009 1:36 PM by WesleyB77 I couldn’t agree with you more.

We had tests in a previous team of mine that tested parts of MSTIME (Microsoft Timed Interactive Multimedia Extensions) by comparing a known good screenshot to a screenshot taken using the binary being tested. I arrived a few years after the code was written, and the nightmare was not over. There were a lot of problems. Due to the sheer magnitude of environments and platforms and machines being tested, there were always a high number of false positives.

Sometimes certain video cards on the machine would colour a pixel differently (FAIL), or a frame would be skipped (FAIL), or it would render slowly (FAIL), or ClearType was enabled on the machine (FAIL), or (FAIL) or (FAIL) or (FAIL) or (FAIL).

It was very frustrating. At the very least though, I learned the hard way that automating everything does not solve all your problems.

Wednesday, August 12, 2009 1:36 PM by WesleyB77

]]>