Cheap User Testing with Amazon Turk
After writing clean, cross-browser code and testing it in all browsers.. what if issues show up on others’ computers and not yours? Recently this happened to me and I was baffled.
Cross-Browser Testing
Upon testing my code in IE6+ through a virtual machine, Firefox, Safari and Chrome.. it all worked. Yet when I handed off the code to the client, they replied, “the submit button isn’t working”. So I went back to doing further testing.
Friend Testing
Friends tested the website and it worked for them. Yet, the client still could not submit the form on their computer. I was now infuriated at this mystery.
After some changes, I revamping my code and making sure every “t” is crossed and “i” dotted. The markup was now perfect and validated with w3c. Still, after sending it through to the client it still didn’t work for them! “Wow?!” I thought, “what do I do now?”
The Problem Still Exists. “What Now?”
At this point I was mad, not at the client but at the fact that I had no idea what was causing the issue. I then said these words to the client:
If I could have 100 PC computers from all around the world to test your website to find the issue, I would pay for it. I am riled up about this and want to fix the issue at all costs.
The Amazon Turk, “Testbed”
To my complete amazement, I actually found out a way to have 100 PC computers from all around the world testing my website ! Welcome to the world of Amazon Turk . For a total of $5, I had just about 80 users go through my tests and give me feedback.
My Process of Testing Through Amazon Turk
I came up with a process that made it easy to report back to the client with findings and numbers
- Setup a test to gather feedback on using the website. Be sure to isolate the code so that you only have the code on it that you think might be causing an issue. Save each file used in the test for future reference.
- Register and pre-pay around $5 on Amazon Turk.
- Created a HIT (Human Intelligence Tasks) on Amazon Turk that details the test. Use the default template and make sure you have a textarea on the HIT for them to report their findings.
- After getting results back, summarize the findings into a paragraph that pinpoint reported issues.
- Based on the summary, create an action item in order to develop a solution.
- Based on the action item, create a new test with the solution.
- Post the new HIT / test on Amazon Turk until your desired results are achieved.
Failing at First
My first HIT test didn’t go well, 75% of the users failed. But, I got great feedback and found a solution based on the feedback.
Resolving Infrequent Problems
In test rounds 2 and 3, I found infrequent problems based on specific cases with users. I made my error messages very specific so that users could copy and paste the results into the Amazon Turk results so that I could easily resolve the issue. These issues normally wouldn’t surface in the testing process unless the website was launched for a while and someone told us about it. Even then, it would have shown up so infrequently that I’m not sure we would have bothered.
Make your error messages specific, so users can report exactly what happened.
From 25% Completion, Up to 96% Completion!
After 4 rounds of tests, the completion results went up to 96% completion rate! During one of the tests, the user said the page didn’t show up. So the website must have temporarily been down and I’m not worried about that one failed test out of so many successful test.
9 comments
Hey Marc, really great article. I’m curious as to some of the technical details of how you pulled it off. What made it easier to identify the commonalities in the problem cases? How did that help you narrow down the HIT after each round of testing?
You left us hanging…why was the submit button not working for your client?
Nate: Thanks, glad you enjoyed it.
When looking at the results, people tell you the same error behavior over and over again. Like, “I entered my data and then the submit button didn’t work”. Then you can skin down the code to only contain things that deal with the form interaction and JavaScript validation. Then post the code to a separate URL and send the people through until you fix it.
Now if you have a server side error or MySQL error, then make sure to make that error message explicit. Tell them to copy and paste it… “Please copy and paste this message: $error”. Then you can see in the SQL statement what data they inputted and update your server side logic accordingly.
The idea of sending random users through is great for testing things in ways you never thought of…
Matt: I returned true if validation was successful instead of simply returning false. It was an oversight on my part. But through user testing I found smaller edge cases that wouldn’t have turned up if I hadn’t sent so many users through.
Augh, I hate it when things just don’t work for clients.
I’ve got a guy on this 15MB webmail hosted by his internet provider. I’ll send him logos of 500KB or more, and so his inbox fills right up and I get bounce message. I put them on my server, instead, and sent him links, but for some odd reason his computer would give him a blue screen and crash if he tried opening them.
I have found more time than not that a client’s machine undoubtedly have something go wrong if at all possible. One of my biggest annoyances is when there are DNS “outages” caused by DOS attacks on companies like Network Solutions, which create a outage for section of the country to certain IP Blocks.
The client always feels as if it has to be something you did or your software that must be at fault, unaware of the true complexity of the internet and its systems. Anyways, TURK is a pretty neat concept, although I think Amazon is guilty over stating its usefulness, but other wise interesting.
Thanks for sharing Marc. I recently sent emails to at least 5 contacts from around the world to help us test a GeoIP solution. This would have been ideal (and given us a much wider audience). Good stuff.
Marc, that’s brilliant!! Amazon Turk seems ideal for cheap iterative user testing!
Cross Browser compatibility is always pain in the a… I remember that 2-3 years ago when I designed a site it always was ok in firefox and opera. But in IE it looked differently. Now in new IE versions some of the problems are solved. But there allways will be some minor issues, because each browser is using different engine. Amazon Turk is a great site, and it will without doubt be very usefull for browser site/application testing. Thank you
Were you able to target specific browers/OS’s on Turk?