This is the first post in a new series of blog posts called “Cooking with Sharon”. The series will identify tips, tricks and best practices for getting quality results from Mechanical Turk.
In my conversations with Requesters, I am often asked “what are the tricks for getting great quality from Mechanical Turk?” The good news is there are many ways. But that’s sort of like my mother-in-law giving me her “recipe” for her delicious marinara sauce in which all ingredients are added “to taste”.
Fortunately there is a formula for getting great quality from Mechanical Turk. The first step to good quality is to ask a good question. This may seem obvious, and you’re probably thinking “my question is fine! That’s not the issue I’m having.” Here’s an example that changed my mind: Is a Miata a Sports Car?
I like to ask this question when I speak about Mechanical Turk. Inevitably half the audience says yes and the other half says no. (And yes it’s usually divided on gender lines.) In order for a Worker to answer this question accurately (and with a high probability that multiple Workers would agree on the same answer) they need to know what you mean by a “Miata” and a “Sports Car”.
It’s easy enough to look up the term Miata on the internet so even if the Worker is in geography where the Miata isn’t sold (India), or where they are sold under a different name (Japan or Europe) it’s likely the Worker will be able to find out what a Miata is. It would be more helpful if you provided the specific model and the specific year as not all Miatas are created equal.
Unless you want a Worker’s “opinion”, you need to define what makes a car a “Sports Car”. Do you mean does a Miata accelerate from 0mph to 60 in under 5 seconds? Or do you want to know if a Miata meets the Merriam-Webster definition (“a low small usually 2-passenger automobile designed for quick response, easy maneuverability, and high-speed driving”)?
The first step to getting good results from Mechanical Turk is to make sure your question is clear. So if you’re asking Workers “Is this image offensive” you shouldn’t be surprised that if Workers answers vary greater. If instead you ask “Does this image contain nudity”, you’re more likely to get more accurate answers.
-- Sharon
Sharon Chiarella is a Vice President at Amazon, responsible for running Amazon Mechanical Turk. And for the record, she can’t cook.
Comments