Editor’s Note: This post is part of our HIT Critique series in which we feature HITs selected from the Mechanical Turk Marketplace and provide feedback intended to help Requesters design better templates and instructions. Below, Steven Ma, a senior designer at Mechanical Turk, provides his insight on a recent HIT from the marketplace.
This HIT asked Workers to research questions using specific webpages as resources by and select the correct answer from a list of potential options.
There were a few things that this
Requester did well that that warrant highlighting:
- First, the pay of rate of $.03/HIT was competitive relative to going rates for similar tasks in the marketplace in terms of both complexity and task type. Competitive pay is important for attracting and engaging Workers.
- The Requester anticipated all potential scenarios and included options for when a correct answer was ambiguous or not available (see image below).
- This Requester made it easy for Workers to read instructions by providing key instructions under the “Important” section at the top of the HIT, while moving secondary information (such as efficiency tips) to the bottom. By separating longer instructions, Requesters can avoid “information overload” – which sometimes causes Workers to skim or ignore important details.
All of that said, this HIT also has room for improvements that could significantly impact results quality. Here are a few recommendations:
- Target Workers with proven performance history. This Requester selected Workers who have submitted 50 approved assignments. Achieving this milestone could take some Workers less than one minute – meaning that many of the Workers doing these HITs have no reputation and may perform poorly. To target Workers with proven experience, we recommend using Masters (Add blurb on what Masters are). Alternately you could use Workers who have completed at least 10,000 approved assignments at an approval rate of 95% or higher.
- Be clear and specific in your instructions. We find there to be a strong correlation between results quality and the precision of instructions. For instance, rather than writing “If you can’t find the answer then …”, I would advise this Requester to write “If the answer is not provided on the page, then ...”.
- Build in quality assurance mechanisms. Be sure to have a strategy to measure quality. For instance, API Requesters can use Known Answers to conduct real-time accuracy audits as Workers submit assignments (see blog post on Known Answers). Manual inspection of the results is the most basic quality measurement but it doesn’t scale well. Alternatively, having a second Worker rate the original answer is more leveraged method. There are many papers that have been written on quality measurement techniques on Mechanical Turk for various use cases.
- Test and iterate on the design. Last, but certainly not least, we recommend Requesters publish smaller test batches first to iron out potential wrinkles in HIT design or instructions. It’s typically the most cost effective approach to getting Worker feedback on your design before publishing large batches of HITs.