The HIT I'll critique today is a Categorization HIT. It's asking Workers to categorize a wide range of facebook pages from "star wars" to "jersey shore quotes" to "cinema."
Categorization is one of the most common use cases for Mechanical Turk. A categorization HIT is one that asks a Worker to select from a list of options. It can include Classification HITs, for instance "What kind of product is this?" "Apparel" or "Electronics". Categorization HITs also include Moderation HITs for instance, "Does this image contain nudity?" or "Are there people in this picture?". They also include Data Cleansing HITs for instance "Are these two items the same?"
There are several simple changes that I'd recommend to today's HIT (see HIT at bottom of post) that will increase the likelihood of accurate results and quicker turnaround time from this HIT.
1) Limit the number of categories. Ideally Workers should be able to keep track of all the category choices in their minds. Science tells us that's 7 - 10 items. This HIT asks Workers to select from over 150 different categories! That's too many.
I'd suggest you break this into a multi-step HIT. In the first HIT, have Workers pick the "main category" for the item (for instance "Local Business" or "Artist/Band/Public Figure".) Create a follow on HIT that asks Workers to pick the sub-category. So if HIT #1 concluded the page is a local business, HIT 2 would ask Workers to categorize this into one of the sub-categories within "local business".
2) Make sure your categories don't overlap. Or if they do overlap provide guidance on which category the Worker should choose. For instance in the HIT below, under "Brand/Product" there is a category called "Baby Goods" as well as a category called "Clothing." What's the right category for Kids Ecco shoes? Is it Baby Goods or Clothing? What's the right category for Jerry Seinfeld -- Actor? Comedian? Entertainer?
Likewise there are 2 "Education" categories. One is under "Company/Business/Institution" and the other is under "Local Business". Where should I put Sylvan Learn Center? Where should I put a neighborhood pre-school?
3) Provide definitions for the categories. I think I know the difference between a "Financial Service" and a "Business Service" but I have no idea if it's the same definition you're using.
When providing definitions for categories, it's a good idea to give an example of the most common mistake. For instance, "Cameras belong in Photo/Camera category NOT in Consumer Electronics. Phones WITH cameras belong in the Consumer Electronics."
4) The instructions indicate that multiple Workers will complete the HIT and Assignments will be approved or rejected based on agreement between Workers. Given the ambiguity associated with the categories in this HIT, I'd expect the best Workers to opt out and not complete these. Great Workers don't want to risk rejections when instructions are unclear.
I encourage you to fine tune their HIT design and instructions until the HIT produces 90% accuracy consistently. Until then, you won't know if Worker discrepancy should be attributed to a Worker or to your HIT design.
Here's a snippet of the HIT: