The Mechanical Turk Blog

Mechanical Turk - Categorization HIT Critique

The HIT I'll critique today is a Categorization HIT.  It's asking Workers to categorize a wide range of facebook pages from "star wars" to "jersey shore quotes" to "cinema."

Categorization is one of the most common use cases for Mechanical Turk.  A categorization HIT is one that  asks a Worker to select from a list of options.  It can include Classification HITs, for instance "What kind of product is this?" "Apparel" or "Electronics". Categorization HITs also include Moderation HITs for instance, "Does this image contain nudity?" or "Are there people in this picture?".  They also include Data Cleansing HITs for instance "Are these two items the same?"

There are several simple changes that I'd recommend to today's HIT (see HIT at bottom of post) that will increase the likelihood of accurate results and quicker turnaround time from this HIT.

1) Limit the number of categories.  Ideally Workers should be able to keep track of all the category choices in their minds.  Science tells us that's 7 - 10 items.  This HIT asks Workers to select from over 150 different categories! That's too many. 

I'd suggest you break this into a multi-step HIT.  In the first HIT, have Workers pick the "main category" for the item (for instance "Local Business" or "Artist/Band/Public Figure".)  Create a follow on HIT that asks Workers to pick the sub-category.  So if HIT #1 concluded the page is a local business, HIT 2 would ask Workers to categorize this into one of the sub-categories within "local business". 

2) Make sure your categories don't overlap. Or if they do overlap provide guidance on which category the Worker should choose.   For instance in the HIT below, under "Brand/Product" there is a category called "Baby Goods" as well as a category called "Clothing." What's the right category for Kids Ecco shoes? Is it Baby Goods or Clothing? What's the right category for Jerry Seinfeld -- Actor? Comedian? Entertainer? 

Likewise there are 2 "Education" categories.  One is under "Company/Business/Institution" and the other is under "Local Business".  Where should I put Sylvan Learn Center? Where should I put a neighborhood pre-school?

3) Provide definitions for the categories.  I think I know the difference between a "Financial Service" and a "Business Service" but I have no idea if it's the same definition you're using. 

When providing definitions for categories, it's a good idea to give an example of the most common mistake.  For instance, "Cameras belong in Photo/Camera category NOT in Consumer Electronics.  Phones WITH cameras belong in the Consumer Electronics."

 4) The instructions indicate that multiple Workers will complete the HIT and Assignments will be approved or rejected based on agreement between Workers.  Given the ambiguity associated with the categories in this HIT, I'd expect the best Workers to opt out and not complete these.  Great Workers don't want to risk rejections when instructions are unclear.

I encourage you to fine tune their HIT design and instructions until the HIT produces 90% accuracy consistently. Until then, you won't know if Worker discrepancy should be attributed to a Worker or to your HIT design.

 Here's a snippet of the HIT:

 

 

Simple tricks for Managing Workers on Mechanical Turk

 

Did you know that you can manage your Workers from the Mechanical Turk Requester website?  This feature has been available for a while now but we find there are a few Myths that cause confusion:

Myth: I can't manage Workers via the Requester website if I create my HITs via the API.

Reality:  You CAN manage Workers from the Requester website regardless of how your HITs are created.  Just visit the Manage>Workers section.

You'll see the list of Workers who have worked on your HITs (and their approval rate on your HITs).  Click on an individual Worker to bonus, block or grant the Worker a Qualification.

 

Myth: I have to page through LOTS of pages of my Workers to find the one I want to bonus, block or assign a Qualification.

Reality: Actually there is a really simple way to manage a Worker without paging through the crowd.  The Manage Workers page includes a list of every Worker who has ever submitted a HIT for you.   If you click on an individual's ID you will be taken to the page that lets you manage this individual Worker.  The url for this detail page includes the Worker's ID in the format https://requester.mturk.com/workers/WORKERID.  Here's an example for a specific Worker:

If you want to bonus a different Worker (suppose it's a Worker with Worker ID 3ABQ167TW). Put this Worker ID after the last backslash in the url (resulting in the url https://requester.mturk.com/workers/3ABQ167TW.)  Once you're on the page for that Individual Worker, you can bonus, block and manage that Worker's Qualifications.

 

Myth: I have to block Workers individually and that takes too long.

Reality: You can take action on multiple Workers at once by using the downloaded csv file.  To do this download the file of your Workers from the Manage>Workers page.  Follow the instructions from the download wizard to block/unblock Workers or Grant/Revoke a Qualification by marking the appropriate columns and rows in the file and upload it to us.  We'll do the rest.  (Sorry you can't grant bonuses this way.)

 

 

 

 

Work hard. Have fun. Make history! The Amazon Mechanical Turk team is hiring.

Amazon Mechanical Turk is a pioneer in crowdsourcing and work distribution.  We are changing the way work is done.  And we're growing. We're looking for Software Engineers, Product Managers, Designers and Business Development professionals to join the Mechanical Turk team. Check out our current job openings here.

 

Come work on Amazon Mechanical Turk if you want to solve hard problems, work with smart people, have fun and make history!