The Mechanical Turk Blog

HIT Critique: Design Tips for Improving Results

Editor’s Note: This post is part of our HIT Critique series in which we feature HITs selected from the Mechanical Turk Marketplace and provide feedback intended to help Requesters design better templates and instructions. Below, Steven Ma, a senior designer at Mechanical Turk, provides his insight on a recent HIT from the marketplace.

This HIT asked Workers to research questions using specific webpages as resources by and select the correct answer from a list of potential options. 

There were a few things that this Requester did well that that warrant highlighting:  

• First, the pay of rate of $.03/HIT was competitive relative to going rates for similar tasks in the marketplace in terms of both complexity and task type.  Competitive pay is important for attracting and engaging Workers.

• The Requester anticipated all potential scenarios and included options for when a correct answer was ambiguous or not available (see image below).

• This Requester made it easy for Workers to read instructions by providing key instructions under the “Important” section at the top of the HIT, while moving secondary information (such as efficiency tips) to the bottom. By separating longer instructions, Requesters can avoid “information overload” – which sometimes causes Workers to skim or ignore important details.

All of that said, this HIT also has room for improvements that could significantly impact results quality.  Here are a few recommendations:

• Target Workers with proven performance history. This Requester selected Workers who have submitted 50 approved assignments.  Achieving this milestone could take some Workers less than one minute – meaning that many of the Workers doing these HITs have no reputation and may perform poorly. To target Workers with proven experience, we recommend using Masters (Add blurb on what Masters are).   Alternately you could use Workers who have completed at least 10,000 approved assignments at an approval rate of 95% or higher. 

• Be clear and specific in your instructions. We find there to be a strong correlation between results quality and the precision of instructions. For instance, rather than writing “If you can’t find the answer then …”, I would advise this Requester to write “If the answer is not provided on the page, then ...”.

• Build in quality assurance mechanisms. Be sure to have a strategy to measure quality.  For instance, API Requesters can use Known Answers to conduct real-time accuracy audits as Workers submit assignments (see blog post on Known Answers). Manual inspection of the results is the most basic quality measurement but it doesn’t scale well. Alternatively, having a second Worker rate the original answer is more leveraged method.  There are many papers that have been written on quality measurement techniques on Mechanical Turk for various use cases.

• Test and iterate on the design. Last, but certainly not least, we recommend Requesters publish smaller test batches first to iron out potential wrinkles in HIT design or instructions. It’s typically the most cost effective approach to getting Worker feedback on your design before publishing large batches of HITs.

Using Mechanical Turk for Categorization

Response to our last HIT critique was so positive, we’re going to continue offering suggestions here.

This HIT asks Requesters to categorize a web page.

There are several things I like about this HIT:

1)      The HIT requires Workers have the Categorization Masters Qualification.  Workers with this Qualification (aka Masters) have demonstrated consistent, accurate performance on categorization HITs from many Requesters.  This is the fastest way to get consistently accurate results.

2)      Not only does the Requester provide the link to the website, he also loads the website into an IFrame window within the HIT so the Worker doesn’t need to switch tabs or windows.  The Worker has everything they need to complete the HIT right in front of them.  This makes Workers more efficient and improves their effective hourly rate without increasing your costs.  Everybody wins!

3)      The Worker needs to select from a list of 10 categories. 

 

These are placed so that the Worker can see them all without scrolling.  As a Worker hovers over a category a pop-up provides the definition of the category and several examples.  And I LOVE that the submit button is placed right near the categories selection so Workers don’t have to scroll to submit.  That’s a BIG Plus for Workers!

4)      The “Notes” are really instructions.  I like that they provide a clear example:

Without these instructions, I might have put the etsy blog page into the Retail category because etsy is primarily a retailer.  The example makes it clear I should look only at the page provided and categorize the broader site.

5)      I did several of these HITs in preparing this blog post. (Yes, I have earned the Master Qualification the hard way, just like all Master Workers).   One of the websites I categorized didn’t load properly so I selected “Error Loading” and  submitted.  Only after hitting “submit” did I see this additional instruction:

I’d recommend this be added to the Instructions/Notes section.   It would have saved me from a wrong answer.

This HIT follows many of our best practices for Qualifying Workers, providing clear instructions with examples and making Workers efficient by limiting scrolling and multiple windows. Do your HITs?

Mechanical Turk - HIT Critique

We’re often asked to critique a Requester’s HIT to make it better.  This can be of great benefit to the individual Requester (and the Worker who is on the receiving end of the HIT) but it doesn’t allow others to learn.  So I thought I’d take a HIT that’s currently available in the system and critique it.

Below is an image of a HIT that was published for Workers on Mechanical Turk. I don’t know this Requester but my guess is they're goal is to generate metadata to improve the search experience on a website. The HIT does some things quite well but with a few easily adjustments I think this Requester will get better results.

 

Design
I LOVE that the Requester provides a link to the search site with the product pre-populated rather than make the Worker type in the product.  This limits operator error (typos) and makes Workers more efficient.

I also LOVE that the link opens in a new tab. When the link opens inside the HIT window, the website replaces the HIT and the Worker can't see the questions anymore.  Opening the reference link in a new tab makes it easier for the Worker to flip between the HIT and the website you provided for reference. 

 

Worker Qualification
This HIT is available to any Worker with a 96% or higher approval rate.  This includes Workers who’ve only completed 2 Assignments and achieved a 96% approval rate on them.  I think you want more proven Workers than that.  I would recommend you either use Mechanical Turk Masters (Workers who have demonstrated accuracy in completing moderation and categorization HITs) or stick with our recommended Qualifications for other types of HITs:
                                       HIT Approval rate is greater than or equal to 95%
                                                                       AND
                                       Number of HITs Approved is greater than 1000 

 

Question #1
The instructions for the first question, ask Workers to select which of the provided keywords fit the product. I like that Workers are told that it’s ok if they select NONE of the provided keywords.  Otherwise Workers would likely return the HITs that have no relevant keywords for fear that if they submitted with no keywords selected the work would be rejected.

Also, the instructions don’t tell the worker how to determine which key words are appropriate.  For example, key words for a product are sometimes listed in the product description on the link or inferred from looking at the picture from the link.  Provide examples of what is a good keyword and what is not.   

Question #2
The second question seems ambiguous to me.  Is it mandatory that I provide descriptors?  How many do I need to provide to get paid?  Will I get a bonus if I provide more than required number?

Also you’ll get better descriptors if you explain what makes a “valid descriptor.”   Something like:

• Valid descriptors include generic product categories (for instance if the link is to a TV set then “TV” would be a valid descriptor). 
• Valid descriptors also include features that differentiate this product from others in the generic category (for instance a “LCD” or “Plasma” would be valid for certain TVs.) 
• Color, brand name, accessories and dimensions are not valid descriptors.