We’re often asked to critique a Requester’s HIT to make it better. This can be of great benefit to the individual Requester (and the Worker who is on the receiving end of the HIT) but it doesn’t allow others to learn. So I thought I’d take a HIT that’s currently available in the system and critique it.
Below is an image of a HIT that was published for Workers on Mechanical Turk. I don’t know this Requester but my guess is they're goal is to generate metadata to improve the search experience on a website. The HIT does some things quite well but with a few easily adjustments I think this Requester will get better results.
Design
I LOVE that the Requester provides a link to the search site with the product pre-populated rather than make the Worker type in the product. This limits operator error (typos) and makes Workers more efficient.
I also LOVE that the link opens in a new tab. When the link opens inside the HIT window, the website replaces the HIT and the Worker can't see the questions anymore. Opening the reference link in a new tab makes it easier for the Worker to flip between the HIT and the website you provided for reference.
Worker Qualification
This HIT is available to any Worker with a 96% or higher approval rate. This includes Workers who’ve only completed 2 Assignments and achieved a 96% approval rate on them. I think you want more proven Workers than that. I would recommend you either use Mechanical Turk Masters (Workers who have demonstrated accuracy in completing moderation and categorization HITs) or stick with our recommended Qualifications for other types of HITs:
HIT Approval rate is greater than or equal to 95%
AND
Number of HITs Approved is greater than 1000
Question #1
The instructions for the first question, ask Workers to select which of the provided keywords fit the product. I like that Workers are told that it’s ok if they select NONE of the provided keywords. Otherwise Workers would likely return the HITs that have no relevant keywords for fear that if they submitted with no keywords selected the work would be rejected.
Also, the instructions don’t tell the worker how to determine which key words are appropriate. For example, key words for a product are sometimes listed in the product description on the link or inferred from looking at the picture from the link. Provide examples of what is a good keyword and what is not.
Question #2
The second question seems ambiguous to me. Is it mandatory that I provide descriptors? How many do I need to provide to get paid? Will I get a bonus if I provide more than required number?
Also you’ll get better descriptors if you explain what makes a “valid descriptor.” Something like:
- Valid descriptors include generic product categories (for instance if the link is to a TV set then “TV” would be a valid descriptor).
- Valid descriptors also include features that differentiate this product from others in the generic category (for instance a “LCD” or “Plasma” would be valid for certain TVs.)
- Color, brand name, accessories and dimensions are not valid descriptors.
Comments