Mechanical Turk is pleased to announce the Categorization Application (App) to take the guess work out of getting fast, accurate results on categorization projects. Before the Categorization Application, each Requester had to become an expert at designing categorization HITs in html, qualifying Workers, pricing HITs and analyzing the results. The Categorization Application automates many of these steps by leveraging the best practices and insights we’ve learned from over 5 years of experience in helping Requesters get better, faster results.
The Categorization Application includes everything you need to be immediately successful – predesigned HITs that don’t require editing in HTML, pre-qualified Master Workers who have demonstrated expertise in categorization HITs, price recommendations based on comparable HITs in the Mechanical Turk marketplace and analysis tools that make it easy to verify results and identify gaps in instructions that may be causing Worker confusion.
Categorization is one of the most common use cases for Mechanical Turk. A categorization HIT is one that asks a Worker to select from a list of options. Categorization projects are used by many companies for many different reasons. For instance, a retailer‘s categorization project may ask Workers to "Select the product category for this item.” A listing or directory service may ask Workers “Is this the correct web address for this business?” Or a stock photo site may ask Workers to “Is this image a photo or illustration?”
The Categorization App was designed to eliminate the learning curve that Requesters face so that Requesters can get good results on their first project. For instance, many Requesters aren’t sure how to qualify their Workers – is a 90% approval rate good enough? Should I create a test for Workers to complete before they do my HITs? The Categorization App sends HITs to Mechanical Turk Categorization Masters – Workers who have demonstrated accuracy in categorization HITs. So Requester’s don’t have to devise a unique qualification plan. Likewise Requesters often don’t know how to price their HITs. Requesters typically set their HIT price by looking at other HITs in the marketplace without knowing if HITs are priced appropriately and being completed by Workers or which HITs require similar amounts of effort. The Categorization App provides Requesters with a pricing recommendation based on similar HITs that Masters are completing in the Marketplace.
The Categorization App is available on the Requester website and guides you through 4 simple steps to create your categorization project. The project I’ll create asks Workers to categorize images for a stock photography website. The first, step asks for a project name and the question I want to ask Workers.
Next I am asked to provide my categories. I want Workers to indicate if there are people in the image or not, so the categories are “People” and “No People”. Previously I would have to edit HTML to add radio buttons and name the various categories. But now I just enter the category names in the UI and the HTML is generated for me. The resulting HIT is structured to eliminate scrolling, maximize Worker productivity and reduce Worker error. I can also add instructions for Workers.
I can also add instructions for Workers.
Now I need to provide the images that Workers will look at. I have an excel file that contains the image urls (conveniently stores on S3 using the Firefox plug in). I convert this file to a csv file using the “Save As” in excel and upload it to Mechanical Turk via the Categorization App UI. I am presented with the contents of my data file and select which fields should be shown to Workers. My file contains a unique image ID which my database uses to track each image but Workers don’t need to see that, so I’ll indicate that the unique image ID shouldn’t be shown to Workers. Also, I indicate that the data I want shown is a link to an image. This will make sure that the image is displayed within the HIT instead of being displayed as a text field or a link that Workers need to click on.
Now I can see how my HIT will look with the actual data that I just uploaded. This is how the Worker will see the HIT. The HITs are automatically formatted to improve Worker productivity by eliminating scrolling and allowing instructions to be collapsed or expanded by the Worker.
The Categorization App recommends a Worker reward price based on the complexity of the categorization task and the price of similar HITs on the Mechanical Turk marketplace. Previously, you would have to search through the HITs on Mechanical Turk to see how similar HITs being done by similarly qualified Workers were priced. And even if you found similar HITs, you would have no way of knowing whether these HITs were priced attractively to Workers and were actually being completed. The Categorization App lets me can change the price if I want to. I want my project done quickly so I’m going to increase the reward I pay Workers.
Each HIT will be completed by two Master Workers because experience shows that when two Masters agree on an answer, there is a higher level of accuracy. And it also makes it easy to identify issues with instructions or guidelines. For instance if Masters are often split between categories, perhaps the guidelines are not clear enough.
My HITs are ready to be published. I’ll receive an email when the project has been completed. The email contains a link to the Requester web site so I can pick up my results. I can download a csv file which contains the result for each of the HITs that Workers completed. The Result Analysis tools provide important statistics for instance how often Workers agreed on an answer.
For my project, Workers agreed on the categorization for most of my images. But for 10 images, Workers disagreed as to whether an image contained a person. When I look at the specific images that caused Workers to disagree, I see that these images contain “people parts”. My instructions didn’t tell Workers what to do if there was a “finger” on the edge of a picture. I’ll clarify that in my instructions for my next batch of images so my results will continue to improve.
The Categorization App provides everything you need to get fast, accurate results using Mechanical Turk. It was designed to eliminate the “learning curve” that new Requesters face. It leverages years of experience from Amazon and other Requesters so you can be successful from Day One. I’d like to thank all the Requesters who provided feedback and input in helping us build this!
The Categorization App is available on the Mechanical Turk Requester web site. To learn more visit requester.mturk.com
Comments