The Mechanical Turk Blog

Mechanical Turk Workers Help MIT Scientists Identify Endangered Species

If you watch animal documentaries or TV shows on Animal Planet, you may be familiar with how scientists use physical tags to track animals in the wild. However, sometimes tagging can be difficult or intrusive on an animal’s environment. A recent article from LaboratoryEquipment.com highlights how scientists at MIT are using photographs to track animals, along with algorithmic pattern recognition and Mechanical Turk Workers to identify animals in the images collected.  

While capturing photographs is less intrusive, reviewing the shots collected is an arduous task. To keep scientists in the field (as opposed to manually reviewing images in front of a computer), Ravela’s team developed a software system that uses algorithmic pattern recognition to identify animals based on their unique features, such as stripes or spots. 

Ravela’s team was able to successfully identify potential image matches algorithmically, so the team turned to Mechanical Turk to create a scalable process for manual verification.  To identify the most qualified Workers for the task, each Worker was presented with four images, where the correct answer was known for three.  If the Worker correctly matched the known images, then the system would automatically trust the accuracy fourth result. 

Read about MIT’s experiment at on LaboratoryEquipment.com.  To learn more about how you can use Known Answers to monitor performance, see our previous post on the topic or read more in our Mechanical Turk’s API documentation.   

Featured Requester: Discover How comScore Benefits from Mechanical Turk

ComScore recently got in front of the camera to share the value their organization derives from Mechanical Turk.  According to Mike Brown, CTO, “comScore started using Mechanical Turk features three years ago … to quickly and easily classify ads online.” With the help of data classifications from Workers, comScore is able to provide their Ad Metrix customers with detailed competitive insights into display advertising activity and trends online. Watch the video below to hear more:

 

For additional detail on comScore’s use of Mechanical Turk and their results - tune into Brown’s re: Invent presentation.

 

Worker Corner: What Makes a Great Requester?

Requesters often ask us for recommendations on how to best attract and engage Workers in Mechanical Turk’s marketplace. We’ve shared recommendations for building Requester reputation in previous posts.  Today, we wanted to revisit the topic, this time sharing insights directly from Workers on what makes a Requester great.

In order to collect insights from Workers, we posted a HIT to Mechanical Turk asking Workers to share their perspective on actions Requesters can take to make their HITs more attractive.  Based on previous surveys we’ve conducted, we know that a primary motivation for Workers is monetary reward. However, in this exercise, we wanted to explore other approaches that Requesters can use to build their relationship with Workers outside of compensation. 

So, what did Workers have to say? In reviewing their feedback, three key themes emerged:

Communicating with Workers by addressing questions and providing feedback can help identify issues in HIT design and improve the quality of results.  

“Communication! A great requester communicates regularly with their workforce. That means answering emails promptly or connecting with us in the forum to address questions and concerns, announce changes and upcoming work, explain work and rejections policies in more detail, etc. Communication makes everything easier, for everybody involved. Nothing worse than being rejected or even blocked and never [knowing] why.”

“The great requestor gives feedback for good work and bad. If there is a way that a worker can improve their work, constructive feedback is the only way to know how to do so. Good feedback inspires more good work.”

“Requesters who take the time to communicate are at the top of my list.  Communication allows the workers to have a better idea of what the requester is looking for and in turn provides the requester with better work.  It doesn't take much just an email or two and has an enormous return on investment.”

"An open line of communication between Worker and Requester can improve any issue on either side. Being open and responsive to feedback is a great sign that a requester is interested in improving the quality of work and the quality of worker/requester experience on MTurk in general.”

“Having an active presence on [a Mechanical Turk Forum, like TurkerNation] affords a Requester the opportunity to be a great Requester. The ability to communicate with a group of Workers, as well as recruit new Workers, is a time saving, cost-effective way to advertise tasks, clarify or modify instructions and/or qualify Workers. A Requester that is willing to do these things is not only a great Requester, but a great Requester [who] will obtain quality results on the tasks they publish because the Workers will be waiting for their HITs to launch.”

Approving (or Rejecting) assignments in a timely manner helps Requesters build trust with Workers.

 “Pay promptly. If you have a lot of HITs and you are a new Requester to me, I will be reluctant to do a large batch of hits before I know that I will get paid. If I start doing a batch of HITs for you, and within a few hours I see "payment pending" (meaning that you’ve approved my work) for the first 100 or more HITs, I will have no reservations in spending the entire day and the next day working on your HITs.”

“I prefer hits that [are approved] quickly. It helps me to judge my work. If there is a problem I can fix it if need be. Plus getting the money quickly helps. Some HITs take days to get approved. By then I have moved on to other hits so if there is a problem it is hard to fix."

Mistakes happen, but quickly reversing mistakenly rejected assignments can encourage Worker loyalty. 

“Great Requesters know when to admit that they have screwed up. I've had many times where I felt that my work was unjustly rejected after I had spent a good amount of time on it. When I contact great Requesters about this type of issue they find a way to make it right, either by offering me a bonus for the rejected work or additional work that will earn back my trust and respect for them.”

“Learning how [Mechanical Turk] works, specifically reversing rejections. Rejections stay with a Worker forever even when the requester is at fault. When a Requester learns how the UI works and realizes they can reverse rejections, I'm more likely to stick with them on future tasks.”

“Quick and honest dispute resolution.  On several occasions I have had work incorrectly rejected.  When I have reached out to the Requester, they have responded quickly by double checking the error in question and either admitting the mistake on their part or acknowledging that the reject was in error.  They either reverse the rejection or they [issue a] bonus...”

To learn more about best practices for achieving results on Mechanical Turk and building your reputation with Workers, be sure to review the Requester Best Practices Guide.

Finally, thank you to all of the Workers who contributed feedback.  We appreciate your insight.  

-- Andrea

Introducing Multi-level Categorization and up to 100 Category Choices

Since we introduced the Categorization Application (App), Requesters have asked us to add support for more categories and multiple levels.  We’re pleased to announce that the Categorization App now supports ten top level categories with ten subcategories each, for a total of one hundred categories.  

The Categorization App incorporates the best practices and insights we’ve collected helping Requesters achieve success including everything you need to be immediately successful:

• predesigned HIT interfaces (no HTML required),
• the ability to customize your own categories,
• built-in tools that make it easy to review results, and
• Recommended pricing for categorization HITs.

To create HITs with two tiers of categories, the Categorization App guides you through four simple steps: creating the categories, uploading your data to be categorized, previewing your HIT and check out.  First provide your top-level categories.  Then, add a second level of categories by selecting “Create Subcategories” and entering the subcategories you wish to include under each top-level category.  In the example below, if a Worker is categorizing a sleigh bed, you would expect a result of “Furniture” as the top level category and “Beds” as the subcategory. 

Once you’ve provided the category names, you will be asked to provide instructions. It’s important to provide clear selection criteria for your categories to guide Workers and avoid confusion. We recommend providing Workers examples of items that should be included in or excluded from each category as additional guidance (see screenshot below).   

At checkout, the Categorization App recommends a Worker reward based on the price of similar HITs in the Mechanical Turk marketplace enabling Requesters to adjust pricing if necessary.

To check out the new Categorization App features, select “Categorization” under the “Create” Tab from the Mechanical Turk Requester home page.   

--Andrea

Twitter Improves Search with Real-Time Human Computation

Twitter is a great example of a company that relies on Mechanical Turk to improve real-time search. In a recent post to their engineering blog, Edwin Chen, a data scientist at Twitter, provides insight into how Mechanical Turk Workers improve their understanding of popular search queries as events happen.

In order to understand what users are searching for in real-time, Twitter developed a process to quickly identify trending search queries, send them to Workers on Mechanical Turk to be judged, and incorporate the data into their back-end models. For instance, when the search query “Big Bird” started spiking in association with a comment from Mitt Romney in a 2012 presidential debate, Twitter was able to quickly determine that users weren’t searching for tweets related to Sesame Street. Making this determination quickly enables Twitter to serve more relevant advertising, as opposed to ill-placed ads for Dora the Explorer.

Read Twitter’s post to learn more about how they were able to create a scalable model for collecting high quality human judgments on demand by building a custom pool of Mechanical Turk Workers using Qualifications. 

Note: This isn’t the first time we’ve talked about Twitter on the Mechanical Turk blog—we highlighted their work back in August with their launch of Clockwork Raven, an open source web application built on Mechanical Turk.

--Kelly

 

Reverse Rejected Assignments in the Requester User Interface

Today, we’re pleased to share that Requesters now have the ability to reverse a rejection by approving a previously rejected assignment in the Requester User Interface.  Sometimes mistakes happen, and we wanted to enable Requesters to correct their mistakes and ensure that Workers are paid for good work.

To approve a previously rejected assignment, download your batch CSV file by selecting “Download CSV” from the “Results” page for your batch.

In the downloaded CSV file, mark an assignment as approved by putting an “X” in the “Approve” column while removing text from the “Reject” column.  In the example below, I approved the assignments with the answers bed, cook and grass.  

Once you’ve marked the rejected assignments that you want to approve, upload the updated CSV file by selecting “Upload CSV” from the “Results” page for your batch.  

You will receive a confirmation to approve the assignments. By approving the assignments, the Worker’s statistics will be updated and they will be paid from your pre-paid balance, so please double check your account to ensure you have funds available.

If your HITs were created in the API, you can reverse a rejection by approving the assignment using the ApproveRejectedAssignment API operation.  More information on this operation is available in our API reference guide here.

We hope you enjoy this new features, and Happy New Year from Mechanical Turk!

--Andrea

Making HITs Available and Attractive to Workers

Many Requesters choose Mechanical Turk because the large Worker community available enables quick results on projects. So, what are the best methods to drive speed as a Requester? In some ways, attracting workers to your HITs can be likened to dating. Similar basic rules apply: Your HITs must be available and attractive to Workers.

By designing HITS that meet these criteria, not only will more Workers want to do your HITs—Requesters typically see higher throughput as a result.

Making HITs Available

Your HITs are constantly competing for Workers’ attention. For this reason, they need to be easy to discover, or rather, available to Workers.

Most Workers use search to target HITs and rarely review beyond the first page of results. Thus, it’s critical to select descriptive titles, relevant keywords and competitive rewards that make it easy for Workers to discover your HITs.

For example, I enjoy surveys, but prefer shorter surveys. In order to discover these sorts of opportunities, I searched for “short survey” and sorted results in descending order based on reward. Here’s the top HIT returned:

Why was this HIT the first result? The Requester listed matching keywords (“survey” and “short”) and is paying $3.50 per HIT, which is the most competitive reward given my search query.

Making HITs Attractive

Just as if you were creating an online dating profile, it’s important to make your HITs appear attractive while at the same time providing a title and description that accurately describe your task. Exaggerations may attract more Workers initially—however, misrepresenting HITs will ultimately tarnish your reputation in the Worker community. For example, adding adjectives like “easy” or “fun” can seem more inviting, but aren’t effective if they are not truthful representations.

Designing HITs that are efficient and enjoyable also has a significant impact on how attractive they are to Workers. Use the best practices outlined in the MTurk Best Practices for Requesters to keep the design of your HITs simple. HITs with confusing instructions, or complicated layouts that require a lot of scrolling are perceived by Workers to be less efficient and have a lower return on their investment of time.

Finally, attract Workers by pricing competitively based on the length of time and complexity of your HIT relative to other HITs available on Mechanical Turk. Workers are constantly evaluating their opportunity cost and will abandon your task for similar, but higher paid HIT if there is one available.

For example, consider the following two opportunities from a Worker’s perspective:

The pay is the same, but one takes roughly three times longer to complete. As a Worker, which one would you choose? More than likely, you would accept the first of the two options.

These are just a few easy strategies that are helpful to consider when looking to get your HIT noticed in a sea of eligible HITs in the marketplace. By focusing on making HITs more available (or discoverable) and attractive using the simple tactics outlined above, you can increase your chances of attracting great Workers, and subsequently, improve the speed of your results.

Improving Quality with Qualifications - Tips for API Requesters

Requesters consistently tell us that using Qualifications is one of the most effective strategies for optimizing the quality of results on Mechanical Turk. So, if you're an API user and unfamiliar with how to implement Qualifications, this post is intended to make it easy to get started.

How do Qualifications work? Qualifications enable you to direct your HITs to specific groups of Workers on Mechanical Turk. When you include a Qualification, it means that Workers who do not meet your Qualification standard cannot contribute to your HIT. As a Requester, you can use pre-defined Qualifications or create your own custom Qualification types (to learn more about creating Qualifications, see this post on the topic).

Mechanical Turk offers two types of pre-defined Qualifications--Masters and System Qualifications. We've discussed the value of Masters Workers in a previous post. For those new to the concept, Masters are an elite group of Workers with demonstrated accuracy across specific types of HITs in the Mechanical Turk Marketplace--specifically, categorization and photo moderation. Making the Masters Qualification a requirement is the easiest way to ensure that your work is completed by Workers with proven expertise in your task.  Keep in mind, your HITs do not have to be photo moderation or categorization—you can send any type of HIT to either Masters group.

In addition to Masters, Mechanical Turk offers several additional System Qualification types that you can leverage to direct your work to experienced Workers with proven performance. To direct your HITs to the most experienced Workers, use the “Number of HITs Approved” System Qualification. Based on Requester feedback, we recommend that you set your requirement at a level greater than 5000 approved assignments. This means that only Workers who’ve completed 5000 approved assignments will have access to your HIT.

In addition to targeting the most experienced Workers, setting your “HIT Approval Rate” System Qualification at greater than 95% will direct your work to the highest quality Workers as determined by their approval rate from other Requesters in the Mechanical Turk Marketplace. Requesters frequently tell us that the combination of these two Qualifications has a significant affect on the overall quality of results.

How can you implement these Qualifications? In the Requester API, the QualificationRequirement data structure describes a Qualification a Worker must have before the Worker is allowed to accept a HIT. The QualificationRequirement data structure is used as a parameter for the following operations:

• CreateHIT
• RegisterHITType

Learn more about QualificationRequirement and access Qualification TypeIDs in MechanicalTurk's API reference.

Additionally, in effort to make implementation easy, we've provided sample code for the following Qualifications in Ruby, .NET, and Java:

• Masters (photo moderation)
• Approval Rate
• Number of HITs Approved 

Ruby

require 'ruby-aws'
@mturk = Amazon::WebServices::MechanicalTurkRequester.new :Host => :Sandbox # remove the :host option to use production
# This is how to require a Photo Moderation Master
photoModerationId = (@mturk.host =~ /sandbox/) ? '2TGBB6BFMFFOM08IBMAFGGESC1UWJX' : '21VZU98JHSTLZ5BPP4A9NOBJEK3DPG'
mustBeImageModerationMaster = { :QualificationTypeId => photoModerationId, :Comparator => 'Exists' }
# This is how to build a qualification requirement for specific statistics
mustHaveHighApprovalRate = {
    :QualificationTypeId => Amazon::WebServices::MechanicalTurkRequester::APPROVAL_RATE_QUALIFICATION_TYPE_ID,
    :Comparator => 'GreaterThan',
    :IntegerValue => 95,
}
mustHave5kHits = {
    :QualificationTypeId => '00000000000000000040' # Worker_NumberHITsApproved
    :Comparator => 'GreaterThan',
    :IntegerValue => 5000,
}
# The create HIT method takes in an array of QualificationRequirements since a HIT can have multiple qualifications.
# Here, we're choosing to require two of the statistic-based qualification requirements we created above:
qualReqs = [mustHaveHighApprovalRate,mustHave5kHits]
# Alternatively, just require the Photo Moderation Master
qualReqs = [mustBeImageModerationMaster]
# Register the Hit Type
hitType = @mturk.registerHITType(
    :Title => title,
    :Description => description,
    :Keywords => keywords,
    :Reward => reward,
    :AssignmentDurationInSeconds => assignmentDurationInSeconds,
    :AutoApprovalDelayInSeconds => autoApprovalDelayInSeconds,
    :QualificationRequirement => qualReqs
)


.NET

    SimpleClient mTurkClient = new SimpleClient();
    List qualList = new List();
    QualificationRequirement qualImageMod = new QualificationRequirement();
    QualificationRequirement qualApprovalRate = new QualificationRequirement();
    QualificationRequirement qualNumHits = new QualificationRequirement();
    bool useMasters = true; // change the default based on preference


    if (useMasters)
    {
        // qualifications for Image Moderation
        if (mTurkClient.Config.WebsiteURL.Contains("sandbox"))
            qualImageMod.QualificationTypeId = "2TGBB6BFMFFOM08IBMAFGGESC1UWJX";
        else
            qualImageMod.QualificationTypeId = "21VZU98JHSTLZ5BPP4A9NOBJEK3DPG";
        qualImageMod.Comparator = Comparator.Exists;
        qualList.Add(qualImageMod);
}
else
{
    // setup the qualification for approval rate
    qualApprovalRate.QualificationTypeId = MTurkSystemQualificationTypes.ApprovalRateQualification;
    qualApprovalRate.Comparator = Comparator.GreaterThan;
    qualApprovalRate.IntegerValue = 95;
    qualApprovalRate.IntegerValueSpecified = true;
    qualList.Add(qualApprovalRate);


    // setup the qualification for number of HITs approved
    qualNumHits.QualificationTypeId = "00000000000000000040";
    qualNumHits.Comparator = Comparator.GreaterThan;
    qualNumHits.IntegerValue = 5000;
    qualNumHits.IntegerValueSpecified = true;
    qualList.Add(qualNumHits);
}


    // register the HIT Type, so that it can be used in later calls to CreateHIT
    string hitTypeId = mTurkClient.RegisterHITType("Blog Sample",
        "This is a HIT created by the Mechanical Turk .Net SDK",
        60*60, 60*60, 0.02m,
        "qualifications sample",
        qualList);


Java

// Choose Categorization Masters Qualification ID based on endpoint
if (service.getWebsiteURL().contains("sandbox")) {
    mastersQualID = "2F1KVCNHMVHV8E9PBUB2A4J79LU20F";
} else {
    mastersQualID = "2NDP2L92HECWY8NS8H3CK0CP5L9GHO";
}
HITProperties props = new HITProperties(propertiesFile);
// Create the Qualification Requirements
// Use the categorization masters qualification or limit the approval rate for Workers
Boolean useMasters = true;
if (useMasters) {
    int mastersQualIndex = 0;
    props.setQualificationComparator(mastersQualIndex, "Exists");
    props.setQualificationType(mastersQualIndex, mastersQualID);
} else {
    int highApprovalRateIndex = 0;
    props.setQualificationComparator(highApprovalRateIndex, "GreaterThanOrEqualTo");
    props.setQualificationType(highApprovalRateIndex,     com.amazonaws.mturk.service.axis.RequesterService.APPROVAL_RATE_QUALIFICATION_TYPE_ID);
    props.setQualificationValue(highApprovalRateIndex, "95");
    int mustHave5kHitsIndex = 1;
    props.setQualificationComparator(mustHave5kHitsIndex, "GreaterThanOrEqualTo");
    props.setQualificationType(mustHave5kHitsIndex, "00000000000000000040");
    props.setQualificationValue(mustHave5kHitsIndex, "5000");
}

// Register the HIT Type and use it in other create calls
String hitType = service.registerHITType(props.getAutoApprovalDelay(),
    props.getAssignmentDuration(),
    props.getRewardAmount(),
    props.getTitle(),
    props.getKeywords(),
    props.getDescription(),
    props.getQualificationRequirements()
);

Top Five Unexpected Benefits of Using Mechanical Turk Masters

We often get questions from Requesters related to Masters.... Who are they? How are they different from other Workers on Mechanical Turk?

For those not familiar, Mechanical Turk Masters are elite Workers who have consistently demonstrated superior performance by completing thousands of HITs with a high degree of accuracy. Currently, Mechanical Turk offers two groups of Masters: Photo Moderation and Categorization.

Why should you use them? Since launching, we’ve received a tremendous amount of feedback from Requesters on their experiences using Masters. This week we wanted to share some of this feedback with you by highlighting some of the unexpected benefits many of our Requesters have discovered over the past year. Here are what we consider to be the top five:

 

1. With Masters You Get More for Your Money

Masters spend more time on each assignment because they are concerned about doing a good job and maintaining their reputation. While it’s true that Amazon charges a higher fee for Masters, statistically speaking, Masters are less expensive based on their estimated hourly rate and the comparative value of their output.

 

2. Sometimes, One Head Can Be Better than Two

Masters have a demonstrated ability to contribute accurate results. Feedback from Requesters is that Masters are consistently submitting results that are 95% accurate on their tasks. If one Worker can achieve this level of accuracy, why ask a second Worker? Asking additional Workers adds complexity to your workflow and requires you to develop a strategy for aggregating results.

This approach may not meet every Requester's specific requirements, for instance when analyzing sentiment you may want to collect multiple opinions. However, for Requesters looking for one result, such as with basic categorization or moderation HITs, using one Master can provide a quicker path to quality.

 

3. Masters Have Valuable Insights and Experience

Workers on Mechanical Turk tend to be very proactive in providing insight and feedback on your HIT. Masters, in particular, can be extremely effective in helping Requesters troubleshoot HITs by identifying ambiguities in instructions, bugs in HIT interfaces, or other complications. By addressing these potential pitfalls early, Requesters can curtail poor results, such as low quality or throughput before their project is complete.

 

4. Masters Make Requesters' Jobs Easier

Many Requesters have realized that with a pre-qualified workforce, like Masters, it's possible to simplify the method you use to verify results. While review policies, such as plurality and known answers (gold), are effective tools for managing quality and are necessary for some projects, using Masters with basic auditing can reduce the time spent on implementation and enable you to receive quality results more quickly.

 

5. Using Masters Can Reduce Your Overall Costs

Implementing a more stringent qualification requirement enables Requesters to achieve quality results at a much lower cost per HIT. Each of the preceding advantages of using Masters demonstrates opportunities for Requesters to more efficiently manage HITs on Mechanical Turk. With simplified quality controls, the overhead cost required to implement, manage, and optimize performance on an ongoing basis is significantly less.

 

For more detail on how to get started using Masters Workers, check out MTurk Best Practices for Requesters.

Simple tricks for Managing Workers on Mechanical Turk

 

Did you know that you can manage your Workers from the Mechanical Turk Requester website?  This feature has been available for a while now but we find there are a few Myths that cause confusion:

Myth: I can't manage Workers via the Requester website if I create my HITs via the API.

Reality:  You CAN manage Workers from the Requester website regardless of how your HITs are created.  Just visit the Manage>Workers section.

You'll see the list of Workers who have worked on your HITs (and their approval rate on your HITs).  Click on an individual Worker to bonus, block or grant the Worker a Qualification.

 

Myth: I have to page through LOTS of pages of my Workers to find the one I want to bonus, block or assign a Qualification.

Reality: Actually there is a really simple way to manage a Worker without paging through the crowd.  The Manage Workers page includes a list of every Worker who has ever submitted a HIT for you.   If you click on an individual's ID you will be taken to the page that lets you manage this individual Worker.  The url for this detail page includes the Worker's ID in the format https://requester.mturk.com/workers/WORKERID.  Here's an example for a specific Worker:

If you want to bonus a different Worker (suppose it's a Worker with Worker ID 3ABQ167TW). Put this Worker ID after the last backslash in the url (resulting in the url https://requester.mturk.com/workers/3ABQ167TW.)  Once you're on the page for that Individual Worker, you can bonus, block and manage that Worker's Qualifications.

 

Myth: I have to block Workers individually and that takes too long.

Reality: You can take action on multiple Workers at once by using the downloaded csv file.  To do this download the file of your Workers from the Manage>Workers page.  Follow the instructions from the download wizard to block/unblock Workers or Grant/Revoke a Qualification by marking the appropriate columns and rows in the file and upload it to us.  We'll do the rest.  (Sorry you can't grant bonuses this way.)