Requesters consistently tell us that using Qualifications is one of the most effective strategies for optimizing the quality of results on Mechanical Turk. So, if you're an API user and unfamiliar with how to implement Qualifications, this post is intended to make it easy to get started.
How do Qualifications work? Qualifications enable you to direct your HITs to specific groups of Workers on Mechanical Turk. When you include a Qualification, it means that Workers who do not meet your Qualification standard cannot contribute to your HIT. As a Requester, you can use pre-defined Qualifications or create your own custom Qualification types (to learn more about creating Qualifications, see this post on the topic).
Mechanical Turk offers two types of pre-defined Qualifications--Masters and System Qualifications. We've discussed the value of Masters Workers in a previous post. For those new to the concept, Masters are an elite group of Workers with demonstrated accuracy across specific types of HITs in the Mechanical Turk Marketplace--specifically, categorization and photo moderation. Making the Masters Qualification a requirement is the easiest way to ensure that your work is completed by Workers with proven expertise in your task. Keep in mind, your HITs do not have to be photo moderation or categorization—you can send any type of HIT to either Masters group.
In addition to Masters, Mechanical Turk offers several additional System Qualification types that you can leverage to direct your work to experienced Workers with proven performance. To direct your HITs to the most experienced Workers, use the “Number of HITs Approved” System Qualification. Based on Requester feedback, we recommend that you set your requirement at a level greater than 5000 approved assignments. This means that only Workers who’ve completed 5000 approved assignments will have access to your HIT.
In addition to targeting the most experienced Workers, setting your “HIT Approval Rate” System Qualification at greater than 95% will direct your work to the highest quality Workers as determined by their approval rate from other Requesters in the Mechanical Turk Marketplace. Requesters frequently tell us that the combination of these two Qualifications has a significant affect on the overall quality of results.
How can you implement these Qualifications? In the Requester API, the QualificationRequirement data structure describes a Qualification a Worker must have before the Worker is allowed to accept a HIT. The QualificationRequirement data structure is used as a parameter for the following operations:
- CreateHIT
- RegisterHITType
Learn more about QualificationRequirement and access Qualification TypeIDs in MechanicalTurk's API reference.
Additionally, in effort to make implementation easy, we've provided sample code for the following Qualifications in Ruby, .NET, and Java:
- Masters (photo moderation)
- Approval Rate
- Number of HITs Approved
Ruby
require 'ruby-aws'
@mturk = Amazon::WebServices::MechanicalTurkRequester.new :Host => :Sandbox # remove the :host option to use production
# This is how to require a Photo Moderation Master
photoModerationId = (@mturk.host =~ /sandbox/) ? '2TGBB6BFMFFOM08IBMAFGGESC1UWJX' : '21VZU98JHSTLZ5BPP4A9NOBJEK3DPG'
mustBeImageModerationMaster = { :QualificationTypeId => photoModerationId, :Comparator => 'Exists' }
# This is how to build a qualification requirement for specific statistics
mustHaveHighApprovalRate = {
:QualificationTypeId => Amazon::WebServices::MechanicalTurkRequester::APPROVAL_RATE_QUALIFICATION_TYPE_ID,
:Comparator => 'GreaterThan',
:IntegerValue => 95,
}
mustHave5kHits = {
:QualificationTypeId => '00000000000000000040' # Worker_NumberHITsApproved
:Comparator => 'GreaterThan',
:IntegerValue => 5000,
}
# The create HIT method takes in an array of QualificationRequirements since a HIT can have multiple qualifications.
# Here, we're choosing to require two of the statistic-based qualification requirements we created above:
qualReqs = [mustHaveHighApprovalRate,mustHave5kHits]
# Alternatively, just require the Photo Moderation Master
qualReqs = [mustBeImageModerationMaster]
# Register the Hit Type
hitType = @mturk.registerHITType(
:Title => title,
:Description => description,
:Keywords => keywords,
:Reward => reward,
:AssignmentDurationInSeconds => assignmentDurationInSeconds,
:AutoApprovalDelayInSeconds => autoApprovalDelayInSeconds,
:QualificationRequirement => qualReqs
)
.NET
SimpleClient mTurkClient = new SimpleClient();
List qualList = new List();
QualificationRequirement qualImageMod = new QualificationRequirement();
QualificationRequirement qualApprovalRate = new QualificationRequirement();
QualificationRequirement qualNumHits = new QualificationRequirement();
bool useMasters = true; // change the default based on preference
if (useMasters)
{
// qualifications for Image Moderation
if (mTurkClient.Config.WebsiteURL.Contains("sandbox"))
qualImageMod.QualificationTypeId = "2TGBB6BFMFFOM08IBMAFGGESC1UWJX";
else
qualImageMod.QualificationTypeId = "21VZU98JHSTLZ5BPP4A9NOBJEK3DPG";
qualImageMod.Comparator = Comparator.Exists;
qualList.Add(qualImageMod);
}
else
{
// setup the qualification for approval rate
qualApprovalRate.QualificationTypeId = MTurkSystemQualificationTypes.ApprovalRateQualification;
qualApprovalRate.Comparator = Comparator.GreaterThan;
qualApprovalRate.IntegerValue = 95;
qualApprovalRate.IntegerValueSpecified = true;
qualList.Add(qualApprovalRate);
// setup the qualification for number of HITs approved
qualNumHits.QualificationTypeId = "00000000000000000040";
qualNumHits.Comparator = Comparator.GreaterThan;
qualNumHits.IntegerValue = 5000;
qualNumHits.IntegerValueSpecified = true;
qualList.Add(qualNumHits);
}
// register the HIT Type, so that it can be used in later calls to CreateHIT
string hitTypeId = mTurkClient.RegisterHITType("Blog Sample",
"This is a HIT created by the Mechanical Turk .Net SDK",
60*60, 60*60, 0.02m,
"qualifications sample",
qualList);
Java
// Choose Categorization Masters Qualification ID based on endpoint
if (service.getWebsiteURL().contains("sandbox")) {
mastersQualID = "2F1KVCNHMVHV8E9PBUB2A4J79LU20F";
} else {
mastersQualID = "2NDP2L92HECWY8NS8H3CK0CP5L9GHO";
}
HITProperties props = new HITProperties(propertiesFile);
// Create the Qualification Requirements
// Use the categorization masters qualification or limit the approval rate for Workers
Boolean useMasters = true;
if (useMasters) {
int mastersQualIndex = 0;
props.setQualificationComparator(mastersQualIndex, "Exists");
props.setQualificationType(mastersQualIndex, mastersQualID);
} else {
int highApprovalRateIndex = 0;
props.setQualificationComparator(highApprovalRateIndex, "GreaterThanOrEqualTo");
props.setQualificationType(highApprovalRateIndex, com.amazonaws.mturk.service.axis.RequesterService.APPROVAL_RATE_QUALIFICATION_TYPE_ID);
props.setQualificationValue(highApprovalRateIndex, "95");
int mustHave5kHitsIndex = 1;
props.setQualificationComparator(mustHave5kHitsIndex, "GreaterThanOrEqualTo");
props.setQualificationType(mustHave5kHitsIndex, "00000000000000000040");
props.setQualificationValue(mustHave5kHitsIndex, "5000");
}
// Register the HIT Type and use it in other create calls
String hitType = service.registerHITType(props.getAutoApprovalDelay(),
props.getAssignmentDuration(),
props.getRewardAmount(),
props.getTitle(),
props.getKeywords(),
props.getDescription(),
props.getQualificationRequirements()
);
