The Mechanical Turk Blog

Introducing Mechanical Turk API support for IAM credentials

Mechanical Turk is pleased to announce an improvement to the Mechanical Turk API. You can now use the Mechanical Turk API without using your root AWS account credentials to authenticate requests. This is made possible with AWS Identity Access Management (IAM). To get started, please review our Getting Started Guide.

IAM allows you to securely control access to AWS resources. Once you’ve created an IAM user, you use the AWS Management Console to grant permission to access the Mechanical Turk API. The IAM user credentials can then be used instead of your root AWS account credentials to authenticate your Mechanical Turk API calls.

A user with access to root AWS account credentials has access to all AWS resources; using IAM user credentials to authenticate your Mechanical Turk API requests allows you to better secure access to AWS resources within your organization. For example, an IAM user can be given permission to use the Mechanical Turk API but not EC2.

The type of access an IAM user has to the Mechanical Turk API can also be controlled; IAM users can be granted full access to all API methods, read-only access, or access limited to specified API methods. Please note, however, that IAM user credentials cannot be used to log into the Requester website.

Learn more about using IAM credentials with the Amazon Mechanical Turk API in our Getting Started Guide.

New Qualification Comparators Add Greater Flexibility to Qualifications

We’re excited to announce three new Qualification Comparators available in the Mechanical Turk API.  The new Comparators, which include In, NotIn, and DoesNotExist, provide Requesters greater flexibility in targeting HITs to specific Workers. 

A Comparator is an element of the QualificationRequirement data structure that represents the type of comparison made against a Qualification’s value.  Using the In Comparator you can limit HITs to Workers with one of multiple Qualification values.  For instance, if you need Workers from the United States, Canada, or Great Britain, your QualificationRequirement would be specified as “Location In US, CA, GB”. “Location” is the Qualification type, “In” is the Comparator and “US, CA, GB” are the Qualification values. 

Here’s the SOAP request based on the example above:

 <QualificationRequirement>
   <QualificationTypeId>00000000000000000071</QualificationTypeId>
   <Comparator>In</Comparator>
   <LocaleValue>US</LocaleValue>
   <LocaleValue>CA</LocaleValue>
   <LocaleValue>UK</LocaleValue>
</QualificationRequirement>

The NotIn Comparator enables you to exclude Workers that have one of multiple Qualification values.  If, in contrast to the example above, if you wanted to exclude Workers from the United States, Canada, or Great Britain from accepting your HITs, you would specify “Location NotIn US, CA, GB”. 

<QualificationRequirement>
   <QualificationTypeId>00000000000000000071</QualificationTypeId>
   <Comparator>NotIn</Comparator>
   <LocaleValue>US</LocaleValue>
   <LocaleValue>CA</LocaleValue>
</QualificationRequirement>

Finally, the DoesNotExist Comparator allows you to limit access to Workers who do not have a specific Qualification type.  For instance, if you’re using Mechanical Turk to conduct a survey, you may want to limit access to Workers who’ve not completed a survey HIT for you previously. In this case you would grant a custom Qualification (which you can create using the CreateQualificationType operation) to each Worker once they’ve completed your survey HIT. Once a Worker has been assigned your custom Qualification, you can exclude the Worker from future HITs using DoesNotExist.   

Below is a sample SOAP Request, using a custom Qualification name of “SurveyCompleted” and the DoesNotExist Comparator:

<QualificationRequirement>
   <QualificationTypeId>123456789SURVEYCOMPLETEDQUAL</QualificationTypeId>
   <Comparator>DoesNotExist</Comparator> 
</QualificationRequirement>

More detail on Qualifications, as well as Comparators available, is available in the Mechanical Turk API documentation.    

Helpful Tips for Choosing the Right Mechanical Turk Interface

We frequently engage with Requesters to help determine the right approach for integrating with Mechanical Turk. Most commonly, Requesters are deciding between using our user interface (UI) or integrating directly with our application programming interface (API).  In today’s post, we thought it might be helpful to highlight some of the key advantages of each interface.

Advantages of the Requester UI

The Requester UI enables Requesters to design and publish HITs, receive results, and manage Workers from an online interface. For Categorization or Sentiment Analysis work, we offer the Categorization Application and the Sentiment Application, both of which feature predesigned templates and allow you to view results directly from the UI.

For Requesters with limited software development resources or for smaller projects, using the UI makes it easier to get started more quickly.  While a few lines of HMTL may be necessary if you choose to design your own template instead of using the Categorization or Sentiment Application, the technical skills required are minimal.  

Key advantages of using the UI include the ability to:

• Quickly publish work and receive results, without integrating with our API.
• Easily review results and approve or reject assignments.  
• Manage Workers including blocking and granting bonuses.
• Monitor Worker Statistics and create custom Qualifications to target Workers who have performed well for you.

Advantages of the Requester API

The Requester API enables Requesters to publish HITs, collect results, and manage Workers programmatically, without human intervention. Integrating with the API requires software development, but it is the optimal approach to efficiently process high volumes of HITs on a continuous basis. 

Key advantages of using the API include the ability to:

• Integrate results directly into your application.
• Design automated workflows for approving and rejecting assignments using Known Answers via the Review Policies.
• Automate performance tracking with Worker Statistics and managing access to your HITs with Qualifications.

For more detailed information, see our Choose the Right Tool guide on the features and relative advantages of each interface. 

Introducing New API Functionality: Worker Statistics and Review Policies

We are excited to announce several additions to the Mechanical Turk Application Programming Interface (API).  These new API operations are intended to make it easier for you to get consistently accurate results from Mechanical Turk.     

The new Worker Statistics API gives you programmatic access to statistics on how each Worker performed  on YOUR HITs, you can specify the time range, for instance life-to-date or last 20 days.  The Worker Statistics API eliminates the need for you to calculate and track Worker statistics such as a Worker's approval rate on your HITs.  This new functionality makes it easy for you to take action on Worker approval rates – for instance awarding bonuses, granting and revoking Qualifications, or blocking Workers.  This functionality is already available via the Requester UI but adding it to the API allows you to integrate this directly into your Mechanical Turk application.

In speaking with many Requesters, we have learned a lot about the various ways Requesters "review" work after it is submitted to determine if it's accurate.  We created the Review Policy API to make it easier for you to apply the two most commonly used quality assessment tools – Known Answers and Worker Agreement.

Known Answers are used by many Requesters to determine how accurate a Worker is.  For instance, a Requester may have 20 images in a HIT that they want Workers to moderate against specific guidlines ("appropriate" or "not").  By including four images that have "Known Answers" among the 20, the Requester is able to use the Worker's performance on the four known images to assess whether they should trust the Worker's answers on the other 16. The Requester may decide that as long as the Worker gets 3 or 4 of the Known Answers correct they will "trust" the remaining answers.  The "Score My Known Answers" Review Policy makes it easy to do this.  You just provide your “Known Answers” for questions when you create HITs.  Mechanical Turk will score the Worker's submission by comparing the Worker’s submitted answer for these questions to the “Known Answer” you provided.  Mechanical Turk will return this score to you.  You can also authorize Mechanical Turk to approve or add assignments based on the Known Answer score.

In addition to calculating a Worker's Known Answer performance within an individual assignment, Mechanical Turk also keeps a running score of a Worker’s performance on Known Answers across all of your assignments.   Worker statistics on your Known Answers are available via the Worker Statistics API.  The "Score My Known Answers" Review Policy API eliminates the need for you to score and track Worker statistics on your Known Answer work items and makes it easy for you to take action based on these statistics – for instance awarding bonuses for high Known Answer performance or revoking Qualifications for poor performance.

The "Simple Plurality" Review Policy calculates agreement among Workers performing the same HIT.  You can specify a target agreement level and authorize Mechanical Turk to add assignments for additional Workers until the specified level of agreement is achieved.  In addition to calculating a Agreement  scores within individual HITs, Mechanical Turk also keeps a running score of a Worker’s agreement statistics across all of these assignments.  A Worker's Agreement Statistics on these HITs is available via the Worker Statistics API.

The most powerful approach is to combine these two review policies.  This allows you to check that each Worker accurately answered the Known Answers (which tells you the Worker is good at your HITs) and then verify that multiple Workers agree on answers to individual work items (which tells you that Workers who are good at your HITS think this is the correct answer).  Requesters who have combined these two review policies have told us they've been very successful at achieving consistently accurate results.  

You can learn more about these new capabilities in the Amazon Mechanical Turk API Reference Document .