We are excited to announce several additions to the Mechanical Turk Application Programming Interface (API). These new API operations are intended to make it easier for you to get consistently accurate results from Mechanical Turk.
The new Worker Statistics API gives you programmatic access to statistics on how each Worker performed on YOUR HITs, you can specify the time range, for instance life-to-date or last 20 days. The Worker Statistics API eliminates the need for you to calculate and track Worker statistics such as a Worker's approval rate on your HITs. This new functionality makes it easy for you to take action on Worker approval rates – for instance awarding bonuses, granting and revoking Qualifications, or blocking Workers. This functionality is already available via the Requester UI but adding it to the API allows you to integrate this directly into your Mechanical Turk application.
In speaking with many Requesters, we have learned a lot about the various ways Requesters "review" work after it is submitted to determine if it's accurate. We created the Review Policy API to make it easier for you to apply the two most commonly used quality assessment tools – Known Answers and Worker Agreement.
Known Answers are used by many Requesters to determine how accurate a Worker is. For instance, a Requester may have 20 images in a HIT that they want Workers to moderate against specific guidlines ("appropriate" or "not"). By including four images that have "Known Answers" among the 20, the Requester is able to use the Worker's performance on the four known images to assess whether they should trust the Worker's answers on the other 16. The Requester may decide that as long as the Worker gets 3 or 4 of the Known Answers correct they will "trust" the remaining answers. The "Score My Known Answers" Review Policy makes it easy to do this. You just provide your “Known Answers” for questions when you create HITs. Mechanical Turk will score the Worker's submission by comparing the Worker’s submitted answer for these questions to the “Known Answer” you provided. Mechanical Turk will return this score to you. You can also authorize Mechanical Turk to approve or add assignments based on the Known Answer score.
In addition to calculating a Worker's Known Answer performance within an individual assignment, Mechanical Turk also keeps a running score of a Worker’s performance on Known Answers across all of your assignments. Worker statistics on your Known Answers are available via the Worker Statistics API. The "Score My Known Answers" Review Policy API eliminates the need for you to score and track Worker statistics on your Known Answer work items and makes it easy for you to take action based on these statistics – for instance awarding bonuses for high Known Answer performance or revoking Qualifications for poor performance.
The "Simple Plurality" Review Policy calculates agreement among Workers performing the same HIT. You can specify a target agreement level and authorize Mechanical Turk to add assignments for additional Workers until the specified level of agreement is achieved. In addition to calculating a Agreement scores within individual HITs, Mechanical Turk also keeps a running score of a Worker’s agreement statistics across all of these assignments. A Worker's Agreement Statistics on these HITs is available via the Worker Statistics API.
The most powerful approach is to combine these two review policies. This allows you to check that each Worker accurately answered the Known Answers (which tells you the Worker is good at your HITs) and then verify that multiple Workers agree on answers to individual work items (which tells you that Workers who are good at your HITS think this is the correct answer). Requesters who have combined these two review policies have told us they've been very successful at achieving consistently accurate results.
You can learn more about these new capabilities in the Amazon Mechanical Turk API Reference Document .
Comments