We recently received the following question from a Requester.
I know I can create HITs that require a Worker have an approval rate of 95% but how do I create HITs that require a Worker have an approval rate of 95% on MY HITs?
This is a common use case so I thought I’d share the answer on the blog.
Mechanical Turk provides a “System Qualification” called HIT Approval Rate. A Worker’s HIT Approval Rate is calculated as the Workers' Lifetime Assignments Approved divided by the Worker's Lifetime Number of Assignments Submitted -- on ALL Assignments. Because it’s calculated across ALL assignments that a Worker completes, it doesn’t necessarily reflect a Worker’s ability to accurate complete YOUR HITs. For instance if a Worker completes mostly photo moderation tasks, they may have a high HIT Approval Rate but they may perform poorly on your HIT that asks them to “Write an article about the motives of the Occupy Wall Street Movement.”
But there is an easy way to require that Workers have a certain approval rate on YOUR HITs. To do this you need to create a Qualification Type and grant it to Workers who have achieved the approval rate you decide is acceptable. Regardless of whether you create HITs via the Requester web site or the developer tools (API or Command Line Tools), you can create a Qualification Type via the Requester web site(Requester.mturk.com >Manage>Qualification Types).
Once you create a Qualification Type, you can grant it to Workers with the acceptable approval rate. So how do you know a Worker’s approval rate on YOUR HITs? Regardless of whether you create HITs via the Requester web site or the developer tools (API or Command Line Tools),you can see your Workers’ approval rates on the Worker Management page of the Requester web site (Requester.Mturk.com > Manage > Workers). Here is the list of Workers who have submitted assignments for you. You can filter to show only Workers who have submitted assignments for you in the last 7 days, 30 days or forever. To grant the Qualification to multiple Workers, use the csv file download/upload option on this page. To grant the Qualification to one Worker, just click on the Worker ID. You’ll be taken to the Worker’s detail page where you can grant them the Qualification as well as block or bonus the Worker.
BTW, the Manage Worker page also shows you how many assignments each Worker has completed for you. This lets you find the Workers who've earned a 95% approval rate on your HITs AND completed enough HITs so that 95% Approval Rate matters.
Once you create and grant the Qualifiction, you can use it to control who can work on your HITs. The next time you create HITs (via the Requester web site or the developer tools) just require Workers have the Qualification you created in order to complete your HITs.
No offense, but if there's a facebook like button, it'll be much easier for me to share.
Posted by: Elliptical reviews | 11/29/2011 at 22:59