The Mechanical Turk Blog

Worker Corner: What Makes a Great Requester?

Requesters often ask us for recommendations on how to best attract and engage Workers in Mechanical Turk’s marketplace. We’ve shared recommendations for building Requester reputation in previous posts.  Today, we wanted to revisit the topic, this time sharing insights directly from Workers on what makes a Requester great.

In order to collect insights from Workers, we posted a HIT to Mechanical Turk asking Workers to share their perspective on actions Requesters can take to make their HITs more attractive.  Based on previous surveys we’ve conducted, we know that a primary motivation for Workers is monetary reward. However, in this exercise, we wanted to explore other approaches that Requesters can use to build their relationship with Workers outside of compensation. 

So, what did Workers have to say? In reviewing their feedback, three key themes emerged:

Communicating with Workers by addressing questions and providing feedback can help identify issues in HIT design and improve the quality of results.  

“Communication! A great requester communicates regularly with their workforce. That means answering emails promptly or connecting with us in the forum to address questions and concerns, announce changes and upcoming work, explain work and rejections policies in more detail, etc. Communication makes everything easier, for everybody involved. Nothing worse than being rejected or even blocked and never [knowing] why.”

“The great requestor gives feedback for good work and bad. If there is a way that a worker can improve their work, constructive feedback is the only way to know how to do so. Good feedback inspires more good work.”

“Requesters who take the time to communicate are at the top of my list.  Communication allows the workers to have a better idea of what the requester is looking for and in turn provides the requester with better work.  It doesn't take much just an email or two and has an enormous return on investment.”

"An open line of communication between Worker and Requester can improve any issue on either side. Being open and responsive to feedback is a great sign that a requester is interested in improving the quality of work and the quality of worker/requester experience on MTurk in general.”

“Having an active presence on [a Mechanical Turk Forum, like TurkerNation] affords a Requester the opportunity to be a great Requester. The ability to communicate with a group of Workers, as well as recruit new Workers, is a time saving, cost-effective way to advertise tasks, clarify or modify instructions and/or qualify Workers. A Requester that is willing to do these things is not only a great Requester, but a great Requester [who] will obtain quality results on the tasks they publish because the Workers will be waiting for their HITs to launch.”

Approving (or Rejecting) assignments in a timely manner helps Requesters build trust with Workers.

 “Pay promptly. If you have a lot of HITs and you are a new Requester to me, I will be reluctant to do a large batch of hits before I know that I will get paid. If I start doing a batch of HITs for you, and within a few hours I see "payment pending" (meaning that you’ve approved my work) for the first 100 or more HITs, I will have no reservations in spending the entire day and the next day working on your HITs.”

“I prefer hits that [are approved] quickly. It helps me to judge my work. If there is a problem I can fix it if need be. Plus getting the money quickly helps. Some HITs take days to get approved. By then I have moved on to other hits so if there is a problem it is hard to fix."

Mistakes happen, but quickly reversing mistakenly rejected assignments can encourage Worker loyalty. 

“Great Requesters know when to admit that they have screwed up. I've had many times where I felt that my work was unjustly rejected after I had spent a good amount of time on it. When I contact great Requesters about this type of issue they find a way to make it right, either by offering me a bonus for the rejected work or additional work that will earn back my trust and respect for them.”

“Learning how [Mechanical Turk] works, specifically reversing rejections. Rejections stay with a Worker forever even when the requester is at fault. When a Requester learns how the UI works and realizes they can reverse rejections, I'm more likely to stick with them on future tasks.”

“Quick and honest dispute resolution.  On several occasions I have had work incorrectly rejected.  When I have reached out to the Requester, they have responded quickly by double checking the error in question and either admitting the mistake on their part or acknowledging that the reject was in error.  They either reverse the rejection or they [issue a] bonus...”

To learn more about best practices for achieving results on Mechanical Turk and building your reputation with Workers, be sure to review the Requester Best Practices Guide.

Finally, thank you to all of the Workers who contributed feedback.  We appreciate your insight.  

-- Andrea

Mturk mail bag: How do I create HITs that require a Worker have an approval rate of 95% on MY HITs?

We recently received the following question from a Requester. 

I know I can create HITs that require a Worker have an approval rate of 95% but how do I create HITs that require a Worker have an approval rate of 95% on MY HITs?

This is a common use case so I thought I’d share the answer on the blog.

Mechanical Turk provides a “System Qualification” called HIT Approval Rate.  A Worker’s HIT Approval Rate is calculated as the Workers' Lifetime Assignments Approved divided by the Worker's Lifetime Number of Assignments Submitted -- on ALL Assignments.   Because it’s calculated across ALL assignments that a Worker completes, it doesn’t necessarily reflect a Worker’s ability to accurate complete YOUR HITs.  For instance if a Worker completes mostly photo moderation tasks, they may have a high HIT Approval Rate but they may perform poorly on your HIT that asks them to “Write an article about the motives of the Occupy Wall Street Movement.”

But there is an easy way to require that Workers have a certain approval rate on YOUR HITs. To do this you need to create a Qualification Type and grant it to Workers who have achieved the approval rate you decide is acceptable.  Regardless of whether you create HITs via the Requester web site or the developer tools (API or Command Line Tools), you can create a Qualification Type via the Requester web site(Requester.mturk.com >Manage>Qualification Types). 

Once you create a Qualification Type, you can grant it to Workers with the acceptable approval rate.  So how do you know a Worker’s approval rate on YOUR HITs? Regardless of whether you create HITs via the Requester web site or the developer tools (API or Command Line Tools),you can see your Workers’ approval rates on the Worker Management page of the Requester web site (Requester.Mturk.com > Manage > Workers).   Here is the list of Workers who have submitted assignments for you.  You can filter to show only Workers who have submitted assignments for you in the last 7 days, 30 days or forever.  To grant the Qualification to multiple Workers, use the csv file download/upload option on this page. To grant the Qualification to one Worker, just click on the Worker ID.  You’ll be taken to the Worker’s detail page where you can grant them the Qualification as well as block or bonus the Worker.

BTW, the Manage Worker page also shows you how many assignments each Worker has completed for you.  This lets you find the Workers who've earned a 95% approval rate on your HITs AND completed enough HITs so that 95% Approval Rate matters. 

Once you create and grant the Qualifiction, you can use it to control who can work on your HITs. The next time you create HITs (via the Requester web site or the developer tools) just require Workers have the Qualification you created in order to complete your HITs.

Get Better Results with Less Effort with Mechanical Turk Masters!

We're excited to introduce Mechanical Turk Masters. Masters are an elite group of Workers who have demonstrated superior performance while completing thousands of HITs across the Marketplace.

By publishing your HITs to Masters, the quality of your answers will increase (you’ll get right answers more often) and managing results and Workers will be easier. You may even find that you can publish fewer assignments per HIT and still get increased accuracy.  Mechanical Turk Masters provide Requesters with an easier way to distribute HITs to Workers who have proven themselves on the Mechanical Turk Marketplace. 

Starting today, Requesters will be able to access two types of Masters -- Photo Moderation Masters and Categorization Masters.   Workers who achieve these distinctions are granted the corresponding Masters Qualification for either Photo Moderation or Categorization.

Although you can send any type of HIT to Mechanical Turk Masters, Workers who have been granted a specific Masters Qualification have demonstrated superior performance at these types of HITs. For instance, Photo Moderation Masters are the best choice if you want to have a photo reviewed to determine if it meets your site guidelines, or need specific types of content identified such as the number of people in a photo. Categorization Masters are the best choice if you need to have companies categorized by industry or identify the department for products in a retail catalog or characterize the sentiment of a tweet or blog post.

Getting Started with Mechanical Turk Masters
Accessing Mechanical Turk Masters is easy. Requesters simply require Workers have one of the Master Qualifications when creating HITs via the Requester web site, API or Command Line Tools.  Requesters can further refine their workforce by requiring Workers have other system or custom Qualifications in addition to the Masters Qualification.

Congratulations Mechanical Turk Masters!
Achieving Mechanical Turk Master status is an honor for Workers – and we want to congratulate the Workers who have achieved this distinction! Kudos!

Search by Requester Name

Hello Turkers! I wanted to let you know that we recently released an update to the search functionality for the HIT catalog on Mechanical Turk. You can now search for a Requester by name – making it easier to find work you enjoy. 

How it works:
Enter the exact name of the Requester in the search box and we will return all of the available HIT capsules (or groups) for this Requester. In the sample below I searched for [Amazon Requester Inc.] and the results returned all the active HITS for this Requester.

Happy New Year!
-- Natala