A couple of weeks ago we held our first AWS sponsored Mechanical Turk Office Hours. It was awesome to connect with our customers – both Workers and Requesters and to hear your questions.
In the Office Hours we spent part of the time sharing updates on Mechanical Turk and spent the rest of the time answering your questions. For those of you who were not able to attend, here is a link to the video (please note that the audio on the video starts at [1:24]), and link to download the transcript (done by Mechanical Turk Workers via CastingWords). You can also find the slides on Slideshare.
Highlights:
We’ve included the time stamp from the video for reference.
[10:24] Worker Management Overview
We reviewed our recently released Worker management features and how you can you use these tools to create qualified workgroups.
[19:52] API Best Practices
- Take advantage of the notification capability that we've built into our API instead of using the polling interfaces in the Search HITs API or Reviewable HITs API.
- Disposing HITs can improve your performance as there is less data to manage for your requests.
[23:30] HIT Best Practices
- Clear & concise instructions
Provide clear and concise instructions to Workers that make it very easy to understand what the task is and what a good task response would be. - Add keywords to your HIT
This helps to make it searchable for Workers. Workers search for certain types of tasks, photo moderation, classification categorization, business listing verification, having those types of keywords in the HIT description is very useful. - User experience or the design of your HIT.
Make your HIT as efficient as possible, minimizing instructions that push the work down to the bottom of the page. Minimizing scrolling is very important to think about when you design your HIT. - Test & Iterate
Most Requesters need to modify their HITs a few times before the question(s) and instructions are clear to Workers and the HIT is designed and working properly. So when you first publish your HIT in the Mechanical Turk marketplace, consider it to be a test. Start by running a low volume of HITs to get feedback from Workers and to validate what the experience was, how long it took, and to make sure that the HIT itself is functioning as expected. Include an additional box that asks Workers for "any feedback on this HIT." The Mechanical Turk Worker community sees lots of HITs and they are able to give very helpful feedback.
Q & A Highlights:
[43:26] How should I price my HIT?
As a marketplace, the going price for a HIT will fluctuate based on basic laws of supply and demand. You determine the offer for your work, and Workers choose what they want to do. When setting the reward for your HIT you should consider:
- How long does it take you to do the task?
- How quickly do you need the work completed?
- How valuable is the task to you?
- Look at the marketplace and see how similar tasks are being priced.
- Test the price in the marketplace with a few HITs – do they get done as you’d expect?
[46:37] “How do I attract good Workers? How do I find them?"
Take the time to build a good reputation, by being quick and fair in your payment of Workers. Audit work and bonus your best Workers. This is a great way to encourage your best Workers to do more work for you. We also reviewed our Worker Management features. You can learn more about them in this SlideShare.
[47:37] A question from TurkerNation: "How should I [as a Worker] handle blocks?
A block prevents a Worker from working for a Requester. It is individual and specific to that relationship. If a Requester blocks you, we recommend that you work directly with that Requester to understand why. They are the ones who have the ultimate control over that block.
[56:24] A question from Brent Lee. "How many Workers are currently registered to work on HITs?"
We have over a half a million Workers worldwide from over 190 different countries.
[57:38] “Where should Requesters post their API questions?”
We have a forum for developers to ask questions about the Requester Web site or API tools. Here’s the link:
https://forums.aws.amazon.com/forum.jspa?forumID=11
For quick viewing, here are the slides for you to view:
P.S. I apologize for the delay in getting the video and write-up posted to the blog!
Comments