We often get questions from Requesters related to Masters.... Who are they? How are they different from other Workers on Mechanical Turk?
For those not familiar, Mechanical Turk Masters are elite Workers who have consistently demonstrated superior performance by completing thousands of HITs with a high degree of accuracy. Currently, Mechanical Turk offers two groups of Masters: Photo Moderation and Categorization.
Why should you use them? Since launching, we’ve received a tremendous amount of feedback from Requesters on their experiences using Masters. This week we wanted to share some of this feedback with you by highlighting some of the unexpected benefits many of our Requesters have discovered over the past year. Here are what we consider to be the top five:
1. With Masters You Get More for Your Money
Masters spend more time on each assignment because they are concerned about doing a good job and maintaining their reputation. While it’s true that Amazon charges a higher fee for Masters, statistically speaking, Masters are less expensive based on their estimated hourly rate and the comparative value of their output.
2. Sometimes, One Head Can Be Better than Two
Masters have a demonstrated ability to contribute accurate results. Feedback from Requesters is that Masters are consistently submitting results that are 95% accurate on their tasks. If one Worker can achieve this level of accuracy, why ask a second Worker? Asking additional Workers adds complexity to your workflow and requires you to develop a strategy for aggregating results.
This approach may not meet every Requester's specific requirements, for instance when analyzing sentiment you may want to collect multiple opinions. However, for Requesters looking for one result, such as with basic categorization or moderation HITs, using one Master can provide a quicker path to quality.
3. Masters Have Valuable Insights and Experience
Workers on Mechanical Turk tend to be very proactive in providing insight and feedback on your HIT. Masters, in particular, can be extremely effective in helping Requesters troubleshoot HITs by identifying ambiguities in instructions, bugs in HIT interfaces, or other complications. By addressing these potential pitfalls early, Requesters can curtail poor results, such as low quality or throughput before their project is complete.
4. Masters Make Requesters' Jobs Easier
Many Requesters have realized that with a pre-qualified workforce, like Masters, it's possible to simplify the method you use to verify results. While review policies, such as plurality and known answers (gold), are effective tools for managing quality and are necessary for some projects, using Masters with basic auditing can reduce the time spent on implementation and enable you to receive quality results more quickly.
5. Using Masters Can Reduce Your Overall Costs
Implementing a more stringent qualification requirement enables Requesters to achieve quality results at a much lower cost per HIT. Each of the preceding advantages of using Masters demonstrates opportunities for Requesters to more efficiently manage HITs on Mechanical Turk. With simplified quality controls, the overhead cost required to implement, manage, and optimize performance on an ongoing basis is significantly less.
For more detail on how to get started using Masters Workers, check out MTurk Best Practices for Requesters.
Comments