We recently attended CrowdConf in San Francisco where we had the pleasure of connecting with a number of Requesters and Partners. These interactions are always a great opportunity to get feedback and better understand how people are using Mechanical Turk.
Over the course of my conversations, I noticed an interesting trend—many of the Requesters I spoke with were unaware that the Mechanical Turk API supports Known Answers. So, in case you’ve not heard about how we support Known Answers, I wanted to take today’s post as an opportunity to reintroduce the concept.
Known Answers are questions with answers that are already known. By comparing a Worker’s answer to a question with the Known Answer, you can measure the Worker’s performance and get a sense of the quality of his or her work overall. For instance, I use Known Answers all the time to quiz my daughter on her homework. Testing that she knows 3x7 is 21 doesn’t mean that she’ll get everything right on her test, but if she gets the 5 questions I ask her wrong, I know she needs more studying.
Known Answers are most effective in HITs that include multiple choice questions for which there are clear right answers. For example, suppose you design a HIT that asks Workers to categorize five different photos. You could use Known Answers by adding a sixth photo for which you already know the correct category. In the course of completing the HIT, the Worker would provide categories for all six images, not knowing that one of the answers is already known. The more Known Answers you ask a Worker, the more confident you are that they know their stuff—similar to how I know if my daughter is sufficiently prepared for her test.
Mechanical Turk’s Known Answer functionality lets you easily incorporate Known Answers into your HITs. You send us your Known Answers when you create the HIT, and we track Workers’ performance on Known Answers within the Assignment and on your HITs over time. This lets you do a few powerful things:
- You can decide if you want to use work submitted. If a Worker gets 4 of the 5 Known Answers in a HIT wrong, you may want to reject their Assignment.
- You can decide whether or not to continue to allow the Worker to work for you. If a Worker has completed 100 Known Answers for you and is less than 90% accurate, you may want to block them or if you’re using a Qualification, revoke their qualification.
To learn more about how to use Known Answers in your HITs read the ScoreMyKnownAnswer under Review Policies in the Mechanical Turk API docs. Additionally, to learn more about how to track a Worker’s Known Answer performance over time, see Worker Statistics under Operations (see GetRequesterWorkerStatistic).