Many Requesters ask multiple Workers to complete the same HIT. We often refer to this as using “plurality”. For example, a travel site may ask Workers to categorize images provided by a hotel so they know which ones to feature on their travel site. The Requester might ask 2 Workers to indicate if the image is of the pool, bedroom, lobby or other location in the hotel. When 2 Workers come to the same conclusion and provide the same answer independently, the Requester has more confidence in the answer then if one Worker was asked or if 2 Workers provided different answers.
Often, the goal of using plurality is to get high confidence in Worker answers with the minimum amount of work. To make sure plurality is working well for you, you need to make sure Workers agree MOST of the time. The best way to do this is to tightly control which Workers can work on your HITs and make sure your instructions are clear and cover MOST of the “gray areas” or “corner cases”.
A good way to check that your instructions are clear and complete is to complete 100 of your own HITs with the real data. An even better way is to have a friend or colleague do them – pick someone who isn’t too close to the project. Have them follow the instructions you provide and make a list of the questions they had that your instructions didn’t cover. Adjust your instructions based on this feedback.
Many Requesters are afraid to restrict the Worker Group – won’t my work get done slower? Maybe but it will be done more accurately. You can restrict your Workers by using Masters – an elite group of Workers who have demonstrated proficiency in certain types of HITs. If you don’t want to use Masters, you can also create your own custom group of Qualified Workers. The better qualified your Workers, the more likely each will provide an accurate answer.
If you follow this advice, when 2 qualified Workers disagree, it will likely be that there is an issue with the HIT data or the instructions. For instance, in our use case what is the correct category for an image of the couch/sitting area of the hotel room?
Plurality is a great tool but like any tool, you have to know when and how to use it. It’s easy to know what to do when Workers agree – approve the work and use the answer. But what do you do when they disagree? Many Requesters ask another Worker and if 2 out of the 3 Workers agree on an answer they use that answer. That’s fine. But do you approve assignments from the Worker who disagreed? You should.
Why? Well let’s think about this from a Worker’s perspective. Workers, especially really good Workers, are reluctant to do HITs with plurality if you are rejecting based on majority rules. Why is that? Workers, especially good Workers, know they submitted a good answer. But they don’t know if the other Worker (I’ll call him Worker B) is another great Worker just like them or a not so good Worker. If Worker B and C didn’t read the instructions carefully or submitted the wrong answer (whatever the reason) you are penalizing Worker A unfairly by rejecting their work. As good Workers get penalized unfairly, they may choose not to do your HITs (because they want their work to be graded fairly and they want to protect their Approval Rate) and more of your HITs are being completed by Workers like B and C. It’s a vicious cycle – as more good Workers avoid your HITs, a larger portion of your HITs are completed by less accurate Workers.
To avoid this trap
1) Tightly qualify your Workers (either by testing and auditing Workers directly or by using Masters)
2) Make sure your instructions are good and revise them based on patterns in Worker disagreement
3) Don’t reject assignments based solely on “majority” – either audit it or approve it. Only reject when you are sure the Worker’s answer was wrong.
Comments