Editor's Note: Today's blog post is part of our Partner Spotlight series featuring best practices from Mechanical Turk Solution Providers. This post is from Sammie Schweissguth, Director of Client Services at CrowdSource.
Whether you are looking to improve your current content generation HITs or are just getting started, using an effective strategy to attract and manage Workers is critical to successful content generation on Mechanical Turk.
With hundreds of thousands of Workers available, Requesters must identify Workers possessing top-notch writing and editing abilities, tap into their skills and manage them on an ongoing basis. How do you accomplish this? Based on our experience at CrowdSource, the following eight elements can make or break your strategy to generate great content:
- Establishing a style guide
- Qualifying and recruiting Workers
- Paying Workers appropriately
- Writing clear instructions
- Communicating with the workforce
- Using a workflow that breaks up tasks
- Developing a quality control strategy
- Providing performance feedback
Establishing a Style Guide
Naturally, every Worker has a different approach to writing. However, communicating clear guidelines relating to style, grammar and format can help create consistency in the quality of content from Workers, especially when you are looking to produce content at scale. Whether you’re using an existing style guide or creating your own from scratch, include detailed descriptions and make sure the guidelines are easily accessible to Workers while they work. Consider the following helpful tips for creating a guide:
- Clearly define any specific style preferences relating to grammar, spelling, formatting, voice, point of view, etc.
- Consider using existing resources, such as the AP or the Yahoo! style guidelines for web content.
- Provide examples of content that meet your style requirements.
Qualifying and Recruiting Workers
Mechanical Turk provides access to an on-demand workforce around-the-clock. However, not every Worker is capable of producing content that meets your requirements. For this reason, creating a qualification test to identify Workers with excellent grammar, strong writing skills and the ability to follow instructions before they work on your HITs is very important. Here are some additional considerations based on our experience building qualified workforces at CrowdSource:
- Create two separate qualification tests (one for writing; one for editing).
- Restrict newly qualified Workers to the most basic HITs and increase access to advanced tasks based on performance.
- Consider incorporating writers on staff into your workflow—these writers can contribute the most advanced tasks, while Workers from Mechanical Turk can be leveraged to achieve scale.
Paying Workers Appropriately
How you price HITs affects the type and quality of Workers you attract. An advantage of crowdsourcing content creation is that it’s less expensive than utilizing an internal team. However, cost efficiency shouldn’t come at the cost of Worker compensation. Pricing your HIT appropriately for the quality of work you expect is vital to cultivating a loyal group of Workers. Here are a few tips we use at CrowdSource to set HIT price:
- Aim to pay at least minimum wage for basic writing tasks.
- Increase the pay for more difficult, specialized or research-intensive writing.
- Understand that paying fairly tends to attract higher-quality, loyal Workers.
Writing Clear Instructions
Workers will not meet your expectations without knowing what they are. Clearly define all instructions and make requirements obvious. At the same time, consider revising your instructions as your project progresses based on Worker feedback and results. For content that requires more structure, provide step-by-step instructions throughout the HIT to keep Workers on track. Ideally, your instructions should address and answer as many potential questions as you can identify. Generally speaking, the following are great tips to keep in mind:
- Keep instructions clear and to the point.
- Provide examples to illustrate ideal content.
- As instructions change, clearly indicate updates in the HIT instructions.
Communicating with the Workforce
Even with clear instructions, Workers are likely to have occasional questions. Respond personally to emails and forum posts asking for clarification. Engage in friendly interactions with Workers on a regular basis through Worker forums. In our experience, direct communication fosters relationships, creates accountability and improves quality. Tips to consider when communicating with Workers:
- Be open and honest.
- Respond to personal emails in a timely fashion.
- Communicate regularly via forums such as TurkerNation.
- Listen to and consider Worker feedback—they can help spot issues and save you time.
Using a Workflow that Breaks Up Tasks
Breaking work into more digestible HITs can make tasks easier for Workers to manage, improving the quality of their work. By separating the tasks of writing and editing, content is cross-checked by another Worker, and subpar work is less likely to make it through your process. Based on our experience, we advise you to do the following when designing your workflows:
- Split writing and editing steps into separate tasks.
- If it’s required, separate preliminary research from the writing step.
- Include a third step for annotation when necessary.
Developing a Quality Control Strategy
In addition to breaking down workflows into multiple tasks to improve quality, implementing caps on the number of writing or editing HITs available to a new Worker provides additional control. Similarly, recruiting Workers with proven performance to review others’ content can be a very effective strategy for managing quality. The following are common methods we use at CrowdSource to ensure quality results:
- Use HIT submission limits for new, unproven Workers.
- Utilize the highest-performing Workers to review and edit submitted content.
- Use feedback from reviews as measures of performance.
Providing Performance Feedback
Providing feedback on performance to Workers can lead to better work. At CrowdSource, we use the ability to reject a Worker only as a last resort to maintain a good relationship and reputation with Workers. Instead, foster the growth of Workers through feedback and monitor their continued improvement. Here are our tips on how to best deliver Worker feedback:
- Provide constructive criticism and coaching to Workers who appear to be making an effort.
- Utilize warnings for work that is not up to standards before rejecting.
- Be cautious about rejecting work and provide clear reasoning for the action.
In summary, clear instruction, fair compensation, consistent feedback and communication and the use of quality controls significantly impact your ability to derive quality content from your Workers at scale. By leveraging the strategies outlined here, you’ll be on your way to executing a successful content generation project and developing a workforce of skilled Workers eager to contribute work to your project.
About the Author: Sammie Schweissguth began working with the CrowdSource team in early 2011. As the Director of Client Services, Sammie works with the fulfillment team to manage client projects from start to finish, ensuring the on-demand workforce of 500,000 is matched up with the right work at the right time.
About CrowdSource: CrowdSource provides full-service crowdsourcing solutions to accomplish labor-intensive tasks with unprecedented speed and superior quality. By providing pre-qualified groups of skilled Workers, breaking complex tasks into microtasks and employing quality controls, CrowdSource eliminates the most common problems associated with scaling labor intensive tasks.
