The Mechanical Turk Blog

Upside Commerce Moves Print Coupons to Mobile with Mechanical Turk

Editor’s Note: Today’s post is part of our Featured Requester series highlighting examples of how Mechanical Turk Requesters are utilizing our marketplace and the results they’re able to achieve. This post is from Jeff Sampson, founder and CEO of Upside Commerce.

Everyone loves a good deal....When advertising firm DDB asked Facebook fans why they “liked” a brand on the platform, 83% shared they were looking for “special deals and offers.”  Two years later, the top 100 brands on Facebook have a combined total 525 million “fans”, indicating a rapid shift in the way consumers engage with their favorite brands. However, despite the growth of brands’ fan bases on social networks, marketers have limited tools to engage.

Upside Commerce provides a platform to connect consumers on social media sites to deals and coupons from the brands they love most. Our platform enables marketers to distribute coupons and deals via a mobile app to “fans” based on their interests.  Users of the app are able to store these coupons on their phones and can easily redeem them at local retailers. For marketers, Upside Commerce provides real-time data to measure campaign performance, including conversions, shares, redemptions, locations and total revenue.

When we first launched the Upside mobile app, we lacked deal inventory. We feared that if users weren’t able to see deals from their favorite brands, their experience using the app would suffer. To address this problem, we considered developing a web scraper to crawl deals published online, but this approach took too much time and required that we divert our developers away from core product features.

Deals for fast moving consumer products are still primarily published in print - not surprising given that fulfillment for these products typically happens at offline retailers, such as grocery stores. We realized we needed a way to digitize thousands of paper coupons every week.

To accomplish this task, we decided to try Mechanical Turk.  We scanned thousands of print coupons, embedded digital images of the deals into a Mechanical Turk HIT, instructed Workers to extract key data elements– things like the face value, brand, expiration, etc .  

Below is a screenshot of the HIT below that illustrates how we designed the interface:

To ensure quality results, we only allowed access to Masters Workers [link to Masters post], and we had up to three Workers complete each task so that we could compare their answers. If the Workers’ inputs matched – we would add the deal to our database.

Our Results

Using Mechanical Turk we were able to digitize receipts in a fraction of the time at a significantly lower cost than other methods. Here are a few statistics that represent the results that we’ve been able to achieve: 

1. Greater than 99% accuracy with Master Workers;
2. Each batch of 100 coupon images was transcribed in less than 15 minutes - in most cases less than 10 minutes; and
3. Workers took an average of 33 seconds to complete each HIT.

Mechanical Turk provides us with the flexibility we need to be nimble and build the product inventory required for our launch. As we move forward, we plan to continue to leverage Mechanical Turk to expand our coupon inventory globally, as well as create new digital content (through Worker sourced images) related to products and brands.  

Learnmore about the Upside Commerce and their mobile app, Upside, on their website.

Working Smart: A Worker's Perspective on Selecting HITs

Editor’s Note: Today’s post is part of our Worker Corner series featuring insights, advice and experiences from Mechanical Turk Workers. We’ve invited Jeremy Lesh, a Master Worker, to share a few thoughts on how he approaches HIT selection – valuable insight for Requesters looking to engage Workers, as well as Workers looking to optimize for efficiency.

How I Discovered Mechanical Turk

In 2010, while searching online for “ways to make extra income,” I found a blog post about Mechanical Turk. I signed up as a Worker and was quickly making money doing a variety of small tasks, such as finding phone numbers and addresses online. I was fascinated by the possibility of earning money anytime, from virtually anywhere.

By the spring of 2012, I was regularly contributing to a variety of HITs posted by Requesters on Mechanical Turk, including photo moderation HITs from Adult Image Review, data categorization HITs from Readabl and survey HITs from top academic institutions. Steadily, I began to develop a group of Requesters for whom I would work when their HITs were available.

Working Smart and Picking the Right HITs

With more than 180,000 Assignments under my belt to date, I’ve developed several strategies for maximizing my efficiency – and subsequently, my earnings – the most important of which is HIT selection. Here are three key factors I consider when reviewing HITs: 

1. My past experience working for the Requester;
2. How well the Requester pays relative to other HITs available in the marketplace – I only accept HITs I feel are fairly priced; and
3. How quickly the Requester approves or rejects HITs, along with my level of confidence that my submissions will be approved.

Typically, I prioritize my favorite Requesters first. This is in part because I have a relationship with the Requester and know what to expect from them, but also because I’m able to become more efficient over time as I gain practice on their HITs.

Since I earned my Masters Qualification, I also tend to focus on HITs that require the Masters. Or even better, HITs that use the Categorization App because of the efficient design of the task interface. I enjoy research surveys as well, which are often amongst the highest paying HITs in the marketplace. 

Occasionally, I consider HITs from new Requesters. However, before accepting a HIT, I check the Requester’s Turkopticon rating – assuming it’s available. Many Workers use Turkopticon, a third-party service that aggregates Workers’ reviews of Requesters, to gauge reputation for communication, generosity, fairness and promptness in payment.  If I see a high volume of relatively low Requester ratings, I typically will pass over the Requester’s HITs – in fact, I often encourage Requesters to create a Worker account and monitor their ratings and feedback using the Turkopticon application.

Here are a handful of some of my personal favorite Requesters and a bit on why I prioritize their HITs:

1. Two Lakes Research typically is one of the fastest-approving categorization Requesters. 
2. William Letson and Matt Park both require Masters and provide simple categorization HITs along with fair and quick approvals.
3. Coman Petre also requires Masters and is the highest rated Requester according to Worker reviews on Turkopticon. This Requester also approves quickly and offers HITs that many Workers find enjoyable.
4. Set Master Account offers very simple Yes/No HITs and typically pays within 24 hours. 

This list of Requesters is not exhaustive; I also recommend CastingWords or SpeechInk to Workers who are interested in transcription HITs and Crowdsource’s writing and editing HITs, which tend to require more skill and more creativity. To Requesters, I recommend taking a look at some of the Requesters listed above, perhaps even attempting a few of their HITs first-hand, to get a sense of some of the features that make these HITs appealing to Workers.

About the Author: Jeremy Lesh, a Master Worker based out of Ann Arbor, Michigan, has completed more than 180,000 assignments with an approval rate of over 99%. 

Partner Spotlight: Successful Strategies for Content Generation from the Experts at CrowdSource

Editor's Note: Today's blog post is part of our Partner Spotlight series featuring best practices from Mechanical Turk Solution Providers.  This post is from Sammie Schweissguth, Director of Client Services at CrowdSource. 

Whether you are looking to improve your current content generation HITs or are just getting started, using an effective strategy to attract and manage Workers is critical to successful content generation on Mechanical Turk.

With hundreds of thousands of Workers available, Requesters must identify Workers possessing top-notch writing and editing abilities, tap into their skills and manage them on an ongoing basis. How do you accomplish this? Based on our experience at CrowdSource, the following eight elements can make or break your strategy to generate great content:

1. Establishing a style guide
2. Qualifying and recruiting Workers
3. Paying Workers appropriately
4. Writing clear instructions
5. Communicating with the workforce
6. Using a workflow that breaks up tasks
7. Developing a quality control strategy
8. Providing performance feedback 

 

Establishing a Style Guide

Naturally, every Worker has a different approach to writing. However, communicating clear guidelines relating to style, grammar and format can help create consistency in the quality of content from Workers, especially when you are looking to produce content at scale. Whether you’re using an existing style guide or creating your own from scratch, include detailed descriptions and make sure the guidelines are easily accessible to Workers while they work. Consider the following helpful tips for creating a guide:

• Clearly define any specific style preferences relating to grammar, spelling, formatting, voice, point of view, etc.
• Consider using existing resources, such as the AP or the Yahoo! style guidelines for web content.
• Provide examples of content that meet your style requirements.

 

Qualifying and Recruiting Workers

Mechanical Turk provides access to an on-demand workforce around-the-clock. However, not every Worker is capable of producing content that meets your requirements. For this reason, creating a qualification test to identify Workers with excellent grammar, strong writing skills and the ability to follow instructions before they work on your HITs is very important. Here are some additional considerations based on our experience building qualified workforces at CrowdSource:

• Create two separate qualification tests (one for writing; one for editing).
• Restrict newly qualified Workers to the most basic HITs and increase access to advanced tasks based on performance.
• Consider incorporating writers on staff into your workflow—these writers can contribute the most advanced tasks, while Workers from Mechanical Turk can be leveraged to achieve scale. 

 

Paying Workers Appropriately

How you price HITs affects the type and quality of Workers you attract. An advantage of crowdsourcing content creation is that it’s less expensive than utilizing an internal team. However, cost efficiency shouldn’t come at the cost of Worker compensation. Pricing your HIT appropriately for the quality of work you expect is vital to cultivating a loyal group of Workers. Here are a few tips we use at CrowdSource to set HIT price:

• Aim to pay at least minimum wage for basic writing tasks.
• Increase the pay for more difficult, specialized or research-intensive writing.
• Understand that paying fairly tends to attract higher-quality, loyal Workers. 

 

Writing Clear Instructions

Workers will not meet your expectations without knowing what they are. Clearly define all instructions and make requirements obvious. At the same time, consider revising your instructions as your project progresses based on Worker feedback and results. For content that requires more structure, provide step-by-step instructions throughout the HIT to keep Workers on track. Ideally, your instructions should address and answer as many potential questions as you can identify. Generally speaking, the following are great tips to keep in mind:

• Keep instructions clear and to the point.
• Provide examples to illustrate ideal content.
• As instructions change, clearly indicate updates in the HIT instructions.  

 

Communicating with the Workforce

Even with clear instructions, Workers are likely to have occasional questions. Respond personally to emails and forum posts asking for clarification. Engage in friendly interactions with Workers on a regular basis through Worker forums. In our experience, direct communication fosters relationships, creates accountability and improves quality. Tips to consider when communicating with Workers:

• Be open and honest.
• Respond to personal emails in a timely fashion.
• Communicate regularly via forums such as TurkerNation.
• Listen to and consider Worker feedback—they can help spot issues and save you time.

 

Using a Workflow that Breaks Up Tasks

Breaking work into more digestible HITs can make tasks easier for Workers to manage, improving the quality of their work. By separating the tasks of writing and editing, content is cross-checked by another Worker, and subpar work is less likely to make it through your process. Based on our experience, we advise you to do the following when designing your workflows:

• Split writing and editing steps into separate tasks.
• If it’s required, separate preliminary research from the writing step.
• Include a third step for annotation when necessary.

 

Developing a Quality Control Strategy

In addition to breaking down workflows into multiple tasks to improve quality, implementing caps on the number of writing or editing HITs available to a new Worker provides additional control. Similarly, recruiting Workers with proven performance to review others’ content can be a very effective strategy for managing quality. The following are common methods we use at CrowdSource to ensure quality results:

• Use HIT submission limits for new, unproven Workers.
• Utilize the highest-performing Workers to review and edit submitted content.
• Use feedback from reviews as measures of performance.

 

Providing Performance Feedback

Providing feedback on performance to Workers can lead to better work. At CrowdSource, we use the ability to reject a Worker only as a last resort to maintain a good relationship and reputation with Workers. Instead, foster the growth of Workers through feedback and monitor their continued improvement. Here are our tips on how to best deliver Worker feedback:

• Provide constructive criticism and coaching to Workers who appear to be making an effort.
• Utilize warnings for work that is not up to standards before rejecting.
• Be cautious about rejecting work and provide clear reasoning for the action.

 

In summary, clear instruction, fair compensation, consistent feedback and communication and the use of quality controls significantly impact your ability to derive quality content from your Workers at scale. By leveraging the strategies outlined here, you’ll be on your way to executing a successful content generation project and developing a workforce of skilled Workers eager to contribute work to your project.

 

About the Author: Sammie Schweissguth began working with the CrowdSource team in early 2011. As the Director of Client Services, Sammie works with the fulfillment team to manage client projects from start to finish, ensuring the on-demand workforce of 500,000 is matched up with the right work at the right time.

About CrowdSource: CrowdSource provides full-service crowdsourcing solutions to accomplish labor-intensive tasks with unprecedented speed and superior quality. By providing pre-qualified groups of skilled Workers, breaking complex tasks into microtasks and employing quality controls, CrowdSource eliminates the most common problems associated with scaling labor intensive tasks.