The Mechanical Turk Blog

Researchers turn to Amazon Mechanical Turk as valuable data collection tool

We wanted to share this Financial Times Magazine article (subscription required) that explores the use of Mechanical Turk to quickly conduct research on large groups of subjects.

“Psychologists, behavioural economists and political scientists have now realised that it is potentially far cheaper to pay MTurk workers — “Turkers” — to answer questionnaires and participate in activities than it is to assemble a bespoke online panel or to conduct the research in a laboratory filled with student participants sitting at computers.”

“‘I think [Mechanical Turk] is one of the most important and beneficial innovations in the history of psychology,’ he says, before adding the obvious caveat that like any research tool, it must be wielded with skill.”

Get started as a Requester at https://requester.mturk.com/.

Crowdsourcing speech rating data through Mechanical Turk

We came across Professor Tara McAllister Byun’s video on her use of Mechanical Turk for clinical studies involving speech ratings. Watch this seven minute video to learn how expert raters and raters crowdsourced through Mechanical Turk performed, as well as the other benefits she realized through the wisdom of crowds.

Top Image Systems’ eFLOW CrowdBridge Voted One of Three Best ECM Solutions of 2014

Mechanical Turk Technology Partner Top Image Systems (TIS) recently announced that its innovative crowdsourcing solution, eFLOW® CrowdBridge, has been nominated as one of the three best Enterprise Content Management (ECM) solutions of 2014 by an independent panel of industry experts at the 2014 annual ECM Day held in Baden, Switzerland on November 11, 2014.

The ECM Awards, sponsored by Pentadoc and ComputerWeek, honor solutions connecting people and knowledge that are strongly focused on users’ needs.  In the ECM Solution of the Year category, eFLOW CrowdBridge was recognized for its data validation capabilities via Amazon Mechanical Turk, which enable eFLOW customers to flexibly and dynamically employ data entry resources in correspondence with the actual volume of work. Through eFLOW CrowdBridge, processing bottlenecks associated with peak times can be avoided, while operating costs can be reduced when the data entry workload is low.

You can learn more about this award at: http://www.topimagesystems.com/news-a-events/awards/1056-2014-ecm-award-finalist-eflow-crowdbridge

Congratulations!

Ted Talk - What’s wrong with your pa$$w0rd?

Chalk this up as another clever use case.

Watch how Lorrie Faith Cranor of Carnegie Mellon University sourced 5000 passwords on Mechanical Turk for her research study on pa$$words.

And kudos to Turkers who participated in this study (fast forward to around 14:20):

... The other interesting thing that we found is that when we compared the Carnegie Mellon passwords to the Mechanical Turk-generated passwords, there was actually a lot of similarities, and so this helped validate our research method and show that actually, collecting passwords using these Mechanical Turk studies is actually a valid way to study passwords. So that was good news.

Assessing risk in pre-emptive shipping program using 'artificial artificial intelligence'

We came across this Q&A with Igor Ebert on techtarget.com that we want to share.  Read how Workers on Mechanical Turk help Gilt know what to ship before you buy it.

Captricity Uses Mechanical Turk to Process Paperwork Backlog for the FDA

Last week the U.S. Food and Drug Administration (FDA) announced an a new partnership with Captricity, a HIPAA compliant document processing firm that leverages a mixture of optical character recognition (OCR) and Mechanical Turk to digitize handwritten documents.

The FDA receives 900,000 drug accident safety reports annually. Most of these documents are submitted digitally, but ten percent are submitted on paper. These paper submissions were digitized manually by data entry staff at the FDA – a process that, according to a June 2013 report, was creating a massive backlog in paperwork.

Captricity recently completed a proof of concept that demonstrated paper submissions could be handled 50 times faster at a fraction of the cost with the same accuracy as their internal staff.  To read more, see Fast Company’s recent commentary on Captricity’s partnership with the FDA, as well as coverage from CivSource, a publication covering technology-related news in the public sector.

Canada’s National Research Council Builds Huge Word-Emotion Lexicon using Mechanical Turk

Recently, MIT Technology Review announced the release of a new word-emotion and word-polarity lexicon developed by Saif Mohammad and Peter Turney from Canada’s National Research Council (NRC). The lexicon includes 10,000 words and their associated emotions and polarity, and according to the researchers, was built cost-effectively in the span of a few weeks using Mechanical Turk.

Companies are increasingly using software to mine data from social sites, like Twitter and Facebook, to understand public opinion toward brands, products and services. However, the accuracy of these applications is limited by the software’s ability to interpret the emotion associated with each word and most existing technologies rely on relatively limited or small emotion lexicons. 

We reached out to the Mohammad and Turney to learn more about their methods for building their lexicon using Mechanical Turk. Here are a few kernels of advice that they had to share with other Requesters based on their experience:

• Design and implement an effective quality control strategy.  Key to Mohammad and Turney’s quality control was their use of Known Answers.  In each HIT, the researchers primed Workers by presenting a word choice question to assess each Worker’s familiarity with the word to which they would later be asked to associate an emotion.  According to Mohammad, “Prompting Workers to consider the word’s meaning at the beginning of the task put them in a better mindset to answer the following emotion association question and led to higher quality annotations.”

• Simplify HITs by breaking workflows into simple steps.  To increase Workers’ speed, while also improving the quality of their outputs, the researchers recommend breaking complex questions into simpler “bite-size” questions.  According to Mohammad, “If it is easy for Workers to quickly comprehend questions and provide answers, the quality of Workers’ answers will be higher and more Workers will be more interested in contributing to your HITs.”

• Present clear and concise instructions.  Early in their HIT design process, Mohammad and Turney tested variations on the wording of their instructions. For instance, in one test they asked Workers if a word “evokes” a specific emotion; whereas, in a second test they asked if a word “is associated” with a specific emotion. The test revealed that agreement between Workers was more frequent when the instructions included the term “is associated”. Mohammad explained, “The experiment confirmed our intuition that 'evokes' may encourage subjective answers whereas 'is associated' guides Turkers to more objective answers.”

• Offer appropriate compensation to target the right Workers.  Setting a competitive HIT price is critical to attracting the right Workers.  However, according to Mohammad, “Paying more does not yield higher quality results, but paying too little may discourage the right Workers from your HITs.”

For Mohammad and Turney, the tactics above significantly impacted the quality of their results and the speed with which they were able to build their lexicon.  By optimizing instructions, workflow, and pricing, they found they not only attracted the best Workers for their task, but kept them engaged in their HIT.  Further, by utilizing scalable quality control tools, such as Known Answers, they were able to reliably infer the accuracy of the annotations collected as their project progressed.

To learn more about their project, see Mohommad and Turney’s research paper available here.

LinkedIn Scales Business Card Transcription with Mechanical Turk

Today, we’re excited to announce a new Mechanical Turk case study from LinkedIn, the world’s largest professional online network. The new case study details how LinkedIn is able to manually transcribe tens of thousands of business cards each day to support CardMunch, an iOS application that turns business cards into LinkedIn connections and digital contacts.

 

CardMunch’s developers integrated with Amazon MechanicalTurk in order to achieve the transcription accuracy required for a great customer experience. They considered using off-the-shelf optical character recognition (OCR) solutions, but the OCR didn’t provide accurate enough results – every character needs to be transcribed correctly in order for the information to be useful to CardMunch users. With Amazon Mechanical Turk, the team was able to achieve near-absolute accuracy using human transcribers from the Mechanical Turk marketplace.

Millions of cards have been transcribed using Amazon Mechanical Turk since the launch of the CardMunch app. “With the Amazon Mechanical Turk marketplace, we manually transcribe tens of thousands of cards each day, while having the ability to burst up and scale down as needed,” says Sid Viswanathan, the developer and product manager of CardMunch. “We’ve been able to drive down operational costs while not having to guess at our capacity needs.”

To learn more about how LinkedIn is able to utilize Mechanical Turk to scale digital transcription, and explore other use cases from customers, such as Acxiom, AOL, and DARPA, see all of our case studies online on the Mechanical Turk Requester site.

 

How SnipSnap Built and Scaled Mobile Couponing with People

When Ted Mann founded SnipSnap, the popular mobile couponing application, he didn’t anticipate the challenge his team would encounter converting printed coupons into a digital, mobile-ready offers using automated solutions, like optical character recognition. In a recent post on Fast Company’s blog, Co.LABs, Mann discusses how SnipSnap addressed the challenge by integrating with Mechanical Turk to develop a scalable data extraction workflow able to process more than 200,000 coupons per day.

Here are some of the kernels of wisdom Mann had to offer for new Requesters based on his own experience implementing Mechanical Turk:

• Implement effective quality control workflows.  Mann specifically recommends using tools like Known Answers to audit Workers in real-time and Plurality to measure consensus between Workers.

• Streamline Workers’ user experience when designing your task to make it as simple and speedy as possible. For instance, consider adding auto-completes to fields, making it easy to tab between fields, and minimizing scrolling.

• Maintain consistent HIT pricing, if possible. Use caution when modifying HIT prices.  Changing prices on HITs (especially reducing the price of HITs) can negatively impact your relationship with Workers.

• Spend an hour doing your own tasks. This will give you a sense of whether or not you’re tasks are appropriately priced and help identify opportunities to streamline your Worker experience. 

By the way, in case this use case sounds familiar to you, we recently featured a guest blog post from the team at Upside Commerce – another mobile app developer using Mechanical Turk to extract data from print coupons for mobile delivery.  You can read more about Upside’s story here.

Upside Commerce Moves Print Coupons to Mobile with Mechanical Turk

Editor’s Note: Today’s post is part of our Featured Requester series highlighting examples of how Mechanical Turk Requesters are utilizing our marketplace and the results they’re able to achieve. This post is from Jeff Sampson, founder and CEO of Upside Commerce.

Everyone loves a good deal....When advertising firm DDB asked Facebook fans why they “liked” a brand on the platform, 83% shared they were looking for “special deals and offers.”  Two years later, the top 100 brands on Facebook have a combined total 525 million “fans”, indicating a rapid shift in the way consumers engage with their favorite brands. However, despite the growth of brands’ fan bases on social networks, marketers have limited tools to engage.

Upside Commerce provides a platform to connect consumers on social media sites to deals and coupons from the brands they love most. Our platform enables marketers to distribute coupons and deals via a mobile app to “fans” based on their interests.  Users of the app are able to store these coupons on their phones and can easily redeem them at local retailers. For marketers, Upside Commerce provides real-time data to measure campaign performance, including conversions, shares, redemptions, locations and total revenue.

When we first launched the Upside mobile app, we lacked deal inventory. We feared that if users weren’t able to see deals from their favorite brands, their experience using the app would suffer. To address this problem, we considered developing a web scraper to crawl deals published online, but this approach took too much time and required that we divert our developers away from core product features.

Deals for fast moving consumer products are still primarily published in print - not surprising given that fulfillment for these products typically happens at offline retailers, such as grocery stores. We realized we needed a way to digitize thousands of paper coupons every week.

To accomplish this task, we decided to try Mechanical Turk.  We scanned thousands of print coupons, embedded digital images of the deals into a Mechanical Turk HIT, instructed Workers to extract key data elements– things like the face value, brand, expiration, etc .  

Below is a screenshot of the HIT below that illustrates how we designed the interface:

To ensure quality results, we only allowed access to Masters Workers [link to Masters post], and we had up to three Workers complete each task so that we could compare their answers. If the Workers’ inputs matched – we would add the deal to our database.

Our Results

Using Mechanical Turk we were able to digitize receipts in a fraction of the time at a significantly lower cost than other methods. Here are a few statistics that represent the results that we’ve been able to achieve: 

1. Greater than 99% accuracy with Master Workers;
2. Each batch of 100 coupon images was transcribed in less than 15 minutes - in most cases less than 10 minutes; and
3. Workers took an average of 33 seconds to complete each HIT.

Mechanical Turk provides us with the flexibility we need to be nimble and build the product inventory required for our launch. As we move forward, we plan to continue to leverage Mechanical Turk to expand our coupon inventory globally, as well as create new digital content (through Worker sourced images) related to products and brands.  

Learnmore about the Upside Commerce and their mobile app, Upside, on their website.