The Mechanical Turk Blog

Mechanical Turk for research studies

A recent article in Science profiled how social scientists are using Mechanical Turk to conduct research studies.   Mechanical Turk lets scientists test and conduct studies quickly and cheaply and choose participants based on specific criteria from over 500,000 Workers in 190 different countries.

The article details how scientists are using Mechanical Turk to conduct bigger, faster and cheaper research studies.   For example, social scientists used 10,000 Turkers (as Mechanical Turk Workers are sometimes called) to create a tool for tracking the emotional content of Twitter messages (Science, 30 September, p. 1814).  In another study, Gabriel Lenz, a political scientist at the University of California, Berkeley, asked Mechanical Turk Workers to look at photographs of Brazilian political candidates and fill in a data sheet.  According to the article, Lenz and his team had all the data he needed in just a few weeks and at a total cost of $160. The results of their research appeared in the social science journal World Politics.

"The use of Mechanical Turk subjects will eventually become mainstream" Adam Berinsky,  a political scientist at the Massachusetts Institute of Technology, says in the article. Berinsky goes on to predict that many more social science papers using Mechanical Turk will appear.  “Everyone I know is using it,” he says.  "The obvious advantage is the speed and cost."

You can find the article here.

Guest Blog Post: Grockit

Editor's Note: This post is part of our series of guest blog posts from customers and developers who are building on top of Mechanical Turk.  Today's post is written by Ari Bader-Natal who is the Chief Learning Architect at Grockit, the fast-growing online social learning startup backed by Benchmark, Atlas Ventures, Integral, Mark Pincus and Reid Hoffman.

Grockit is an online social learning startup.  The primary learning activity on Grockit is problem solving, so it’s probably not surprising that Grockit has developed a large (and growing) library of problems to solve. What began as a few hundred problems has quickly turned into hundreds of thousands of questions, answers, explanations, and more.

When you’ve got tens or hundreds of people authoring problems, things inevitably get messy. Mathematical expressions are input in several different ways, questions often get modified and revised by a sequence of different authors, and it gets increasingly difficult to figure out who changed what, when, and how. In short, our process required that we develop a more fully-featured content management system (CMS) earlier this year. Our new CMS  assumed that text is in Markdown format. All of the existing questions, answers, and explanations weren’t, though, and that left something of a challenge.

The challenge: convert each bit of content in the Grockit system from free-form HTML to a Grockit-flavored Markdown without losing the necessary visual styling, then verify that the conversion was done correctly, fix it if it wasn’t, and then deploy the approved version to the production system once ready. Then repeat, hundreds of thousands of times. Clearly, we needed an automated process. We know that automated doesn’t necessarily mean accurate, however, so we decided that a partially-automated, partially-manual process was the best way to ensure that Grockit questions, answers, and explanations would continue to be accurate.

An automated process to convert from free-form HTML to a Grockit-flavored Markdown got us started. Some of the changes were so minor that no manual verification was necessary, and the new version could be immediately deployed to the production system. For the rest, we used Amazon’s Mechanical Turk service and asked three different people whether or not the before and after content looked the same. If all three agreed that the conversion matched the original, we accepted the new content. If not, we needed someone else to check and fix the change.

Here’s an example of what the HIT on Mechanical Turk looked like:

 

 

In our trial run with Algebra I in the Academy, 45,000 conversion quality ratings were submitted in the first hour alone! Once a conversion's quality was verified by Mechanical Turk Workers, Grockit would start displaying the converted version. The result: A rolling process (without race conditions!) to update all of Grockit’s content, one field at a time, to a much cleaner, simpler, more trackable, more searchable, more flexible form moving forward.