Editor's Note: This post is part of our series of guest blog posts from customers and developers who are building on top of Mechanical Turk. Today's post is written by Ari Bader-Natal who is the Chief Learning Architect at Grockit, the fast-growing online social learning startup backed by Benchmark, Atlas Ventures, Integral, Mark Pincus and Reid Hoffman.
Grockit is an online social learning startup. The primary learning activity on Grockit is problem solving, so it’s probably not surprising that Grockit has developed a large (and growing) library of problems to solve. What began as a few hundred problems has quickly turned into hundreds of thousands of questions, answers, explanations, and more.
When you’ve got tens or hundreds of people authoring problems, things inevitably get messy. Mathematical expressions are input in several different ways, questions often get modified and revised by a sequence of different authors, and it gets increasingly difficult to figure out who changed what, when, and how. In short, our process required that we develop a more fully-featured content management system (CMS) earlier this year. Our new CMS assumed that text is in Markdown format. All of the existing questions, answers, and explanations weren’t, though, and that left something of a challenge.
The challenge: convert each bit of content in the Grockit system from free-form HTML to a Grockit-flavored Markdown without losing the necessary visual styling, then verify that the conversion was done correctly, fix it if it wasn’t, and then deploy the approved version to the production system once ready. Then repeat, hundreds of thousands of times. Clearly, we needed an automated process. We know that automated doesn’t necessarily mean accurate, however, so we decided that a partially-automated, partially-manual process was the best way to ensure that Grockit questions, answers, and explanations would continue to be accurate.
An automated process to convert from free-form HTML to a Grockit-flavored Markdown got us started. Some of the changes were so minor that no manual verification was necessary, and the new version could be immediately deployed to the production system. For the rest, we used Amazon’s Mechanical Turk service and asked three different people whether or not the before and after content looked the same. If all three agreed that the conversion matched the original, we accepted the new content. If not, we needed someone else to check and fix the change.
Here’s an example of what the HIT on Mechanical Turk looked like:
In our trial run with Algebra I in the Academy, 45,000 conversion quality ratings were submitted in the first hour alone! Once a conversion's quality was verified by Mechanical Turk Workers, Grockit would start displaying the converted version. The result: A rolling process (without race conditions!) to update all of Grockit’s content, one field at a time, to a much cleaner, simpler, more trackable, more searchable, more flexible form moving forward.