The Mechanical Turk Blog

Upcoming Meetup: Edwin Chen of Google on Human Powered Machine Learning

The San Francisco Data Mining Meetup Group is hosting a crowdsourcing-themed event Thursday, June 6^th at Trulia’s Headquarters in downtown San Francisco.  The event will feature Edwin Chen, an engineer at Google, discussing: 

• Use cases including measuring the relevance of personalized search, gathering training data for machine learning classifiers and designing a system that incorporates crowdsourcing in real-time, high-quality research.
• Methods for quality control including algorithmic techniques, gold standards and the advantages, disadvantages and challenges related to building a trusted workforce.
• Getting Started with Mechanical Turk, including the Mechanical Turk Developer API, as well as the opensource tool, Clockwork Raven.

The event begins at 6:30pm with refreshments provided by G33kTalk.  Space is filling up fast, so be sure to reserve your space today.

See you there!

SF Data Mining is a bay area Meetup group that focuses on facilitating discussion related to all aspects of the data pipeline--from data acquisition and big data storage to machine learning and data visualization. To learn more about future events, check them out on Meetup.com.

Join Mechanical Turk at the San Francisco AWS Summit April 30th!

After wrapping up a great AWS Summit in New York City last week, Mechanical Turk is headed west for the next installment of the AWS Summit Series on April 30th at Moscone Center in San Francisco.

The San Francisco summit will feature a keynote by Andy Jassy, Senior Vice President of AWS, presentations from customers including Mailbox (recently acquired by Dropbox) and Nokia, and technical “boot camp” sessions from AWS engineers for both new users and experienced users looking to dive deeper.

Join us at booth #503 in the expo hall to learn more about how your business can leverage Mechanical Turk and connect with our team. We’ll be available to discuss use cases and address questions, in person.

To see the full event agenda, read about the speakers and reserve your free ticket (if you haven’t already), visit the AWS Summit – San Francisco website.

 

 

Highlights from the AMPLab Winter Retreat

Last week we had the honor of joining the team from UC Berkeley’s AMPLab at their semi-annual retreat. Focused addressing “big data” challenges using Algorithms, Machines, and People (AMP), the AMPLab is an interdisciplinary collaboration started by UC Berkeley‘s Electrical Engineering and Computer Sciences department.

For us the retreat presents an opportunity to engage with researchers focused on exploring innovative methods for collecting and applying input from the crowd. This year, we were impressed by the discussion spurred by AMP graduate students focused on using crowdsourcing to generate lists and improve query understanding. Highlights included the following two presentations:

CrowdDB: Crowdsourced Enumeration Queries

Tim Kraska (Brown) and Beth Trushkowsky (AMPLab), have previously published research based on CrowdDB, a system they developed that uses human input to address difficult computational tasks, such as matching, ranking, or aggregating search results based on fuzzy criteria. Building on their previous research, the team returned to present their work addressing challenges related to building lists using a crowd. For instance, when compiling a list, such as Italian restaurants in Seattle—is it possible to predict how much time a crowd will take to complete the list based on early progress? Further, how do you know when the list is complete? Is it possible speed completion by identifying and prompting Workers to fill gaps?

CrowdQ: Crowdsourced Query Understanding

Developed by Daniel Haas and Jonathan Harper (both of AMPLab), CrowdQ is a system that leverages human input to incorporate semantic meaning by identifying a searcher’s intent based on keyword combinations. For instance, the query “annual rainfall in Seattle” is searching for meteorological data for Seattle. Using human input, the CrowdQ system can determine that “annual rainfall in <place>” queries can be satisfied using the same retrieval process.

To learn more about the projects featured above or other research from the AMPLab, visit their website for more information and to access recent publications.

Thanks again to the AMPLab for hosting Mechanical Turk—we look forward to supporting their work as they continue to explore innovative new crowdsourcing methods and applications.

-Dr. Adam D. Bradley

Mechanical Turk in the Bay Area this week

I'll be talking about Mechanical Turk at 2 events in the Bay Area this week.

Wednesday November 9th, I'll be at the Sentiment Symposium discussing how companies are using Mechanical Turk to accurately characterize sentiment in social media. 

On Thursday November 10th, I'll be at Xerox PARC participating in a panel on Crowdsourcing: From Phenomenon to Business Model. 

Hope to see some of you at these events.

--Sharon

 

Fixing Dirty Data with Mechanical Turk – Werner Vogel’s Keynote @ Strata

A few weeks ago, I attended the O’Reilly Strata conference where Werner Vogels, CTO of Amazon, gave a keynote.  He spoke about big data – how to collect, store, organize, analyze and share data. In his talk (at minute 13:04) he covered how enterprises are using Mechanical Turk to organize their data. Specifically, how you can leverage the hundreds of thousands Workers on Mechanical Turk to: 

• Verify your data
• Correct data
• Enrich data by adding meta information to your data set
• Filter data through content moderation

He then shared a common case study of how a company managing millions of business listings incorporated Mechanical Turk into their data management workflow to optimize the accuracy of their data and improve their customer experience. In the scenario he described, the provider of business listings built a data processing engine that automatically sent data exceptions to Mechanical Turk. These exceptions were reviewed and corrected by Workers; their answers were automatically validated, placed back in the data flow and then updated on their website:

You can see all of the keynotes with transcripts on the O’Reilly summary page – click on the purple icon
( ) to read the transcript.  
Thanks to Mechanical Turk partners   and  who  provided the transcriptions for the conference!  Here’s the transcript from Werner’s presentation.

 

-- Natala

 

Net:Work Conference Recap

Last week Sharon Chiarella, VP Mechanical Turk and I attended the GigaOm Net:Work Future of Work conference in San Francisco. These events are so awesome – it is great to spend time with the Crowdsourcing community, our customers and partners. We hear such useful and insightful feedback. 

Sharon participated on The Human Cloud: Elastic Workforce in the Enterprise panel along with Alex Edelstein, CEO of CloudCrowd, Maynard Web, CEO of LiveOps and Doron Reuveni, CEO of uTest and was moderated by Niel Robertson, CEO of Trada. 

The panel was asked to provide insight into the value of labor-as-a-service within the Enterprise, the impact of Cloud-based labor on outsourcing and how the category has changed over the last few years.

Some key highlights: 

• Platforms like Mechanical Turk are speeding up the pace of innovation by allowing businesses to get work done more quickly and scale up and down with an on-demand labor pool as their needs change. 
• Anonymity of Workers was discussed. Alex Edelstein had a great parallel about the importance of a Workers identity – do you need to know who installed the steering wheel in your car to know if the work that was done was high quality? Sharon agreed that it is not an individual’s identity that’s important but the quality of their work. 

Watch the video below to get the full details from the panel. Thanks to GigaOm for hosting such a great event – we were glad to be a part of it! 

-- Natala