The Mechanical Turk Blog

Introducing eFLOW5: A Mechanical Turk-Assisted Digital Transcription Solution from Top Image Systems

Top Image Systems (TIS) – the maker of eFLOW, an end-to-end document processing solution that extracts, classifies and validates data from documents – will launch the latest version of their eFLOW product line at a special reception in New York City on June 6^th. In association with the launch, TIS will share detail on how the latest version of eFLOW integrates with Mechanical Turk to handle difficult to automate data processes, as well as their participation in the Amazon Partner Network.

According to Omri Gelb, SVP & GM TIS’ US Operations:

We’re excited about the benefits Mechanical Turk will provide our customers in their document processing workflows. Combining our unique eFLOW technology with the scalability and on-demand nature of the Mechanical Turk workforce will increase both accuracy and throughput.

If you’d like to learn more about eFLOW’s document processing capabilities and the new partnership between Amazon Mechanical Turk and TIS, please join TIS at the launch reception next week.  John Hoskins from Mechanical Turk will present how one customer is leveraging eFLOWand Mechanical Turk to improve the speed and scalability of their billing process, significantly reducing fixed costs.

Click here to register and reserve your space at the event.

Upcoming Meetup: Edwin Chen of Google on Human Powered Machine Learning

The San Francisco Data Mining Meetup Group is hosting a crowdsourcing-themed event Thursday, June 6^th at Trulia’s Headquarters in downtown San Francisco.  The event will feature Edwin Chen, an engineer at Google, discussing: 

• Use cases including measuring the relevance of personalized search, gathering training data for machine learning classifiers and designing a system that incorporates crowdsourcing in real-time, high-quality research.
• Methods for quality control including algorithmic techniques, gold standards and the advantages, disadvantages and challenges related to building a trusted workforce.
• Getting Started with Mechanical Turk, including the Mechanical Turk Developer API, as well as the opensource tool, Clockwork Raven.

The event begins at 6:30pm with refreshments provided by G33kTalk.  Space is filling up fast, so be sure to reserve your space today.

See you there!

SF Data Mining is a bay area Meetup group that focuses on facilitating discussion related to all aspects of the data pipeline--from data acquisition and big data storage to machine learning and data visualization. To learn more about future events, check them out on Meetup.com.

The AWS Partner Network Helps Partners Build Their Businesses

We’re pleased to announce the addition of Mechanical Turk Partners to the AWS Partner Network (APN) Portal, a Partner-only section of AWS’ website. The APN Portal provides our Partners the tools and resources they need to grow their businesses on Mechanical Turk. Through the APN Portal, Partners can access technical and business teams, request marketing and sales support, download Partner-focused content and connect with other Partners in our global community.

Partners are organized into two functional categories:  

1. APN Technology Partners who provide a service or capability on top of Mechanical Turk. Technology partners include SaaS and PaaS providers. 
2. APN Consulting Partners who provide services to help customers build their own implementations on top of Mechanical Turk. Consulting Partners include systems integrators and managed services providers. 

The Standard and Advanced Partner Tiers allow Partners to differentiate themselves based on their longevity and expertise with Mechanical Turk.

Visit the new APN Portal on the AWS site to explore how to become a Partner.

Join Mechanical Turk at the San Francisco AWS Summit April 30th!

After wrapping up a great AWS Summit in New York City last week, Mechanical Turk is headed west for the next installment of the AWS Summit Series on April 30th at Moscone Center in San Francisco.

The San Francisco summit will feature a keynote by Andy Jassy, Senior Vice President of AWS, presentations from customers including Mailbox (recently acquired by Dropbox) and Nokia, and technical “boot camp” sessions from AWS engineers for both new users and experienced users looking to dive deeper.

Join us at booth #503 in the expo hall to learn more about how your business can leverage Mechanical Turk and connect with our team. We’ll be available to discuss use cases and address questions, in person.

To see the full event agenda, read about the speakers and reserve your free ticket (if you haven’t already), visit the AWS Summit – San Francisco website.

 

 

Introducing Multi-level Categorization and up to 100 Category Choices

Since we introduced the Categorization Application (App), Requesters have asked us to add support for more categories and multiple levels.  We’re pleased to announce that the Categorization App now supports ten top level categories with ten subcategories each, for a total of one hundred categories.  

The Categorization App incorporates the best practices and insights we’ve collected helping Requesters achieve success including everything you need to be immediately successful:

• predesigned HIT interfaces (no HTML required),
• the ability to customize your own categories,
• built-in tools that make it easy to review results, and
• Recommended pricing for categorization HITs.

To create HITs with two tiers of categories, the Categorization App guides you through four simple steps: creating the categories, uploading your data to be categorized, previewing your HIT and check out.  First provide your top-level categories.  Then, add a second level of categories by selecting “Create Subcategories” and entering the subcategories you wish to include under each top-level category.  In the example below, if a Worker is categorizing a sleigh bed, you would expect a result of “Furniture” as the top level category and “Beds” as the subcategory. 

Once you’ve provided the category names, you will be asked to provide instructions. It’s important to provide clear selection criteria for your categories to guide Workers and avoid confusion. We recommend providing Workers examples of items that should be included in or excluded from each category as additional guidance (see screenshot below).   

At checkout, the Categorization App recommends a Worker reward based on the price of similar HITs in the Mechanical Turk marketplace enabling Requesters to adjust pricing if necessary.

To check out the new Categorization App features, select “Categorization” under the “Create” Tab from the Mechanical Turk Requester home page.   

--Andrea

Reverse Rejected Assignments in the Requester User Interface

Today, we’re pleased to share that Requesters now have the ability to reverse a rejection by approving a previously rejected assignment in the Requester User Interface.  Sometimes mistakes happen, and we wanted to enable Requesters to correct their mistakes and ensure that Workers are paid for good work.

To approve a previously rejected assignment, download your batch CSV file by selecting “Download CSV” from the “Results” page for your batch.

In the downloaded CSV file, mark an assignment as approved by putting an “X” in the “Approve” column while removing text from the “Reject” column.  In the example below, I approved the assignments with the answers bed, cook and grass.  

Once you’ve marked the rejected assignments that you want to approve, upload the updated CSV file by selecting “Upload CSV” from the “Results” page for your batch.  

You will receive a confirmation to approve the assignments. By approving the assignments, the Worker’s statistics will be updated and they will be paid from your pre-paid balance, so please double check your account to ensure you have funds available.

If your HITs were created in the API, you can reverse a rejection by approving the assignment using the ApproveRejectedAssignment API operation.  More information on this operation is available in our API reference guide here.

We hope you enjoy this new features, and Happy New Year from Mechanical Turk!

--Andrea

Announcing the Sentiment App on Mechanical Turk

Today, we’re pleased to announce the launch of the Sentiment Application (App)—an automated approach to crowdsourcing sentiment ratings using Mechanical Turk. Whether you're looking to analyze sentiment on reviews, photos, or other digital assets, the Sentiment App streamlines the process of managing projects on Mechanical Turk, making it easy to get quick results. 

The Mechanical Turk crowd is commonly used by Requesters to rate sentiment. For instance, a consumer products company may want to analyze the sentiment of social media content to understand public perception of a new product. Alternatively, an e-retailer may be interested in analyzing reviews based on how positive or negative consumers feel about a product. With the Sentiment App, Requesters have the ability to access scalable human intelligence to assign meaning to digital content, a process that often requires nuanced and subjective analysis to produce accurate results.

Similar to the Categorization App, the Sentiment App incorporates the best practices and insights we’ve collected helping Requesters achieve success and includes everything you need to be immediately successful:

• predesigned HIT interfaces (no HTML required),
• the ability to specify sample size, and
• built-in tools that make it easy to analyze results.

Getting started is simple.  Access the Sentiment App by selecting “Sentiment” under the “Create” tab from the Requester home page.

Enter the title of your project and define the question that you would like to ask Workers.  In this example, we’re asking Workers to analyze whether or not tweets related to the Sentiment App are positive, neutral, or negative in nature.

Next, define your Worker instructions and determine the number of Workers you’d like to include in your sample. The Sentiment App asks Workers to rate the sentiment of your items based a five point scale and the instructions you provide. 

Once your instructions are defined, upload your CSV, and select the type of data you would like analyzed (text or photos).  You also have the option of labeling the type of data for Workers.  For example, in this example we’re analyzing tweets, so I’ve labeled this data element as a tweet.

After you preview your HIT, select the rate that you intent to pay for each Worker’s submission from the Checkout page and publish your HIT.  We recommend a price based on similar work on Mechanical Turk. 

When your work is complete, download and review Worker submissions from the “Manage Tab”.  Select your batch to see your "Answer Summary" (pictured) for a high-level view of how Workers have rated your items and click the chart to see Worker responses to individual HITs filtered by rating.

Alternatively, you can view all results by clicking on “Results” in the left navigation.  From the “Results” section all results using the review carousel or filter based on ratings using the dropdown on the upper right.

We hope that you enjoy this new simplified approach to collecting sentiment ratings on Mechanical Turk. To learn more about the Sentiment App visit the Requester website at requester.mturk.com.

Sharon Chiarella to Speak at MCNY in New York City

Sharon Chiarella, VP of Mechanical Turk, is speaking in NYC on October 2^nd at an event celebrating the National Endowment for the Humanities Grant that was recently awarded to the Museum of the City of New York and Tagasauris (a Mechanical Turk partner).The National Endowment for the Humanities Grant will improve the Museum’s digital record annotation.  Sharon will be speaking on crowdsourcing via Mechanical Turk and the benefits of on-demand, elastic staffing.

If you're located in New York City, we'd love to see you at the event. Learn more and register here. 

Twitter Launches Clockwork Raven: an Open Source Mechanical Turk Application

Last week, Twitter announced the launch of Clockwork Raven, an open source web application built on Mechanical Turk’s platform.

Originally, Twitter developed Clockwork Raven as an internal tool with the following goal in mind: optimize the speed and cost efficiency of collecting accurate human evaluations used in the development of new and existing products.

Twitter leverages Clockwork Raven in critical workflows requiring human intellect that cannot be automated. Examples include collecting thousands of subjective judgments from Workers (“How relevant is this Tweet to this user?”), conducting research (“Find 10 Twitter users that relate to this topic”) and evaluating the effectiveness of algorithms (“Does this topic match this Tweet?”).

Now, as an open source application, Clockwork Raven’s flexible interface can be customized and used by anyone to access Mechanical Turk’s scalable, on-demand workforce. The application’s features include a drag-and-drop HIT builder that enables you to design your own components, the ability to target and manage top-notch workers and integrated tools for analyzing results across multiple dimensions.

To learn more about how you or your company can use Clockwork Raven to leverage Mechanical Turk see the application on GitHub.

Read more about the launch on Twitter’s Engineering Blog.

Clockwork Raven was developed by Ben Weissman with the support of Edwin Chen and David Buchfuher.

New Categorization Application Features

We wanted to take a moment to share a few exciting and recent enhancements to the Categorization Application that make it easier for Requesters to analyze results and reuse previously created templates.

New Results Review Carousel

First, we’ve launched a new feature enabling Requesters to review Workers’ results in the context of the HIT as it was presented each Worker.  By reviewing results in this way, Requesters can more thoroughly understand the reasoning behind the Workers’ results and discover potential opportunities for improvement. 

For example, if you’re using the Categorization Application to categorize images, you now have the ability to review the image and the instructions alongside the Workers’ answers.  Reviewing results in this way, as opposed to spreadsheet format, can be helpful in understanding potential causes of Worker disagreement, or alternatively, understanding why Workers agree on a result. 

As a Requester, you have the ability to review all results in the carousel or filter results based on whether or not Workers agreed or disagreed (see example of filter options in the image above).  In some cases, reviewing disagreements and agreements can expose opportunities to clarify instructions and subsequently improve the quality of results on future batches.

Access the results carousel from the “Manage” tab.  Once there, select the project you wish to review and then click “Results” under “Categorization Project” in the left-side navigation.  Click the arrow on the right of the result image to scroll through individual HIT results.

Reuse Projects Created with Categorization App

Additionally, you can now launch new batches using previously created categorization projects. Reusing projects was not previously possible for projects created with the Categorization Application.

To launch a new batch using an existing project, select the “Create” tab to get started.


Next, click the “New Batch” button next to the appropriate project (pictured above) under “Start a New Batch with an Existing Project”. If you would like to make edits to your project, click “Edit” to make changes to the HIT properties or design layout.

We hope that you find both of these enhancements to be helpful and that they make the process of reviewing results and launching new batches on existing projects easier and more efficient.

--The Mechanical Turk Team