The Mechanical Turk Blog

A simpler, more intuitive checkout experience for purchasing Prepaid HITs

Requester customers have asked to make purchasing Prepaid HITs for Mechanical Turk simpler. Requesters use Prepaid HITs to cover Worker payments and Mechanical Turk fees when HITs are approved or bonuses granted. Today, we’re pleased to announce improvements to the way Requesters purchase Prepaid HITs.

With these improvements, Requesters can now complete their purchase in as few as three clicks, all without leaving the Mechanical Turk site. Now, Prepaid HITs can be purchased with a credit or debit card directly within Mechanical Turk.

This new functionality is available for Requesters from the “My Account” page. After selecting a purchase amount, customers proceed to the Purchase Prepaid HITs page. Here, customers can select or add a payment method directly. After confirming the purchase, the transaction is complete.

To begin using the new Prepaid HIT experience, log in and select “My Account” in the upper right corner. For additional information on purchasing Prepaid HITs please visit our FAQ.

We expect these improvements will make it even easier for Requesters to create even more HITs for Workers in the Mechanical Turk marketplace. For any questions about the new Prepaid HITs purchasing experience, please contact us.

Helpful Blog Posts To Help You Design Your HITs

A well-designed HIT can improve Worker participation, accuracy, and throughput. We have written a series of blog posts on the subjects of how-to, HIT design, tips & tricks, and HIT critiques to help Requesters design their HITs. This post compiles a list of some of the most popular posts, making them easily accessible from a single location.

How To's

If you are new to Mechanical Turk, you may want to start by learning which interface (UI vs. programming interface) to choose for your need.  Once you have made your choice, you may wonder how you can provide your input data to your HIT.  These three posts answer such questions.

• Helpful Tips for Choosing the Right Mechanical Turk Interface
• Using CSV Files to Create Multiple HITs in the Requester UI
• Hosting Your HIT Images Using Amazon S3

HIT Design – Working with Project Templates

There is a number of project templates available on the Requester website to help you create your HIT.  The following blog posts provide detailed step-by-step walkthroughs on how you can create and publish your HITs for some of the common use cases.

• Editing the Survey Link Project Template in the UI
• Editing Project Templates that Include Images
• Editing the Moderation of an Image Template

Tips & Tricks

We’ve learned a lot about designing HITs over the years – below are some of our favorites.  The posts featuring Worker perspectives are NOT to be missed.

• Helpful Tips for Attracting the Right Workers
• Working Smart: A Worker’s Perspective on Selecting HITs
• Worker Corner: What Makes a Great Requester?

HIT Critiques

Do you want to see examples of what to do and NOT to do in a HIT?  Head over to our HIT Critiques section of the blog and learn from both good design and bad design:

• Mechanical Turk – HIT Critique
• Using Mechanical Turk for Categorization
• HIT Critique – Design Tips for Improving Results

Finally, if you’re hungry for inspirations on what else you can use Mechanical Turk for, check out the Case Study section or the Press section of our blog.

Hosting Your HIT Images Using Amazon S3

In this tutorial we'll walk you through the steps to host your HITs' images using Amazon S3. This tutorial assumes you are already comfortable including images in your HITs but need a place to store them.  If you are not sure how to include images in your HITs please read this blog post first.

About Amazon S3

Amazon S3 is an online cloud storage service from Amazon Web Services (AWS). It lets you store, retrieve, and link to files.

There are two important concepts for S3: Objects and Buckets. Objects are the files you store. They can be any type of file but for this tutorial we'll stick to images. A bucket is a place in S3 you control that holds objects. Think of a bucket like a folder on a computer - except it's on AWS's servers. S3 works with your AWS account and lets you store up to 5 GB of data for free. For more information on data usage, refer to AWS’s free tier guide.

For example, this image of Abbey the dog (abbey.jpg) is stored in an S3 bucket we created named 'mturk-s3-demo'.

The image is located at https://s3.amazonaws.com/mturk-s3-demo/abbey.jpg where "mturk-s3-demo" is the Bucket name, while "abbey.jpg" is the Object name.

Uploading Your Images to S3

There are three high level steps to host your HITs' images on S3: 1) Create an S3 bucket, 2) Upload your images, and 3) Use the image URLs in your HITs. We’ll walk you through the details below.

Step 1 Create an S3 bucket

• Go to the S3 web console.
• Sign in with your Amazon Web Services email address/password.
• Click “Create Bucket”.

• Enter a bucket name and click "Create". This name has to be unique across all S3 accounts. If you receive a bucket name error, use a different name and try again.

Step 2 Upload your images

• Click on the bucket name from the All Buckets list.  Once you are inside the bucket, click "Upload".

• Click "Add Files" and select your files.  Before clicking "Start Upload", click "Set Details".

• From the Set Details page, click "Set Permissions".

• Select "Make everything public" to allow Workers to see your images and then click "Start Upload".

Step 3 Use the image URLs in your HITs

• Select an image by clicking on its checkbox, then open your image's properties to view its URL.

• Copy the link.

• Go to the URL to verify that your image's permission was set correctly. If you get an Access Denied error that means you haven't set the permissions correctly.
• Collect the URLs for all the images you want to include in your HITs and add them to the input CSV file.  If you are using the Requester API, you can reference the images directly by the URLs.

Once the input file is uploaded, preview the HITs in your batch to verify that the images are loaded properly.

Editing the Moderation of an Image Template

Requesters who have images that they need evaluated by Workers against a set of guidelines can use the Moderation of an Image project template. This blog post outlines the steps you can take to customize this project template.

Designing HIT Instructions

The Moderation of an Image template contains sample instructions that you should customize to suit your project.  Use this section to communicate the precise conditions by which Workers should select an image.

We have emphasized in a number of our blog posts the importance of providing clear instruction and the impact it has on the quality of your results.  In the case of moderation of images this is especially true.  Eliminate ambiguity in the moderation guidelines so that there is little to no room for subjective interpretations.  For example, if your HIT asks Workers to evaluate whether human subjects exist in images, provide guidelines for both acceptance and rejection.   This will help Workers understand not only when to select an image, but also when they should not.  For example,

Providing Your Images

The default project template contains two rows of image placeholders for a total of 6 images:

Customize the layout to suit your need using either the Design view (which is the default) or by editing the HTML code from the Source view.  The screen capture below shows the HTML code for the first of the two rows, and the code that defines each image placeholder is highlighted in red:

Make sure you update the src attribute and the value attribute for each instance of your image.  The src attribute specifies the URL of the image to be displayed in your HIT while the value attribute provides a label for the checkbox of the corresponding image.   Provide a meaningful name for each value so that you can tell what image(s) Workers selected when you download your results.

The dimension of each image is specified in the internal CSS via styling of the img tag.  Customize it so that your image display is sized appropriately for Workers to view.

Worker productivity is significantly impacted by the size of the image files that are contained in your HIT, the responsiveness of the content server that serves the images, and the Worker’s own connectivity.  If your image size is large, consider reducing image quality or other compression techniques to reduce file size.  Doing so can reduce the image load time for your HIT significantly.

Publishing

When you publish the HIT, you will provide the input images using a Comma Separated Values (CSV) file.  This CSV file should contain a column header (i.e. the first row) with the names of the variables used in your HIT design (i.e. the value of the src attribute in each image placeholder mentioned above).  The following example illustrates what the CSV file layout looks like for a single HIT with 6 images:

Once you have finished editing the HIT contents, adjust the frame height of your HIT to minimize scrolling for Workers.  You can set the frame height of the HIT from the Design Layout tab by adjusting the height in pixels in the field labeled Frame Height located above the editor.

Testing

Finally, you can test your HIT design before you publish your HITs using our Requester Sandbox.  Making sure the HIT displays the images correctly, load efficiently, and functions correctly in advance will help you save time and avoid paying for extra HITs you can’t use.

Further Customization

We use Bootstrap, a free collection of HTML and CSS-based design templates, to design the format and styling of the Moderation of Images template.  You can use Bootstrap to further customize these templates or build your own. For more information, visit their website.

Editing Project Templates that Include Images

Requesters frequently use Mechanical Turk for image-related tasks, including moderating, tagging, or transcribing text from images. For Requesters using the UI, Mechanical Turk offers project templates that make it easier for Requesters to design and publish HITs for these types of tasks.  In this post, we will highlight some tips for customizing the Tagging of an Image template and the Transcription from an Image template, since the formats of each template are similar.  We’ll focus on the Moderation of an Image project template in a follow-up post.

Designing HIT Instructions

The project templates include sample instructions you can edit to meet your requirements.  Your instructions should communicate your expectations with specific guidance related to how Workers should tag each image or the text they should transcribe.  For instance, if you’re asking Workers to tag an image, you might consider including requirements for the maximum allowable characters or words per tag.  Keep in mind that the clarity of your instructions will directly impact the quality of your results.

Placing Your Image

Both the Tagging of an Image project template and the Transcription from an Image template include a placeholder for a single image followed by three input text fields.   This image placeholder lets you embed your images by using a variable “${image_url}” to represent the location where your images are hosted (More detail on how to embed your image is included in the Publishing section below.) 

You can change the variable name of the image placeholder from “image_url”, if necessary. There are two ways to change the variable name.  The first is to double-click on the image placeholder and edit the URL field.

Alternately, you can edit it by going to the editor’s Source view to change the img src attribute of the image:

Either way, when you change the name of your variable make sure to leave the dollar sign and the curly braces (“${variable_name}”) since this is the required format for specifying an input variable.  Also make sure that your variable name is a single word containing no space.

Adding and Removing Input Fields

You can add or remove input fields based on your project requirements.  To remove an input field, click on the field and press the Del key (if using Firefox) or press CTRL+X (if using Chrome).  Alternatively, you can delete the HTML associated with the input field from the Source view.   In the example below, we’ve highlighted the HTML code you would delete if you wanted to delete the third input field in the Tagging of an Image project template. 

Similarly, if you need an additional input field, you can insert additional fields by copy and pasting the same HTML (highlighted above), and updating the name of the input field attribute with a new, unique name.  For instance, based on the example above, you might name the additional attribute “tag4”.  

Publishing Your HITs

You upload the images to your HIT using a Comma Separated Values (CSV) file.  Your CSV file should contain a column header that matches the variable name referenced in the previous section. The following example illustrates what the CSV file looks like if the variable name used in your HIT is “image_url”:

In this example, the published batch would include 5 HITs, each featuring a different image.

After you’ve finished editing your project template, remember to adjust the frame height of your HIT to minimize Workers’ need to scroll when viewing your HITs.  You can set the frame height of the HIT from the Design Layout tab by adjusting the height in pixels in the field labeled Frame Height located above the editor.

Testing Your HITs

Finally, you can test your HIT design before you publish your HITs using our Requester Sandbox.  Taking the time to test your HITs in advance can save time and help you to avoid paying for extra HITs you can’t use.

Further Customization

We use Bootstrap, a free collection of HTML and CSS-based design templates, to design the format and styling of the Tagging of an Image template and the Transcription from an Image template.  You can use Bootstrap to further customize these templates or build your own. For more information, visit their website 

Editing the Survey Link Project Template in the UI

Requesters frequently conduct surveys with Workers from Mechanical Turk, often using online survey solutions, such as Qualtrics or SurveyMonkey to collect responses. The Survey Link project template in the Mechanical Turk UI makes it easier for Requesters to design and publish HITs that link to surveys hosted on third-party sites.  Today’s post outlines the steps and some tips for getting started.  

Designing HIT Instructions

The Survey Link project template includes sample instructions that you can edit to meet your requirements.  For instance, Requesters often use this section to communicate important details about the survey, such as qualification requirements and instructions for participating, as well as information on how they intend to reward Workers. Keep in mind that the clarity of your instructions will have an impact on the quality of your results.

Inserting Your Survey Link

The project template features a pre-formatted HTML table in which you can insert a link to your survey and an input field in which Workers can enter the code they receive when they complete your survey.  

You will need to replace the survey link with your own link and update the help text in the survey code field so that it matches the formatting of the completion code you expect to receive from Workers.  To update the help text, edit the placeholder attribute of your survey code input field.

Once you’ve finished editing the HIT contents, adjust the frame height of your HIT to minimize scrolling.  You can set the frame height of the HIT from the Design Layout tab by adjusting the height in pixels in the field labeled Frame Height located above the editor.

Testing

Finally, you can test your HIT design before you publish your HITs using our Requester Sandbox.  Making sure the HIT functions correctly in advance will help you save time and avoid paying for extra HITs you can’t use.

Further Customization

We used of Bootstrap, a free collection of HTML and CSS-based design templates, to design the format and styling of the Survey Link project template.  You can use Bootstrap to further customize this template or build your own. For more information, visit their website.

Helpful Tips for Choosing the Right Mechanical Turk Interface

We frequently engage with Requesters to help determine the right approach for integrating with Mechanical Turk. Most commonly, Requesters are deciding between using our user interface (UI) or integrating directly with our application programming interface (API).  In today’s post, we thought it might be helpful to highlight some of the key advantages of each interface.

Advantages of the Requester UI

The Requester UI enables Requesters to design and publish HITs, receive results, and manage Workers from an online interface. For Categorization or Sentiment Analysis work, we offer the Categorization Application and the Sentiment Application, both of which feature predesigned templates and allow you to view results directly from the UI.

For Requesters with limited software development resources or for smaller projects, using the UI makes it easier to get started more quickly.  While a few lines of HMTL may be necessary if you choose to design your own template instead of using the Categorization or Sentiment Application, the technical skills required are minimal.  

Key advantages of using the UI include the ability to:

• Quickly publish work and receive results, without integrating with our API.
• Easily review results and approve or reject assignments.  
• Manage Workers including blocking and granting bonuses.
• Monitor Worker Statistics and create custom Qualifications to target Workers who have performed well for you.

Advantages of the Requester API

The Requester API enables Requesters to publish HITs, collect results, and manage Workers programmatically, without human intervention. Integrating with the API requires software development, but it is the optimal approach to efficiently process high volumes of HITs on a continuous basis. 

Key advantages of using the API include the ability to:

• Integrate results directly into your application.
• Design automated workflows for approving and rejecting assignments using Known Answers via the Review Policies.
• Automate performance tracking with Worker Statistics and managing access to your HITs with Qualifications.

For more detailed information, see our Choose the Right Tool guide on the features and relative advantages of each interface. 

Helpful Tips for Attracting the Right Workers

When Amy Marre realized she could earn a double-digit hourly income transcribing audio on Mechanical Turk, she decided to share her tips with other Workers by writing an e-book about it, titled Cashing in on Transcription, Turk Style.

While Marre’s book is targeted to Workers interested in transcription work, her book provides helpful lessons for Requesters as well.  In today’s post, we’ve highlighted a few of the insights she shared with Workers and how they apply to Requesters, regardless of the type of project:

Communicate the skills required so the right Workers try your HITs. In the book, Marre notes that not all Workers are successful transcribers.  As a Requester, you can help Workers determine if they’re a match for your task by communicating the skills required for your HIT in your instructions, keywords and HIT title.  By clearly stating the skills required for success, you’re more likely to attract the right Workers.

Know your competition. Marre points out that Workers have a choice of HITs in the marketplace and are frequently assessing which HITs enable them to earn the most money.  To make sure that your HITs are competitive, explore other HITs available in the marketplace, especially those that require similar skills. Requesters with user-friendly task interfaces and competitive compensation tend to attract more Workers and have the higher Worker retention.

Set Worker expectations.  Marre also provides Workers tips on how to earn and keep the Qualifications required for some transcription work.  As a Requester, you can communicate your expectations for quality in the HIT instructions.  If you’re using Known Answers to score Workers, make sure the Known Answers you’re comparing Workers answers against are unambiguous to avoid rejecting assignments or revoking Qualifications unfairly. 

Provide tips to make Workers more efficient.  Marre includes tips to help Workers improve productivity on transcription HITs.  Generally speaking, Workers’ productivity on a HIT increases over time as they become more familiar with the task.  Requesters can help Workers become more efficient (and earn more money) by sharing tips for improving productivity in the HIT instructions.

We hope that these insights are helpful.  To read Marre’s book for yourself, you can find the Kindle edition on Amazon.com.

To learn more about best practices for Requesters, see our Requester Best Practices Guide.  

Canada’s National Research Council Builds Huge Word-Emotion Lexicon using Mechanical Turk

Recently, MIT Technology Review announced the release of a new word-emotion and word-polarity lexicon developed by Saif Mohammad and Peter Turney from Canada’s National Research Council (NRC). The lexicon includes 10,000 words and their associated emotions and polarity, and according to the researchers, was built cost-effectively in the span of a few weeks using Mechanical Turk.

Companies are increasingly using software to mine data from social sites, like Twitter and Facebook, to understand public opinion toward brands, products and services. However, the accuracy of these applications is limited by the software’s ability to interpret the emotion associated with each word and most existing technologies rely on relatively limited or small emotion lexicons. 

We reached out to the Mohammad and Turney to learn more about their methods for building their lexicon using Mechanical Turk. Here are a few kernels of advice that they had to share with other Requesters based on their experience:

• Design and implement an effective quality control strategy.  Key to Mohammad and Turney’s quality control was their use of Known Answers.  In each HIT, the researchers primed Workers by presenting a word choice question to assess each Worker’s familiarity with the word to which they would later be asked to associate an emotion.  According to Mohammad, “Prompting Workers to consider the word’s meaning at the beginning of the task put them in a better mindset to answer the following emotion association question and led to higher quality annotations.”

• Simplify HITs by breaking workflows into simple steps.  To increase Workers’ speed, while also improving the quality of their outputs, the researchers recommend breaking complex questions into simpler “bite-size” questions.  According to Mohammad, “If it is easy for Workers to quickly comprehend questions and provide answers, the quality of Workers’ answers will be higher and more Workers will be more interested in contributing to your HITs.”

• Present clear and concise instructions.  Early in their HIT design process, Mohammad and Turney tested variations on the wording of their instructions. For instance, in one test they asked Workers if a word “evokes” a specific emotion; whereas, in a second test they asked if a word “is associated” with a specific emotion. The test revealed that agreement between Workers was more frequent when the instructions included the term “is associated”. Mohammad explained, “The experiment confirmed our intuition that 'evokes' may encourage subjective answers whereas 'is associated' guides Turkers to more objective answers.”

• Offer appropriate compensation to target the right Workers.  Setting a competitive HIT price is critical to attracting the right Workers.  However, according to Mohammad, “Paying more does not yield higher quality results, but paying too little may discourage the right Workers from your HITs.”

For Mohammad and Turney, the tactics above significantly impacted the quality of their results and the speed with which they were able to build their lexicon.  By optimizing instructions, workflow, and pricing, they found they not only attracted the best Workers for their task, but kept them engaged in their HIT.  Further, by utilizing scalable quality control tools, such as Known Answers, they were able to reliably infer the accuracy of the annotations collected as their project progressed.

To learn more about their project, see Mohommad and Turney’s research paper available here.

How SnipSnap Built and Scaled Mobile Couponing with People

When Ted Mann founded SnipSnap, the popular mobile couponing application, he didn’t anticipate the challenge his team would encounter converting printed coupons into a digital, mobile-ready offers using automated solutions, like optical character recognition. In a recent post on Fast Company’s blog, Co.LABs, Mann discusses how SnipSnap addressed the challenge by integrating with Mechanical Turk to develop a scalable data extraction workflow able to process more than 200,000 coupons per day.

Here are some of the kernels of wisdom Mann had to offer for new Requesters based on his own experience implementing Mechanical Turk:

• Implement effective quality control workflows.  Mann specifically recommends using tools like Known Answers to audit Workers in real-time and Plurality to measure consensus between Workers.

• Streamline Workers’ user experience when designing your task to make it as simple and speedy as possible. For instance, consider adding auto-completes to fields, making it easy to tab between fields, and minimizing scrolling.

• Maintain consistent HIT pricing, if possible. Use caution when modifying HIT prices.  Changing prices on HITs (especially reducing the price of HITs) can negatively impact your relationship with Workers.

• Spend an hour doing your own tasks. This will give you a sense of whether or not you’re tasks are appropriately priced and help identify opportunities to streamline your Worker experience. 

By the way, in case this use case sounds familiar to you, we recently featured a guest blog post from the team at Upside Commerce – another mobile app developer using Mechanical Turk to extract data from print coupons for mobile delivery.  You can read more about Upside’s story here.