The Mechanical Turk Blog

Introducing “Cooking with Sharon” & Tip #1 Ask a good question

This is the first post in a new series of blog posts called “Cooking with Sharon”.  The series  will identify tips, tricks and best practices for getting quality results from Mechanical Turk. 

In my conversations with Requesters, I am often asked “what are the tricks for getting great quality from Mechanical Turk?” The good news is there are many ways.  But that’s sort of like my mother-in-law giving me her “recipe” for her delicious marinara sauce in which all ingredients are added “to taste”.

Fortunately there is a formula for getting great quality from Mechanical Turk.  The first step to good quality is to ask a good question.  This may seem obvious, and you’re probably thinking “my question is fine! That’s not the issue I’m having.”   Here’s an example that changed my mind: Is a Miata a Sports Car?    

I like to ask this question when I speak about Mechanical Turk.  Inevitably half the audience says yes and the other half says no.  (And yes it’s usually divided on gender lines.)  In order for a Worker to answer this question accurately (and with a high probability that multiple Workers would agree on the same answer) they need to know what you mean by a “Miata” and a “Sports Car”.  

It’s easy enough to look up the term Miata on the internet so even if the Worker is in geography where the Miata isn’t sold (India), or where they are sold under a different name (Japan or Europe) it’s likely the Worker will be able to find out what a Miata is.  It would be more helpful if you provided the specific model and the specific year as not all Miatas are created equal.

Unless you want a Worker’s “opinion”, you need to define what makes a car a “Sports Car”.  Do you mean does a Miata accelerate from 0mph to 60 in under 5 seconds? Or do you want to know if a Miata meets the Merriam-Webster definition (“a low small usually 2-passenger automobile designed for quick response, easy maneuverability, and high-speed driving”)?  

The first step to getting good results from Mechanical Turk is to make sure your question is clear.  So if you’re asking Workers “Is this image offensive” you shouldn’t be surprised that if Workers answers vary greater.  If instead you ask “Does this image contain nudity”, you’re more likely to get more accurate answers.  

 

-- Sharon 

Sharon Chiarella is a Vice President at Amazon, responsible for running Amazon Mechanical Turk.  And for the record, she can’t cook. 

 

Register now for our Sentiment Analysis Webinar with Freedom OSS on April 26th!

 
Are you trying to analyze the sentiment of content such as Tweets or Blog Posts? Trying to figure out how to integrate crowdsourcing into your sentiment analysis? Do you need to have content analyzed quickly, with a high level of accuracy, and at a low cost?

Join us for an informative session on Amazon Mechanical Turk and learn how your company can leverage the human crowd for human sentiment analysis of content such as tweets, articles, RSS feeds and blog posts. This session will dig into the details of getting started and provide information on how to be successful so you get accurate results.

Additionally, Freedom OSS will share their experiences designing and managing sentiment tasks and will also demo their CrowdControl crowdsourcing platform that is built on top of the Mechanical Turk platform. We will share a presentation and then be open for Q & A.

Here are the details:

Mechanical Turk Best Practices for Sentiment Analysis - Webinar
Tuesday, April 26^th at 11:00 AM PST.

 

 

Overview: Lifecycle of a HIT

One of the questions I've been hearing from Requesters is about the lifecycle of a Human Intelligence Task (HIT) — specifically, how does a HIT progress from creation to disposal? In this post, I’m going to outline the basics of the HIT lifecycle.

There are 5 states Requesters will find their HITs in:

• Assignable – a HIT can be accepted by a Worker
• Unassignable – a HIT has been accepted by a Worker and is being worked on and therefore cannot be accepted by another Worker
• Reviewable – a Worker has submitted answers to a HIT and the HIT is available for review
• Reviewing (optional state) – the HIT is currently being reviewed by the Requester
• Disposed – the HIT has been deleted and can no longer be retrieved.

 

Once HITs are created, they are in an Assignable State
Every HIT is created in the Assignable State. In the Requester User Interface (RUI), you design your HIT template in the UI, and upload a file of your data to populate your HITs.  We do a “mail merge” of the template and the data for the HITs. When you click “Publish”, HITs are created and become available to Workers. If you use the API, you call CreateHit to create or publish HITs in the system (Users of the Command line tools – aka CLT – use LoadHIT for this). HITs are then created and become available to Workers in the Mechanical Turk marketplace. At this point, HITs are in the Assignable state.

 

HITs typically go from Assignable to Unassignable
HITs become Unassignable when either the last assignment for that HIT is accepted by a Worker or when the HIT expires. HITs expire when the time allotted for the HIT (which you define) passes or when you manually expire a HIT before its default expiration date either through the Requester UI by cancelling a batch or by calling ForceExpireHIT via the API (CLT users would use deleteHits with the ‘-expire’ option) .

Once Workers have submitted all the assignments for a HIT, the HIT will go into Reviewable state.

A HIT will return to the Assignable state if a Worker returns a HIT assignment or if you add assignments to the HIT.  This allows other Workers to accept it. 

 

Once Work is complete, HITs become Reviewable
As Workers submit their work, HITs continue to make progress towards being Reviewable.  An assignment is Reviewable once a Worker submits it and a Reviewable Assignment can be approved or rejected.  But a HIT doesn’t move to the Reviewable state until all HIT assignments are submitted by Workers, or the HIT expires. The HIT remains in Reviewable state until it is Disposed.

If you add assignments to a HIT or extend it by changing the expiration date this will change the HIT state from being Reviewable back to Assignable.

 

Optional Reviewing State
Some Requesters who use the API choose to change a HIT’s state to Reviewing by calling SetHITAsReviewing while they determine whether to approve or reject the assignment.  HITs in the Reviewing state will not be returned when a Requester uses the GetReviewable HITs API call (making it useful for checking new reviewable HITs).  

 

Disposing HITs
Once all Assignments for a Reviewable HIT are either approved or rejected, the HIT can be put into Disposed state – which is a terminal state for a HIT.  API Users call DisposeHIT to accomplish this, while CLT users call deleteHIT. Once HITs are disposed, they can no longer be accessed from Mechanical Turk.  HITs are automatically disposed when they have been inactive for 120 days.  

We recommend API and CLT users dispose HITs often because as the number of undisposed HITs increases, you will have more (inactive) data to sift through to get the active data you are looking for when you make calls such as GetReviewableHITs. 

 

The graphic below captures the various states we’ve discussed, and transitions between them.

-- Navdip
Senior Development Manager, Amazon Mechanical Turk

There are 5 states Requesters will find their HITs in :

---