Some thoughts on invalid HITs

There has been some discussion about the quality of work in the Mechanical Turk marketplace so we wanted to provide some information on our Requester Participation Agreement (PA) and how we go about ensuring high quality work in the system. The Amazon Mechanical Turk PA outlines our terms of service for Requesters including the type of work considered invalid. When we find invalid work in the system, we work quickly to remove those HITs.   

Some examples of invalid HITs include requests that:  

• Require disclosure of the Worker's identity or e-mail address, either directly or indirectly
• Require registration at another website or group
• Directly or indirectly promote a site, service, or opinion 
• Violate the terms and conditions of an activity or website (for instance asking Workers to vote for something)
• Have explicit or offensive content, for instance, nudity, but do not have the Adult Content Qualification
• Ask Workers to solicit third parties 
• Generate "referred" site visits or click through traffic
• Ask workers to take action to manipulate a search engine's relationship data
• Violate copyrights  

Reporting Invalid HITs
In addition to the measures we have in place to ensure quality work in the system, anybody using Amazon Mechanical Turk can also provide accurate, timely feedback on the HITs in the system. If you are a Worker or Requester and you identify a HIT that you think violates our PA, please report it and we will take appropriate action.   To report an invalid HIT, select the “violates the Amazon Mechanical Turk policies” link at the bottom right of the HIT. 

We monitor our marketplace closely and want you to know that we take quality very seriously and appreciate the time users take to report invalid work.

-- The Mechanical Turk Team 

Comments

[shannon]

I am finding that that the reporting HITS that violate the PA of your Site they end up being taken off but put right back up as soon as the workers are reporting this information to you guys. These Requesters are continually renewing and coming back again and again into the system. I am wondering how this is being closely monitored.

[anon]

Amazon, you do exactly zero pro-active work to eliminate bad requesters. Evidence that you do nothing to remove scammers unless requested can easily be seen by sorting all HITS by reward amount high to low. All of the highest paid HITs are scams and are quite easy to find and you have not removed them.

(And that doesn't even include the time you let someone solicit tales of child abuse I though were solicitation for child pornography.)

[Sharon]

@shannon - it is definitely a tough game managing aggressive invalid HIT requesters. We unfortunately can't speak specifically about our internal methodologies, but we do monitor this activity at a very granular level.
@anon We've made a few modifications to improve that sort specifically, have you noticed an improvement?

[spamgirl]

Easy answer to managing aggressive invalid HIT requesters - change policy to require a $100 payment upfront which is not refundable. Tack on increasing ID checks for requesters (driver's license, phone check, credit card number on file, etc.) and you'd be good to go.

[Broken turk]

Did you change something in the past few days, since Panos Ipeirotis published his review of spam on mTurk? Can you explain how that change was implemented, or rather, why it took so long to do something that changed so quickly?

[J. Yendriga]

I wouldn't claim success for getting rid of the spam HITS quite yet.
The is still being posted:

Join this free website and report the feedback

This is a very short sign up job for you to take. It is a very fast job that can be done in under 1 minute! You may only take this HIT once.

* Go to our website here
* Complete the sign up process of the first 4 pages with your real information.
* Once you receive the thank you page you are finished. If this has gone wrong you need to try again.
As part of the survey it is possible you will be contacted. Please answer the person as normal as possible as he/she is also doing her job to follow up your online request.
Do not reveal you take this as a job but ask for more information, or tell that you have changed your mind.

And many more from the requester mrutknews on this December 26 morning. Also, like before reporting them does not get them removed

[Jimfribly]

I have noticed a TINY difference.
It really is difficult when you continually see 7 or 8 hits that you have already reported still on screen hours/even days later.
You know a newbie is going to fall for it.
So many are CLEARLY a violation of TOS, that even a turker who has only been working on the site a week can see.
Oh well, thanks for trying Amazon, at least we know someone is trying to bring MTurk back to what it used to be before the spammers found it.

[Sharon]

Hey everyone -- just wanted to let you know that we are listening to your feedback -- and we really appreciate it.