One of the questions I've been hearing from Requesters is about the lifecycle of a Human Intelligence Task (HIT) — specifically, how does a HIT progress from creation to disposal? In this post, I’m going to outline the basics of the HIT lifecycle.
There are 5 states Requesters will find their HITs in:
- Assignable – a HIT can be accepted by a Worker
- Unassignable – a HIT has been accepted by a Worker and is being worked on and therefore cannot be accepted by another Worker
- Reviewable – a Worker has submitted answers to a HIT and the HIT is available for review
- Reviewing (optional state) – the HIT is currently being reviewed by the Requester
- Disposed – the HIT has been deleted and can no longer be retrieved.
Once HITs are created, they are in an Assignable State
Every HIT is created in the Assignable State. In the Requester User Interface (RUI), you design your HIT template in the UI, and upload a file of your data to populate your HITs. We do a “mail merge” of the template and the data for the HITs. When you click “Publish”, HITs are created and become available to Workers. If you use the API, you call CreateHit to create or publish HITs in the system (Users of the Command line tools – aka CLT – use LoadHIT for this). HITs are then created and become available to Workers in the Mechanical Turk marketplace. At this point, HITs are in the Assignable state.
HITs typically go from Assignable to Unassignable
HITs become Unassignable when either the last assignment for that HIT is accepted by a Worker or when the HIT expires. HITs expire when the time allotted for the HIT (which you define) passes or when you manually expire a HIT before its default expiration date either through the Requester UI by cancelling a batch or by calling ForceExpireHIT via the API (CLT users would use deleteHits with the ‘-expire’ option) .
Once Workers have submitted all the assignments for a HIT, the HIT will go into Reviewable state.
A HIT will return to the Assignable state if a Worker returns a HIT assignment or if you add assignments to the HIT. This allows other Workers to accept it.
Once Work is complete, HITs become Reviewable
As Workers submit their work, HITs continue to make progress towards being Reviewable. An assignment is Reviewable once a Worker submits it and a Reviewable Assignment can be approved or rejected. But a HIT doesn’t move to the Reviewable state until all HIT assignments are submitted by Workers, or the HIT expires. The HIT remains in Reviewable state until it is Disposed.
If you add assignments to a HIT or extend it by changing the expiration date this will change the HIT state from being Reviewable back to Assignable.
Optional Reviewing State
Some Requesters who use the API choose to change a HIT’s state to Reviewing by calling SetHITAsReviewing while they determine whether to approve or reject the assignment. HITs in the Reviewing state will not be returned when a Requester uses the GetReviewable HITs API call (making it useful for checking new reviewable HITs).
Disposing HITs
Once all Assignments for a Reviewable HIT are either approved or rejected, the HIT can be put into Disposed state – which is a terminal state for a HIT. API Users call DisposeHIT to accomplish this, while CLT users call deleteHIT. Once HITs are disposed, they can no longer be accessed from Mechanical Turk. HITs are automatically disposed when they have been inactive for 120 days.
We recommend API and CLT users dispose HITs often because as the number of undisposed HITs increases, you will have more (inactive) data to sift through to get the active data you are looking for when you make calls such as GetReviewableHITs.
The graphic below captures the various states we’ve discussed, and transitions between them.
-- Navdip
Senior Development Manager, Amazon Mechanical Turk
Comments