Cleaning Up
While developing AutoMan programs, it is not uncommon to create a large number of orphaned jobs in the MTurk sandbox. While AutoMan normally cleans up after itself, if your program crashes or if you terminate it with a SIGINT signal (Ctrl-C), AutoMan will leave state behind on MTurk. AutoMan state consists of three kinds of runtime objects:
HITs, which are the MTurk equivalent of a human function call,Assignments, which are the equivalent of a human function return value, andQualifications, which are a runtime data structure that limits who can participate in a given crowdsourcing job.
Precisely how the AutoMan language utilizes these MTurk data structures is described in our 2012 OOPSLA paper.
There is little harm in leaving orphaned state behind, but it is often helpful to remove it for debugging purposes. Or, if you're like me, because you just want to be tidy.
To obtain these tools, you will need to checkout AutoMan from source.
git clone https://github.com/automan-lang/AutoMan.gitOnce you have a copy of the source, look in the tools directory.
These tools are also useful if you, like me, occasionally botch a live job and want to cancel it immediately. Just be aware--it is very bad practice to stiff workers, so these tools will automatically approve work and pay workers, whether their work is good or not.
Installing Prerequisites
These tools are written in Java and require that you install the Apache Maven tool. On the Mac, Maven is available via Homebrew.
Access Keys
These tools require that you store your access keys in a Java .properties file. I typically call my file mturk.properties and keep it someplace safe.
Your mturk.properties file should be a text file formatted using the following convention:
For example, here is a sample mturk.properties file (with bogus keys):
Remember: never post your MTurk access keys to a public site, such as your GitHub repository. If you do so, remove your keys from your MTurk account using AWS IAM immediately.
Removing orphaned HITs and Assignments
The DeleteAllHITs tool, found in AutoMan's tools directory, will delete all the HITs for your account, either in the MTurk sandbox or in the live production site. You can see a help message by running the program without arguments.
For example,
will delete all HITs and Assignments on the MTurk sandbox.
By changing true to false, you can also delete all HITs and Assignments on the live, production MTurk site.
Be aware that this script will pay workers for any completed work on the production MTurk site before deleting assignments and HITs.
Removing all Qualifications
The DeleteAllQualifications tool will delete all Qualification objects from MTurk. As with the DeleteAllHITs tool, it can be run on either the sandbox or the live production site.
For example,
will delete all Qualification objects on the MTurk sandbox.
By changing true to false, you can also delete all Qualification objects on the live, production MTurk site.
Before running this tool, be sure that you have no active AutoMan programs running. AutoMan uses Qualification objects internally and expects that the ones it creates remain on the site until it deletes them. Deleting a Qualification for a running job will likely result in a program crash.
Last updated