Cleaning Up

While developing AutoMan programs, it is not uncommon to create a large number of orphaned jobs in the MTurk sandbox. While AutoMan normally cleans up after itself, if your program crashes or if you terminate it with a SIGINT signal (Ctrl-C), AutoMan will leave state behind on MTurk. AutoMan state consists of three kinds of runtime objects:

• HITs, which are the MTurk equivalent of a human function call,

• Assignments, which are the equivalent of a human function return value, and

• Qualifications, which are a runtime data structure that limits who can participate in a given crowdsourcing job.

Precisely how the AutoMan language utilizes these MTurk data structures is described in our 2012 OOPSLA paper.

There is little harm in leaving orphaned state behind, but it is often helpful to remove it for debugging purposes. Or, if you're like me, because you just want to be tidy.

To obtain these tools, you will need to checkout AutoMan from source.

git clone https://github.com/automan-lang/AutoMan.git

Once you have a copy of the source, look in the tools directory.

These tools are also useful if you, like me, occasionally botch a live job and want to cancel it immediately. Just be aware--it is very bad practice to stiff workers, so these tools will automatically approve work and pay workers, whether their work is good or not.

Installing Prerequisites

These tools are written in Java and require that you install the Apache Maven tool. On the Mac, Maven is available via Homebrew.

Access Keys

These tools require that you store your access keys in a Java .properties file. I typically call my file mturk.properties and keep it someplace safe.

Your mturk.properties file should be a text file formatted using the following convention:

access_key=<your access key here>
secret_key=<your secret key here>

For example, here is a sample mturk.properties file (with bogus keys):

access_key=AKIA5KJ00DF3HJKJ2B9N
secret_key=qGCCxrnAUlvW12KRuuCo5i5m80ptEVclf7kjAbsK

Remember: never post your MTurk access keys to a public site, such as your GitHub repository. If you do so, remove your keys from your MTurk account using AWS IAM immediately.

Removing orphaned HITs and Assignments

The DeleteAllHITs tool, found in AutoMan's tools directory, will delete all the HITs for your account, either in the MTurk sandbox or in the live production site. You can see a help message by running the program without arguments.

$ ./run.sh
Usage:
  You should use the "run.sh" shell script.

  ./run.sh <path to mturk.properties file> <sandbox mode true/false>

  For example:
    /run.sh ~/mturk.properties false

For example,

$ ./run.sh mturk.properties true

will delete all HITs and Assignments on the MTurk sandbox.

By changing true to false, you can also delete all HITs and Assignments on the live, production MTurk site.

Be aware that this script will pay workers for any completed work on the production MTurk site before deleting assignments and HITs.

Removing all Qualifications

The DeleteAllQualifications tool will delete all Qualification objects from MTurk. As with the DeleteAllHITs tool, it can be run on either the sandbox or the live production site.

$ ./run.sh 
Usage:
  You should use the "run.sh" shell script.

  ./run.sh <path to mturk.properties file> <sandbox mode true/false>

  For example:
    /run.sh ~/mturk.properties false

For example,

$ ./run.sh mturk.properties true

will delete all Qualification objects on the MTurk sandbox.

By changing true to false, you can also delete all Qualification objects on the live, production MTurk site.

Before running this tool, be sure that you have no active AutoMan programs running. AutoMan uses Qualification objects internally and expects that the ones it creates remain on the site until it deletes them. Deleting a Qualification for a running job will likely result in a program crash.