# Introduction

AutoMan: A Domain-Specific Language for Crowdsourcing Tasks

## What is AutoMan? <a href="#overview" id="overview"></a>

AutoMan is the first fully automatic **crowdprogramming** system. With AutoMan, you declaratively define **human functions** and use them just as you would ordinary functions. Focus on your application logic instead of MTurk code.

AutoMan is currently available as a library for [Scala](https://www.scala-lang.org/).

## Example

```scala
  def which_one() = radio (
    budget = 5.00,
    text = "Which one of these does not belong?",
    options = (
      choice('oscar, "Oscar the Grouch", "https://tinyurl.com/y2nf2h76"),
      choice('kermit, "Kermit the Frog", "https://tinyurl.com/yxh2emmr"),
      choice('spongebob, "Spongebob Squarepants", "https://tinyurl.com/y3uv2oew"),
      choice('cookiemonster, "Cookie Monster", "https://tinyurl.com/y68x9zvx"),
      choice('thecount, "The Count", "https://tinyurl.com/y6na5a8a")
    )
  )

```

This function produces an MTurk task that looks like this:

![A "radio button" question with 5 options, including images.](https://4139267154-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MKQ7zHwDtbhybf6LYpN%2F-MKQBQEwIsL-VjE_nHV2%2F-MKQCudxlPj-svLUBlSw%2Fspongebob.png?alt=media\&token=78fb2a38-e12b-4c6d-96e2-99c9ff06e733)

The function can be called like any other function in Scala:

```scala
which_one()
```

Notice in the above declaration and function call, there was no need to specify task wages, the number of workers, how to handle network errors or other system failures, or how to determine whether answers are good.  AutoMan *automatically* handles pricing, quality control, and task management.

## Learn More

To get started, check out our **Getting Started** guide.  See the navbar to the left, or click on the link below.

## Acknowledgements <a href="#acknowledgements" id="acknowledgements"></a>

This material is based on work supported by National Science Foundation Grant Nos. CCF-1144520 and CCF-0953754 and DARPA Award N10AP2026. Microsoft Research also generously supported research and development by funding experiments on Mechanical Turk.


# Installing Prerequisites

Before you start, you must have a few things installed.

1. [Download and install Java](https://www.oracle.com/java/technologies/javase-jdk15-downloads.html) (version 11+ recommended, but 1.8+ should work).
2. [Download and install Scala](https://www.scala-lang.org/download/) (specifically, version 2.12; Scala 2.13 is not yet supported).\
   Note that on the Mac, [the easiest way to install Scala is by using Homebrew](https://formulae.brew.sh/formula/scala@2.12).
3. [Download and install `sbt`](https://www.scala-sbt.org/download.html)†.

   Note that on the Mac, [the easiest way to install `sbt` is by using Homebrew](https://formulae.brew.sh/formula/sbt).

#### Download AutoMan

The easiest way to install AutoMan is to manage your application's build using `sbt` by creating a `build.sbt` file.  This tutorial will explain how to do that, step-by-step.  But if you already know how to use `sbt`, just add the following dependency to your `libraryDependencies` section.

```
"org.automanlang" %% "automan" % "1.4.2"
```

`sbt` will download an AutoMan JAR and store it locally on your machine, along with all of AutoMan's dependencies.

† [Pronounced like this](https://www.youtube.com/watch?v=pQlPjUSj7no). The `sbt` authors [insist "sbt" does not stand for "Scala build tool"](https://www.scala-sbt.org/1.x/docs/Faq.html), so... fair game.


# Tutorial: Obtain Mechanical Turk Credentials

To use AutoMan, you will need an Amazon Mechanical Turk account.  If you already have one, and you know your secret and access keys, you may skip ahead to [Tutorial: Create an AutoMan Project](/getting-started/creating-a-new-project).

1. Go to <https://www.mturk.com/>.
2. Click on the button "Get Started with Amazon Mechanical Turk."
3. Click on the button "Create a Requester Account."
4. Follow the instructions for registering an account.
5. Once you have an MTurk account, you will need to go to the page [Getting Started as an Amazon Mechanical Turk Developer](https://requester.mturk.com/developer).  Follow the steps on that page up to and including Step 3.  **Once you link your AWS account to your MTurk account, it is important to save a copy of your AWS access keys somewhere PRIVATE** :lock\_with\_ink\_pen:**.**\
   To reiterate, you do not need to perform steps 4 ("Download an AWS Software Development Kit") or 5 ("Make Your First API Call").

{% hint style="warning" %}
As of the time of this writing, step 3 above leads to a "Looking for something?" error page on MTurk.  I have reported the broken link to Amazon, but in the meantime, look at the top of the error page for the [Create an Account](https://requester.mturk.com/begin_signin) link (or just click on my link).
{% endhint %}

{% hint style="info" %}
You will need your MTurk "access key" and "secret access key" to use AutoMan; I call these "MTurk credentials" from here on.  Note that, for security reasons, if you lose your MTurk credentials, you cannot retrieve them from Amazon a second time.  Instead you will need to regenerate them.

Your keys should look something like:

```
access key: AKIA5KJ00DF3H&KJ2B9N
secret key: qGCCxrnAUlvW12KRuuCo5i5m80ptEVclf7kjAbsK
```

{% endhint %}

{% hint style="danger" %}
***You are strongly cautioned not to store your MTurk credentials (or any other password-like information) on any publically-accessible site, especially GitHub.***  The author of this article personally knows someone who accidentally stored their passwords in GitHub, and later discovered that cyber-criminals had used that information to drain money out of their account!  Be careful!
{% endhint %}


# Tutorial: Create an AutoMan Project

Once you have Java, Scala, and `sbt` installed (if you don't go back to [Installing Prerequisites](/getting-started/installation-tutorial)), you can create an AutoMan project.  Note that these instructions assume that you are familiar with the command line on a UNIX machine.  You can run AutoMan in other operating systems, like Windows, but you will need to adapt the instructions below.

## Create a project folder

We're going to start by creating a new directory for our project.  This will create a new directory in our current location.

```bash
$ mkdir my-first-automan-project
```

Now go ahead and `cd` into that directory.

```bash
$ cd my-first-automan-project
```

In any `sbt` project, there are two primary components:

* `build.sbt`: Is a build specification, where you will configure your library dependencies, among other things.
* `src`: Is a directory where you will store your application code.

## build.sbt

Let's start by creating a `build.sbt` file.  I am going to do this with `emacs build.sbt` but you can use any text editor with which you are comfortable.  Paste the following into your `build.sbt` file:

```scala
scalaVersion := "2.12.12"
name := "my-first-automan-app"
version := "1.0"
libraryDependencies += "org.automanlang" %% "automan" % "1.4.2"
```

Most of this should be self-explanatory, but I will explain anyway.  Note that you must use the `:=` assignment operator and not `=` in `build.sbt` files.

* `scalaVersion`: The version of Scala you want to use.  I have specified the latest version of Scala 2.12 at the time of this writing. If you don't have this precise version of Scala on your machine, don't worry,`sbt` will install it for you.  Note that AutoMan does not currently support Scala 2.13.
* `name`: The name of your application.
* `version`: The version of your application.  This won't be all that important for this example, but if you ever decide to publish your app, this number will be used as a part of your app's metadata.
* `libraryDependencies`: This is where we specify that our app depends on one or more third-party libraries.  I have pre-filled this entry with the dependency information for the latest version of AutoMan.  `sbt` will automatically download and install whatever dependencies you specify.  Observe that we use `+=` for `libraryDependencies`; this is because `libraryDependencies` is a list.  `+=` adds a dependency to that list.

{% hint style="info" %}
You can find more libraries at [MVNrepository](https://mvnrepository.com/).  For example, [here is the MVNrepository page for AutoMan](https://mvnrepository.com/artifact/org.automanlang/automan_2.12/1.4.2).  Note the "SBT" tab which includes the line we pasted into our `build.sbt` file above.  You can include *any* third-party library you find in MVNrepository, with some small caveats, most notably that some libraries are tied to specific versions of Scala.  Just add another `libraryDependencies` line using `+=`.
{% endhint %}

## src/main/scala and source files

Let's now create our `src` directory.  `sbt` saves you configuration work by being rather picky about where you store your source code.  Let's create that location:

```
$ mkdir -p src/main/scala
```

&#x20;And let's create a new Scala file in that folder (again, use your favorite text editor):

```
$ emacs src/main/scala/MyFirstAutoManApp.scala
```

Once you have a new, blank `MyFirstAutoManApp.scala` file loaded up in your editor, we can start creating our app.


# Tutorial: Write and Run Your First App

An AutoMan app always has at least four parts.

## Part 1. Import AutoMan and Create Main Class

As is typical in most programming languages, we have to tell Scala that we want to use AutoMan.  Paste the following `import` statement into the top of your source file:

```scala
import org.automanlang.adapters.mturk.DSL._
```

Specifically, we're telling Scala that we want to use AutoMan to manage Amazon Mechanical Turk (`mturk`) jobs using its MTurk domain-specific language (DSL).

If you have not yet created an account for Amazon Mechanical Turk, [now would be a good time](/getting-started/tutorial-obtain-mechanical-turk-credentials).

We also need to create the main class for our application.  Scala makes this a little easier than Java.  Paste the following into your editor below your `import` statement.

```scala
object MyFirstAutoManApp extends App {
  println("Hello world!")
}
```

{% hint style="info" %}
An `object` in Scala is just a special kind of class that does not need to be instantiated.  It is also sometimes referred to as a *singleton* class, since you can only have one of them.
{% endhint %}

{% hint style="info" %}
`extends App` tells Scala to augment the `MyFirstAutoManApp` with a `public` and `static` `main` method that takes a `String[]`.  Any code you write inside the class will be interpreted as being a `main` method implementation.  Arguments are available via a variable called `args`.  If you come from Java, this is a lightweight version of the `public static void main(String[] args)` which you have probably used many times.
{% endhint %}

The code above is a complete Scala hello-world app.  As a little sanity check, before we dig into AutoMan, let's just try it out.  Run the following in your terminal:

```bash
$ sbt run
```

{% hint style="info" %}
The first time you run `sbt`, it will likely download large numbers of libraries, printing huge amounts of diagnostic information along the way.  Do not be alarmed.  Subsequent runs should produce substantially less output.
{% endhint %}

You should see output that looks a bit like this:

```
[info] welcome to sbt 1.3.13 (N/A Java 12.0.2)
[info] loading project definition from /home/dbarowy/my-first-automan-app/project
[info] loading settings for project my-first-automan-app from build.sbt ...
[info] set current project to my-first-automan-app (in build file:/home/dbarowy/my-first-automan-app/)
[info] Compiling 1 Scala source to /home/dbarowy/my-first-automan-app/target/scala-2.12/classes ...
[info] running SimpleRadioProgram 
Hello world!
[success] Total time: 4 s, completed Oct 24, 2020, 3:34:12 PM
```

Notice, buried in all that output, that your program printed out `Hello world!`.  If you see `Hello world!` in your output, move on to the next step.  If not, read the `sbt` output carefully to diagnose and fix the problem.

{% hint style="info" %}
[Stack Overflow](https://stackoverflow.com/) is a wonderful resource for Scala and SBT questions.
{% endhint %}

## Part 2. Initialize the AutoMan Platform Adapter

Paste the following into your `MyFirstAutoManApp` class.

```scala
implicit val a = mturk (
    access_key_id = args(0),
    secret_access_key = args(1),
    sandbox_mode = true
)
```

The code above defines a variable `a` that stores an instance of `mturk`.  AutoMan needs what we call a *platform adapter* in order to know which service to connect to.  In this case, we are connecting to Mechanical Turk.

When you run this code, you will need to supply your MTurk access key and secret key on the command line.  `args` is an argument array, and `args(0)` is the first element of that array (if you come from Java, note that arrays in Scala use `()` instead of `[]`).  These are the same credentials you downloaded [in an earlier step of this tutorial](/getting-started/tutorial-obtain-mechanical-turk-credentials).

{% hint style="danger" %}
Never embed your `access_key_id` or your `secret_access_key` in your source code!  Doing so makes it easy to accidentally push your code to a public site like GitHub where they can be stolen and abused.
{% endhint %}

{% hint style="info" %}
Mechanical Turk comes equipped with a *sandbox mode*.  The sandbox lets you run test code without worrying about real MTurk workers or real money.  When `sandbox_mode` is set `true` in the `mturk` adapter, you will run in sandbox mode.
{% endhint %}

{% hint style="warning" %}
Be sure that you really want to run your program before setting `sandbox_mode` to `false`. It will run real jobs and it will spend real money!
{% endhint %}

## Part 3. Define a Human Function

Paste the following code below your platform adapter code.

```scala
def which_one() = radio (
  budget = 5.00,
  text = "Which one of these does not belong?",
  options = (
    choice('oscar, "Oscar the Grouch", "http://tinyurl.com/qfwlx56"),
    choice('kermit, "Kermit the Frog", "http://tinyurl.com/nuwyz3u"),
    choice('spongebob, "Spongebob Squarepants", "http://tinyurl.com/oj6wzx6"),
    choice('cookiemonster, "Cookie Monster", "http://tinyurl.com/otb6thl"),
    choice('thecount, "The Count", "http://tinyurl.com/nfdbyxa")
  )
)
```

The above defines a human function called `which_one` that takes no arguments.  It is important to note that `which_one` is just an ordinary function in Scala, although it does behave in some special ways that we will describe in the next section.

This function creates a "radio button question" on Mechanical Turk by calling the `radio` constructor.  "Radio button questions" allow MTurk users to select **one of n** options.

{% hint style="info" %}
AutoMan has constructors for numerous question types.  Refer to the [AutoMan API Reference](/technical-documentation/automan-api-reference) for more information.
{% endhint %}

{% hint style="info" %}
The constructor above has *many* parameters which are set to "sane defaults," so that you do not need to specify a task in great detail.  These defaults are designed to minimize your surprise.
{% endhint %}

The key elements in the question function above are:

* `budget`: This parameter specifies the maximum amount of money AutoMan will spend on this task.  AutoMan always tries to spend less.  If the cost of a task exceeds the budget you supply, AutoMan will shut down the task and return a "low-confidence answer."
* `text`: This parameter supplies the text of the question.  You describe what you want workers to do here.
* `options`: This parameter supplies the valid options.  Since this is a radio button question, each option will produce a radio button.\
  The `choice` constructor takes three parameters:
  * A `label` of type `Symbol`.  You can think of a `Symbol` as a special string designed for easy comparison.  This parameter is not visible to MTurk workers.
  * A `name`, which is visible to MTurk workers.
  * An **optional** `image_url`, which is a link to an image hosted somewhere on the Internet, [like this one](http://tinyurl.com/nuwyz3u).
* `confidence` (not shown): This parameter stands for the [statistical confidence level](https://en.wikipedia.org/wiki/Statistical_significance#Related_concepts) and is a floating-point number between `0` and `1` (exclusive).  A number approaching zero tells AutoMan that virtually any answer is fine.  A number approaching one tells AutoMan that you want to be *very* certain that it is correct.  Although the `confidence` parameter is not shown above, it is set to the default of `0.95`, which is something of a standard threshold across empirical science.

## Part 4. Call Your Human Function Inside an AutoMan Block

Now that you have a human function defined, you can call it.  First, paste the following *AutoMan block* below your function definition.

```scala
automan(a) {
  // your code will go here
}
```

We are going to call our function inside that block.  The purpose of an AutoMan block is to delineate when you are **done using MTurk**.  AutoMan needs this information so that it knows you are ready to shut down your program.

{% hint style="warning" %}
Failing to tell AutoMan to shutdown will cause your program to hang.
{% endhint %}

Now we can call `which_one()` inside our AutoMan block.

```scala
automan(a) {
  println("Answer is: " + which_one())
}
```

## The Complete Program

Here is the complete source code for our first AutoMan program (with the hello-world bit removed).  You can find a copy of this program in the [sample applications directory](https://github.com/automan-lang/AutoMan/tree/master/apps) of AutoMan's GitHub repository, along with many other examples.

```scala
import org.automanlang.adapters.mturk.DSL._

object MyFirstAutoManApp extends App {
  implicit val a = mturk (
    access_key_id = args(0),
    secret_access_key = args(1),
    sandbox_mode = true
  )

  def which_one() = radio (
    budget = 5.00,
    text = "Which one of these does not belong?",
    options = (
      choice('oscar, "Oscar the Grouch", "https://tinyurl.com/y2nf2h76"),
      choice('kermit, "Kermit the Frog", "https://tinyurl.com/yxh2emmr"),
      choice('spongebob, "Spongebob Squarepants", "https://tinyurl.com/y3uv2oew"),
      choice('cookiemonster, "Cookie Monster", "https://tinyurl.com/y68x9zvx"),
      choice('thecount, "The Count", "https://tinyurl.com/y6na5a8a")
    )
  )

  automan(a) {
    println("Answer is: " + which_one())
  }
}
```

{% hint style="danger" %}
Notice that the above program does not store any private access keys in the source text!  Instead, they must be passed in using command line arguments.  For additional examples with more sophisticated command line parsing, see the [sample applications directory](https://github.com/automan-lang/AutoMan/tree/master/apps).
{% endhint %}

## Run Your First App

Back on the command-line, we can now run our app with:

```
$ sbt "run <your access key> <your secret key>"
```

Assuming you have not changed the `mturk` initializer, this program will post jobs to the MTurk sandbox. When you run this program in the sandbox, no work will get done, because only the live production site has active workers.  You must simulate the job yourself.  We describe the process of simulating a job in the section titled [Pro Tip: Use the MTurk Sandbox](/getting-started/pro-tip-use-the-mturk-sandbox).

After posted jobs are completed, somewhere in AutoMan's voluminous output, you should see something like:

```
Answer is: 'spongebob
```

{% hint style="success" %}
If you want to see AutoMan's quality control algorithm in action, use the sandbox to supply worker responses that disagree.  You should observe that after AutoMan obtains all of the responses for a given round, it will decide whether disagreement is strong enough to warrant asking the crowd for more responses.
{% endhint %}

{% hint style="info" %}
The program above works, but it's a little simplistic.  Without some special configuration, MTurk will not let you post tasks with fewer than 10 HITs.  For an easy task like this, which may only need a few responses, this program is a little inefficient.  Also, if something goes wrong, this program hides that fact from you.

We provide additional example programs that work around these issues in the `apps` directory of our [GitHub repository](https://github.com/automan-lang/AutoMan).  An efficient, error-handling program that defines essentially the same task [can be found here](https://github.com/automan-lang/AutoMan/blob/master/apps/simple/SimpleRadioProgram/src/main/scala/SimpleRadioProgram.scala).
{% endhint %}

{% hint style="success" %}
If something goes wrong as your job runs, be sure to read AutoMan's (voluminous) output.  AutoMan provides a great deal of detail to make diagnosing problems easy.
{% endhint %}

##


# Pro Tip: Use the MTurk Sandbox

Although AutoMan is designed to make programming crowdsourcing jobs easier and more reliable, simulating MTurk jobs during development is slightly more complicated.  The reason is that AutoMan utilizes the MTurk Qualification system to ensure that a sample of completed assignments is [*i.i.d.*](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables)

## Multiple MTurk sandbox accounts = multiple email addresses

From a practical perspective, the presence of AutoMan's quality control algorithm implies that you will need  multiple MTurk sandbox accounts to simulate work in the sandbox.  You will also need a unique email address for every sandbox account that you set up.  We recommend creating at least five sandbox accounts; for estimates, you will need many more (the minimum sample size is 12 for estimates).

{% hint style="info" %}
An easy way to obtain unique email addresses that all forward to the same mailbox is to use GMail's [task-specific email address](https://support.google.com/a/users/answer/9308648?hl=en) feature.
{% endhint %}

## Debugging with multiple accounts = multiple browser sessions

While debugging, we find it useful to stay logged into multiple MTurk sandbox accounts simultaneously.  Until recently, doing so was not supported by default in browsers.  Fortunately, the [Firefox](https://getfirefox.com) web browser now ships with an addon called [Multi-Account Containers](https://addons.mozilla.org/en-US/firefox/addon/multi-account-containers/) that makes managing multiple sessions easy.

After you have set up containers, click and hold on the `+` sign on the browser's toolbar: it will give you the option of creating a new tab in a new session.  Signing in to a website in that tab will limit session state (e.g., cookies) to that one tab, which means that you can open another tab using another container and login using a different account.  This feature is a life saver for us!


# AutoMan Memoizer

AutoMan comes with a unique feature: it can store the results of crowdsourcing jobs in a database.  Remembering computed results, referred to as *memoization* in computer science, can speed up programs that perform repeated computation at the expense of space.  The memoization database can also be used to recover from a program crash if you make a programming mistake.  We refer to this feature as AutoMan's "memo DB."

Enabling the memo DB is easy.  Just add it to your `mturk` initializer:

```scala
implicit val a = mturk (
  /* .. whatever ... */
  database_path = "path/to/your/database/file"
)
```

{% hint style="warning" %}
Don't forget that your memo DB is enabled.  Although it can save your program some work and it often helps for programs in production, forgetting that you've enabled it in development can sometimes cause some baffling results during development.
{% endhint %}


# Cleaning Up

While developing AutoMan programs, it is not uncommon to create a large number of orphaned jobs in the MTurk sandbox.  While AutoMan normally cleans up after itself, if your program crashes or if you terminate it with a SIGINT signal (`Ctrl-C`), AutoMan will leave state behind on MTurk.  AutoMan state consists of three kinds of runtime objects:

* `HIT`s, which are the MTurk equivalent of a human function call,&#x20;
* `Assignment`s, which are the equivalent of a human function return value, and
* `Qualification`s, which are a runtime data structure that limits who can participate in a given crowdsourcing job.

Precisely how the AutoMan language utilizes these MTurk data structures is described in [our 2012 OOPSLA paper](/technical-documentation/papers).

There is little harm in leaving orphaned state behind, but it is often helpful to remove it for debugging purposes.  Or, if you're like me, because you just want to be tidy.

{% hint style="info" %}
To obtain these tools, you will need to [checkout AutoMan from source](https://github.com/automan-lang/AutoMan).

```
git clone https://github.com/automan-lang/AutoMan.git
```

Once you have a copy of the source, look in the [`tools`](https://github.com/automan-lang/AutoMan/tree/master/tools) directory.
{% endhint %}

{% hint style="info" %}
These tools are also useful if you, like me, occasionally botch a *live* job and want to cancel it immediately.  Just be aware--it is very bad practice to stiff workers, so these tools will *automatically approve* work and *pay* workers, whether their work is good or not.
{% endhint %}

## Installing Prerequisites

These tools are written in Java and require that you install the [Apache Maven](https://maven.apache.org/) tool.  On the Mac, Maven is [available via Homebrew](https://formulae.brew.sh/formula/maven).

## Access Keys

These tools require that you store your access keys in a Java `.properties` file.  I typically call my file `mturk.properties` and keep it someplace safe.

Your `mturk.properties` file should be a text file formatted using the following convention:

```java
access_key=<your access key here>
secret_key=<your secret key here>
```

For example, here is a sample `mturk.properties` file (with bogus keys):

```java
access_key=AKIA5KJ00DF3HJKJ2B9N
secret_key=qGCCxrnAUlvW12KRuuCo5i5m80ptEVclf7kjAbsK
```

{% hint style="danger" %}
Remember: never post your MTurk access keys to a public site, such as your GitHub repository.  If you do so, remove your keys from your MTurk account using [AWS IAM](https://aws.amazon.com/iam/) immediately.
{% endhint %}

## Removing orphaned HITs and Assignments

The `DeleteAllHITs` tool, found in AutoMan's [`tools`](https://github.com/automan-lang/AutoMan/tree/master/tools) directory, will delete all the HITs for your account, either in the MTurk sandbox or in the live production site.  You can see a help message by running the program without arguments.

```bash
$ ./run.sh
Usage:
  You should use the "run.sh" shell script.

  ./run.sh <path to mturk.properties file> <sandbox mode true/false>

  For example:
    /run.sh ~/mturk.properties false
```

For example,

```
$ ./run.sh mturk.properties true
```

will delete all HITs and Assignments on the MTurk sandbox.

By changing `true` to `false`, you can also delete all HITs and Assignments on the live, production MTurk site.

{% hint style="warning" %}
Be aware that this script will pay workers for any completed work on the production MTurk site before deleting assignments and HITs.
{% endhint %}

## Removing all Qualifications

The `DeleteAllQualifications` tool will delete all `Qualification` objects from MTurk.  As with the `DeleteAllHITs` tool, it can be run on either the sandbox or the live production site.

```
$ ./run.sh 
Usage:
  You should use the "run.sh" shell script.

  ./run.sh <path to mturk.properties file> <sandbox mode true/false>

  For example:
    /run.sh ~/mturk.properties false
```

For example,

```
$ ./run.sh mturk.properties true
```

will delete all `Qualification` objects on the MTurk sandbox.

By changing `true` to `false`, you can also delete all `Qualification` objects on the live, production MTurk site.

{% hint style="danger" %}
Before running this tool, be sure that you have no active AutoMan programs running.  AutoMan uses `Qualification` objects internally and expects that the ones it creates remain on the site until it deletes them.  Deleting a `Qualification` for a running job will likely result in a program crash.
{% endhint %}


# What is AutoMan?

AutoMan is the first fully automatic *crowdprogramming* system. AutoMan integrates human-based ("crowdsourced") computations into a standard programming language as ordinary function calls that can be intermixed freely with traditional functions. This abstraction lets programmers focus on their programming logic.

An AutoMan program specifies a *confidence level* for the overall computation and a *budget*. The AutoMan runtime system then transparently manages all details necessary for scheduling, pricing, and quality control. AutoMan automatically schedules human tasks for each computation until it achieves the desired confidence level; monitors, reprices, and restarts human tasks as necessary; and maximizes parallelism across human workers while staying under budget.&#x20;

AutoMan is available as a library for Scala.&#x20;

## Where did it all come from?&#x20;

AutoMan is being actively developed by [Daniel Barowy](http://www.cs.williams.edu/~dbarowy/) at [Williams College](https://csci.williams.edu/) and [Emery Berger](https://emeryberger.com/) at the [PLASMA Laboratory](http://plasma.cs.umass.edu/) at the [University of Massachusetts Amherst](https://www.cics.umass.edu/). Portions of AutoMan were developed as a collaboration with researchers at [Microsoft Research NYC](https://www.microsoft.com/en-us/research/lab/microsoft-research-new-york/).


# Getting AutoMan

The easiest way to get AutoMan is via the Maven Central Repository. If you're using SBT:

```scala
libraryDependencies += "org.automanlang" %% "automan" % "1.4.2"
```

*or* if you're using Maven:

```bash
    <dependency>
      <groupId>org.automanlang</groupId>
      <artifactId>automan_2.12</artifactId>
      <version>1.4.2</version>
    </dependency>
```


# Quick Start Guide

Follow this guide if you are \_already\_ familiar with Scala, SBT, etc.

{% hint style="info" %}
If you have little experience with Scala or SBT, we recommend that you follow our [Getting Started guide](/getting-started/installation-tutorial) instead.
{% endhint %}

In your source file, import the Mechanical Turk adapter (Scala syntax):

```scala
import org.automanlang.adapters.mturk.DSL._
```

After that, initialize the AutoMan runtime with an MTurk config:

```scala
implicit val mt = mturk (
  access_key_id = "my key",         // your MTurk "access key"
  secret_access_key = "my secret",  // your MTurk "secret key" 
  sandbox_mode = true               // if true, run on MTurk sandbox
)
```

and then define your task:

```scala
def which_one() = radio(
  budget = 5.00,
  text = "Which one of these does not belong?",
  options = (
    "Oscar the Grouch",
    "Kermit the Frog",
    "Spongebob Squarepants",
    "Cookie Monster",
    "The Count"
  )
)
```

You may then call `which_one` just like an ordinary function (which it is). Note that AutoMan functions immediately return an `Outcome`, but continue to execute asynchronously in the background.  AutoMan builds on top of a Scala feature called a [future](https://docs.scala-lang.org/overviews/core/futures.html) to make this happen.

To access return values, pattern-match on the `Outcome`'s `answer` field, e.g.,

```scala
val outcome = which_one()

// ... do some other stuff ...

// then, when you want answers ...
val answer = outcome.answer match {
  case Answer(value, _, _) => value
  case _ => throw new Exception("Oh no!")
}
```

Other possible cases are `LowConfidenceAnswer` and `OverBudgetAnswer`. If you run out of money during a computation, a `LowConfidenceAnswer` will let you access to lower-confidence results. An `OverBudgetAnswer` signals that you didn't have enough money in your budget to begin with.

#### Cleanup of AutoMan Resources

Note that, due to AutoMan's design, you must inform it when to shut down, otherwise it will continue to execute indefinitely and your program will hang:

```scala
mt.close()
```

Alternately, you may wrap your program in an `automan` statement, and cleanup will happen automatically. This feature was [inspired](https://msdn.microsoft.com/en-us/library/vstudio/yh598w02%28v=vs.100%29.aspx) by the C# `using` statement:

```scala
automan(mt) {
  ... your program ...
}
```

We will add more documentation to this site in the near future. In the interim, please see the collection of sample programs in the `apps` directory.


# AutoMan API Reference

## Supported Question Types

| Question Type | Purpose                                                                                                                  | Quality-Controlled | Number of Answers Returned |
| ------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------ | -------------------------- |
| `radio`       | The user is asked to choose one of n options.                                                                            | yes                | 1                          |
| `checkbox`    | The user is asked to choose one of m of n options, where m <= n.                                                         | yes                | 1                          |
| `freetext`    | The user is asked to enter a textual response, such that the response conforms to a simple pattern (a "picture clause"). | yes                | 1                          |
| `estimate`    | The user is asked to enter a numeric (real-valued) response.                                                             | yes                | 1                          |
| `radios`      | Same as `radio`, except that it returns the entire distribution.                                                         | no                 | sample size                |
| `checkboxes`  | Same as `checkbox`, except that it returns the entire distribution.                                                      | no                 | sample size                |
| `freetexts`   | Same as `freetext`, except that it returns the entire distribution.                                                      | no                 | sample size                |

{% hint style="info" %}
The primary difference between "quality controlled" and "non-quality controlled" questions is whether you want a single, quality-controlled answer, or all of the answers.  The former is useful in batch computation, where you are relying on the "wisdom of the crowd" to choose the best answer.  The latter is used to obtain *i.i.d.* samples of the crowd.
{% endhint %}

### Calling a Question Type

We describe question type signatures below.  It is important to note that calling a question type constructor *immediately launches a crowdsoucing task*.  This is not usually what you want, which is why [all of our sample applications](https://github.com/automan-lang/AutoMan/tree/master/apps) utilize the following pattern:

```scala
def my_function(<arg>, ...) = <AutoMan constructor>(<configuration>)
```

For example, here is a sample human function for calorie counting:

```scala
def howManyCals(imgUrl: String) = estimate (
    budget = 6.00,
    confidence_interval = SymmetricCI(50),
    text = "Estimate how many calories (kcal) are " +
           "present in the picture shown in the photo.",
    image_url = imgUrl,
    min_value = 0
)
```

Observe how we use a Scala user-defined function (`def`) to pass the `imgUrl` parameter through to the `estimate` constructor.  See our [sample apps](https://github.com/automan-lang/AutoMan/tree/master/apps) for additional examples.

### Question Return Types

Another thing to note is that all AutoMan question constructors return a result belonging to the supertype `Outcome`.  Although you can call `toString` on such return values to obtain a simple, printable string, you should probably pattern-match on the result value returned by calling `answer` (or `answers`, depending on the question) on the returned `Outcome` object.  Each question type has a different set of possible return values.  We describe them in the next section.

You are encouraged to look at the [sample apps](https://github.com/automan-lang/AutoMan/tree/master/apps) for examples.

{% hint style="info" %}
AutoMan question function constructors return *immediately* and run *asynchronously* in a background thread.  This is an intentional design decision to allow you to start a crowdsourcing job and do other work while the task runs.  Calling `answer` (or `answers`, depending on the question type) will *block* execution until the task is done running, which may be a substantial amount of time.  Be sure that you want blocking behavior when you call `answer`.
{% endhint %}

{% hint style="warning" %}
The `toString` method for `Outcome` calls `answer` internally, which means that it blocks!
{% endhint %}

### Question Type Constructor Signatures

We provide question constructors here.  Note that all of them take a very large number of parameters, but that most of those parameters are the same between question types and nearly all of them have "sane defaults."  Defaults are managed by requiring the use of [named arguments](https://docs.scala-lang.org/tour/named-arguments.html).

Therefore, we provide two constructor signatures for each question type: the 1) simplified constructor showing only mandatory parameters, and 2) the full ScalaDoc-generated constructor with all parameters.  We also describe common parameters at the end.

We describe the variants used in the `mturk` DSL here.

#### Radio Button Questions

The following constructor parameters are mandatory:

```scala
def radio(
  options: List[MTQuestionOption],
  text: String
  (implicit a: A): ScalarOutcome[Symbol] 
```

* `options` are the selection options seen by the user, along with optional images.  Options can be created using one of the following `choice` constructors:

  * `choice(key: Symbol, text: String)` or&#x20;
  * `choice(key: Symbol, text: String, image_url: String)`

  where `key` denotes a stable identifier for a choice (e.g., `kermit`) not shown to the worker, `text` is the text label shown to the worker, and `image_url` is a url of an image shown beside the text label.
* `text` is the text of the question shown in a `HIT` and, by default, also as the task title.  You can override the title by setting the `title` parameter.

Radio button questions can return the following values:

* `Answer[Symbol]`: An object that represents a selected radio button, where each possible `Symbol` was defined with the `key` parameter of the `choice` constructors described above.  This object has the following fields:
  * `value`: the answer (`Symbol`).
  * `cost`: the total cost (`BigDecimal`)
  * `confidence`: the final confidence value (`Double`).
  * `distribution`: raw sample responses (`Array[Response[Symbol]]`)
* `LowConfidenceAnswer`, which has the same fields as `Answer` but which indicates that a quality-controlled response has a `confidence` lower than the desired threshold.
* `OverBudgetAnswer`, which indicates that a specified task cannot run at all due to insufficient funds.  This object has the following fields:
  * `need`: the funds needed to start a job (`BigDecimal`)
  * `have`: the funds at hand (`BigDecimal`)
* `NoAnswer`, which indicates that an unexpected runtime error occurred.

The following is a ScalaDoc-generated signature:

```scala
def radio[A <: AutomanAdapter, O](
  confidence: Double = MagicNumbers.DefaultConfidence,
  budget: BigDecimal = MagicNumbers.DefaultBudget,
  dont_reject: Boolean = true,
  dry_run: Boolean = false,
  image_alt_text: String = null,
  image_url: String = null,
  initial_worker_timeout_in_s: Int = ...,
  minimum_spawn_policy: MinimumSpawnPolicy = null,
  mock_answers: Iterable[MockAnswer[Symbol]] = null,
  options: List[AnyRef],
  pay_all_on_failure: Boolean = true,
  question_timeout_multiplier: Double = ...,
  text: String,
  title: String = null,
  wage: BigDecimal = MagicNumbers.USFederalMinimumWage)
  : ScalarOutcome[Symbol] 
```

#### Checkbox Questions

The following constructor parameters are mandatory:

```scala
def checkbox(
  options: List[MTQuestionOption],
  text: String)
  : ScalarOutcome[Set[Symbol]] 
```

* `options` are the selection options seen by the user, along with optional images.  Options can be created using one of the following `choice` constructors:

  * `choice(key: Symbol, text: String)` or&#x20;
  * `choice(key: Symbol, text: String, image_url: String)`

  where `key` denotes a stable identifier for a choice (e.g., `kermit`) not shown to the worker, `text` is the text label shown to the worker, and `image_url` is a url of an image shown beside the text label.
* `text` is the text of the question shown in a `HIT` and, by default, also as the task title.  You can override the title by setting the `title` parameter.

Checkbox questions can return the following values:

* `Answer[Set[Symbol]]`: An object that represents a set of selected checkboxes, where each `Symbol` was defined with the `key` parameter of the `choice` constructors described above.  This object has the following fields:
  * `value`: the answer (`Set[Symbol]`).
  * `cost`: the total cost (`BigDecimal`)
  * `confidence`: the final confidence value (`Double`).
  * `distribution`: raw sample responses (`Array[Response[Set[Symbol]]]`)
* `LowConfidenceAnswer`, which has the same fields as `Answer` but which indicates that a quality-controlled response has a `confidence` lower than the desired threshold.
* `OverBudgetAnswer`, which indicates that a specified task cannot run at all due to insufficient funds.  This object has the following fields:
  * `need`: the funds needed to start a job (`BigDecimal`)
  * `have`: the funds at hand (`BigDecimal`)
* `NoAnswer`, which indicates that an unexpected runtime error occurred.

The following is a ScalaDoc-generated signature:

```scala
def checkbox[A <: AutomanAdapter, O](
  confidence: Double = MagicNumbers.DefaultConfidence,
  budget: BigDecimal = MagicNumbers.DefaultBudget,
  dont_reject: Boolean = true,
  dry_run: Boolean = false,
  image_alt_text: String = null,
  image_url: String = null,
  initial_worker_timeout_in_s: Int = ...,
  minimum_spawn_policy: MinimumSpawnPolicy = null,
  mock_answers: Iterable[MockAnswer[Set[Symbol]]] = null,
  options: List[AnyRef],
  pay_all_on_failure: Boolean = true,
  question_timeout_multiplier: Double = ...,
  text: String,
  title: String = null,
  wage: BigDecimal = MagicNumbers.USFederalMinimumWage)
  (implicit a: A)
  : ScalarOutcome[Set[Symbol]] 
```

#### Free-Text Questions

The following constructor parameters are mandatory:

```
def freetext(
  pattern: String,
  text: String)
  : ScalarOutcome[String] 
```

* `pattern` is a COBOL-style *picture clause* pattern that states what inputs are valid.  AutoMan uses this pattern to perform probability calculations.  `A` matches an alphabetic character, `B` matches an optional alphabetic character, `X` matches an alphanumeric character, `Y` matches an optional alphanumeric character, `9` matches a numeric character, and `0`matches an optional numeric character. For example, a telephone number recognition application might use the pattern `09999999999`.
* `text` is the text of the question shown in a `HIT` and, by default, also as the task title.  You can override the title by setting the `title` parameter.

The following parameters are `freetext`-specific:

* `allow_empty_pattern` means that the empty string is a valid worker response.\
  **default**: `false`
* `before_filter` is not currently used.
* `pattern_error_text` is a helpful message that is displayed to the user when their input does not match a pattern.  It is not mandatory but it is highly recommended that you use this setting.

{% hint style="info" %}
You should strongly consider using `pattern_error_text` for `freetext` questions as the default MTurk help message is not helpful.  This parameter gives you an opportunity to provide an error explanation in non-technical terms.
{% endhint %}

Free-text questions can return the following values:

* `Answer[String]`: An object that represents a response string.  This object has the following fields:
  * `value`: the answer (`String`).
  * `cost`: the total cost (`BigDecimal`)
  * `confidence`: the final confidence value (`Double`).
  * `distribution`: raw sample responses (`Array[Response[String]]`)
* `LowConfidenceAnswer`, which has the same fields as `Answer` but which indicates that a quality-controlled response has a `confidence` lower than the desired threshold.
* `OverBudgetAnswer`, which indicates that a specified task cannot run at all due to insufficient funds.  This object has the following fields:
  * `need`: the funds needed to start a job (`BigDecimal`)
  * `have`: the funds at hand (`BigDecimal`)
* `NoAnswer`, which indicates that an unexpected runtime error occurred.

The following is a ScalaDoc-generated signature:

```scala
def freetext[A <: AutomanAdapter](
  allow_empty_pattern: Boolean = false,
  confidence: Double = MagicNumbers.DefaultConfidence,
  before_filter: (String) ⇒ String = (a: String) => a,
  budget: BigDecimal = MagicNumbers.DefaultBudget,
  dont_reject: Boolean = true,
  dry_run: Boolean = false,
  image_alt_text: String = null,
  image_url: String = null,
  initial_worker_timeout_in_s: Int = ...,
  minimum_spawn_policy: MinimumSpawnPolicy = null,
  mock_answers: Iterable[MockAnswer[String]] = null,
  pay_all_on_failure: Boolean = true,
  pattern: String,
  pattern_error_text: String = null,
  question_timeout_multiplier: Double = ...,
  text: String,
  title: String = null,
  wage: BigDecimal = MagicNumbers.USFederalMinimumWage)
  (implicit a: A)
  : ScalarOutcome[String] 
```

#### Estimates

There is [an entire paper](/technical-documentation/papers) (VoxPL) about this one question type.

The following constructor parameters are mandatory:

```scala
def estimate(
  confidence_interval: ConfidenceInterval,
  text: String)
  : EstimationOutcome 
```

* `confidence_interval` lets you denote the confidence interval of an estimate.  The options are:
  * `UnconstrainedCI()` which will only even perform one round of tasks using the default sample size, returning the $$L\_1$$ median.
  * `SymmetricCI(err: Double)` which returns the $$L\_1$$ median $$\pm$$ `err` with `confidence` level confidence.
  * `AsymmetricCI(lerr: Double, herr: Double)` which returns the $$L\_1$$ median of an estimate between `-lerrr` and `+herr` with `confidence` level confidence.
* `text` is the text of the question shown in a `HIT` and, by default, also as the task title.  You can override the title by setting the `title` parameter.

Estimates can return the following values:

* `Estimate`: An object that represents a "best estimate".  This object has the following fields:
  * `value`: the estiamte (`Double`).
  * `low`: the low bound of a confidence interval's estimate (`Double`).
  * `high`: the high bound of a confidence interval's estimate (`Double`).
  * `cost`: the total cost (`BigDecimal`)
  * `confidence`: the final confidence value (`Double`).
  * `distribution`: raw sample responses (`Array[Response[Double]]`)
* `LowConfidenceEstimate`, which has the same fields as `Estimate` but which indicates that a quality-controlled response has a `confidence` lower than the desired threshold.
* `OverBudgetEstimate`, which indicates that a specified task cannot run at all due to insufficient funds.  This object has the following fields:
  * `need`: the funds needed to start a job (`BigDecimal`)
  * `have`: the funds at hand (`BigDecimal`)
* `NoEstimate`, which indicates that an unexpected runtime error occurred.

The following is a ScalaDoc-generated signature:

```scala
def estimate[A <: AutomanAdapter](
  confidence_interval: ConfidenceInterval = UnconstrainedCI(),
  confidence: Double = MagicNumbers.DefaultConfidence,
  budget: BigDecimal = MagicNumbers.DefaultBudget,
  default_sample_size: Int = -1,
  dont_reject: Boolean = true,
  dry_run: Boolean = false,
  estimator: (Seq[Double]) ⇒ Double = null,
  image_alt_text: String = null,
  image_url: String = null,
  initial_worker_timeout_in_s: Int = ...,
  max_value: Double = Double.MaxValue,
  minimum_spawn_policy: MinimumSpawnPolicy = null,
  min_value: Double = Double.MinValue,
  mock_answers: Iterable[MockAnswer[Double]] = null,
  pay_all_on_failure: Boolean = true,
  question_timeout_multiplier: Double = ...,
  text: String,
  title: String = null,
  wage: BigDecimal = MagicNumbers.USFederalMinimumWage)
  (implicit a: A)
  : EstimationOutcome 
```

#### Sampling Questions

We describe the `checkboxes` constructor here, but `freetexts` and `radios` are similar.  There is also a buggy `multiestimate` constructor that should probably not be used at the moment.

The following constructor parameters are mandatory:

```scala
def checkboxes(
  sample_size: Int = ...,
  options: List[MTQuestionOption],
  text: String)
  : VectorOutcome[Set[Symbol]] 
```

* `sample_size` is the size of the sample.
* `options` are the selection options seen by the user, along with optional images.  Options can be created using one of the following `choice` constructors:

  * `choice(key: Symbol, text: String)` or&#x20;
  * `choice(key: Symbol, text: String, image_url: String)`

  where `key` denotes a stable identifier for a choice (e.g., `kermit`) not shown to the worker, `text` is the text label shown to the worker, and `image_url` is a url of an image shown beside the text label.
* `text` is the text of the question shown in a `HIT` and, by default, also as the task title.  You can override the title by setting the `title` parameter.

The following is a ScalaDoc-generated signature:

```scala
def checkboxes[A <: AutomanAdapter, O](
  sample_size: Int = ...,
  budget: BigDecimal = MagicNumbers.DefaultBudget,
  dont_reject: Boolean = true,
  dry_run: Boolean = false,
  image_alt_text: String = null,
  image_url: String = null,
  initial_worker_timeout_in_s: Int = ...,
  minimum_spawn_policy: MinimumSpawnPolicy = null,
  mock_answers: Iterable[MockAnswer[Set[Symbol]]] = null,
  options: List[AnyRef],
  pay_all_on_failure: Boolean = true,
  question_timeout_multiplier: Double = ...,
  text: String,
  title: String = null,
  wage: BigDecimal = MagicNumbers.USFederalMinimumWage)
  (implicit a: A): VectorOutcome[Set[Symbol]] 
```

#### Common Default Parameters

The following are parameters common to all calls:

* `budget` is the total amount of money to be spent by a *given* human question function call.  Note that this means that *each function call* has its own budget.\
  **default**: $5.00
* `dont_reject`, when set to `true`, will always accept completed assignments and pay workers for their work.  This is useful when work is difficult and errors are likely, or when you just don't want to deal with the hassle of reputation management. \
  **default**: `false`
* `dry_run`, when set to `true`, will not actually post jobs on MTurk.\
  **default**: `false`
* `image_alt_text` adds an HTML `ALT` annotation to the `IMG` tag created by the `image_url` parameter.\
  **default**: none (`null`)
* `image_url` adds an image to a question.  Such images should be hosted someplace publically-accessible, such as Amazon S3 or a personal website.\
  **default**: none (`null`)
* `initial_worker_timeout_in_s` is the amount of time permitted to a worker in the initial round of tasks.  Note that the actual time permitted depends on the number of rounds and is determined by the quality control policy.  The default policy uses the formula $$w m^r$$ , where $$w$$ is the `initial_worker_timeout_in_s`, $$m=2$$, and $$r$$ is the round.  In other words, task timeout are doubled.\
  **default**: 30 seconds
* `minimum_spawn_policy` states what the smallest number of assignments for a given `HIT` are on MTurk.  This is necessary because MTurk has two totally boneheaded policies:

  * HITs posted with 10 or fewer assignments are charged a 20% fee while HITs with more than 10 assignments are charged a 40% fee.
  * HITs with 10 or fewer assignments cannot be "extended" to have more assignments.

  For now, what this means is that, if you do not change the default, AutoMan will post tasks with at least 10 assignments.  If you anticipate that your tasks will likely need fewer than 10 assignments, you can set the anticipated amount by setting this to `UserDefinableSpawnPolicy(n)` where `n` is the number you want.\
  **default**: 10\
  **note**: I am actively unhappy about this and am thinking of ways to simplify it.  [Suggestions welcome](https://github.com/automan-lang/AutoMan/issues).
* `mock_answers` sets AutoMan to be used in *mock* mode for testing purposes.  This is used interally by AutoMan for testing.  You should not change this.\
  **default**: `null`
* `pay_all_on_failure` controls whether workers are paid when a task runs out of money.  Setting this to `false` means that workers will not be paid when an `OverBudget` result is returned, which generally makes workers unhappy.\
  **default**: `true`
* `question_timeout_multiplier` controls how much time a HIT exists on MTurk before it is timed out.  Note that this is a distinct timeout from the amount of time a worker is given to complete a task, which is controlled by the `initial_worker_timeout_in_s` parameter.  A HIT's total time is determined by the formula $$w m^r t$$ , where $$w$$ is the `initial_worker_timeout_in_s`, $$m=2$$, $$r$$ is the round, and $$t$$ is the `question_timeout_multiplier`.

  \
  **default**: 500
* `wage` controls the base wage for a worker.  The actual reward paid depends on how much time a worker is given to do a task.  The default policy uses a maximum likelihood estimate of the probability that a task is accepted in order to compute a wage that disincentivzes wage gaming behavior.  It is complicated enough that if you want to know its inner workings, you should [read our 2016 CACM article](/technical-documentation/papers). Generally you should think of the reward as "probably doubling."\
  **default**: the U.S. Federal Minimum wage, or $7.25/hour
* `a` is an initialized AutoMan platform adapter.  Typically this will be an `implicit` variable that you return from a platform initializer expression like `mturk`.  When marked `implicit`, you do not need to pass the parameter yourself; Scala will find it in the environment and pass it, simplifying human function calls.  AutoMan needs this information in order to bind a human function call to a given crowdsourcing platform.

## Using AutoMan with a Different Crowdsourcing Backend

We currently only support Amazon's Mechanical Turk. However, AutoMan was designed to accommodate arbitrary backends. If you are interested in seeing your crowdsourcing platform supported, please contact us.

#### Memoization

AutoMan can be configured to save all intermediate human-computed results.  Set the location of the database with `database_path = "/path/to/your/database"`. The format of the database is H2.

## Sample Applications <a href="#sample_apps" id="sample_apps"></a>

Sample applications can be found in the `apps` directory. Apps can also be built using `pack`. E.g.,

```bash
$ cd apps/simple_program
$ sbt pack
```

Unix/DOS shell scripts for running the programs can then be found in `apps/[the app]/target/pack/bin/`.


# Papers

More detailed information is available in our papers:

* &#x20;[CHI ‘17](https://web.archive.org/web/20180831211003/http://barowy.net/papers/voxpl-chi.pdf): **VoxPL: Programming with the Wisdom of the Crowd**\
  &#x20;Daniel W. Barowy, Emery D. Berger, Daniel G. Goldstein, and Siddharth Suri
  * [talk slides](https://web.archive.org/web/20180831211003/https://s3.amazonaws.com/dbarowy-cics-assets/VoxPL-CHI-2017.key)
* &#x20;[CACM RH ‘16](https://web.archive.org/web/20180831211003/http://dl.acm.org/citation.cfm?id=2927928): **AutoMan: A Platform for Integrating Human-Based and Digital Computation (Research Highlight)**\
  &#x20;Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, and Andrew McGregor
* &#x20;[OOPSLA ‘12](https://web.archive.org/web/20180831211003/http://www.cs.umass.edu/~emery/pubs/res0007-barowy.pdf): **AutoMan: A Platform for Integrating Human-Based and Digital Computation**\
  &#x20;Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, and Andrew McGregor
  * [talk slides](https://web.archive.org/web/20180831211003/https://s3.amazonaws.com/dbarowy-cics-assets/automan_oopsla_2012.ppt)
* [Tech report UMass CS TR 2011-44](https://web.archive.org/web/20180831211003/http://www.cs.umass.edu/~emery/pubs/AutoMan-UMass-CS-TR2011-44.pdf)
* Second Workshop on Computational Social Science and the Wisdom of Crowds, ‘11
  * [NIPS poster](https://web.archive.org/web/20180831211003/https://s3.amazonaws.com/dbarowy-cics-assets/automan_nips_poster.jpg)
  * [NIPS workshop paper](https://web.archive.org/web/20180831211003/http://web.archive.org/web/20160808010421id_/https://people.cs.umass.edu/~wallach/workshops/nips2011css/papers/Barowy.pdf)

There are two versions of the original AutoMan paper, a shortened Research Highlight that appeared in the June 2016 issue of the Communications of the ACM and a longer version that appeared at OOPSLA 2012. You should probably start with the CACM version which has a bit more polish and some updated results.

The VoxPL paper describes our approach to automate the original wisdom of the crowd task, estimation. We introduce a novel quality control algorithm and describe how we rebuilt AutoMan’s architecture to handle these new kinds of tasks.

Full citations are given below:

*VoxPL*

```javascript
@inproceedings{Barowy:2017:VPW:3025453.3026025,
 author = {Barowy, Daniel W. and Berger, Emery D. and Goldstein, Daniel G. and Suri, Siddharth},
 title = {VoxPL: Programming with the Wisdom of the Crowd},
 booktitle = {Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems},
 series = {CHI '17},
 year = {2017},
 isbn = {978-1-4503-4655-9},
 location = {Denver, Colorado, USA},
 pages = {2347--2358},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/3025453.3026025},
 doi = {10.1145/3025453.3026025},
 acmid = {3026025},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {crowdprogramming, crowdsourcing, domain-specific languages, quality control, scalability, wisdom of the crowd},
}
```

*AutoMan (Communications of the ACM Research Highlight)*

```javascript
@article{Barowy:2016:API:2942427.2927928,
 author = {Barowy, Daniel W. and Curtsinger, Charlie and Berger, Emery D. and McGregor, Andrew},
 title = {AutoMan: A Platform for Integrating Human-based and Digital Computation},
 journal = {Commun. ACM},
 issue_date = {June 2016},
 volume = {59},
 number = {6},
 month = may,
 year = {2016},
 issn = {0001-0782},
 pages = {102--109},
 numpages = {8},
 url = {http://doi.acm.org/10.1145/2927928},
 doi = {10.1145/2927928},
 acmid = {2927928},
 publisher = {ACM},
 address = {New York, NY, USA},
}
```

*AutoMan (original)*

```javascript
@inproceedings{Barowy:2012:API:2384616.2384663,
 author = {Barowy, Daniel W. and Curtsinger, Charlie and Berger, Emery D. and McGregor, Andrew},
 title =  {{AutoMan} : a platform for integrating human-based and digital computation},
 booktitle = {Proceedings of the ACM International Conference on Object-Oriented Programming Systems Languages and Applications},
 series = {OOPSLA '12},
 year = {2012},
 isbn = {978-1-4503-1561-6},
 location = {Tucson, Arizona, USA},
 pages = {639--654},
 numpages = {16},
 url = {http://doi.acm.org/10.1145/2384616.2384663},
 doi = {10.1145/2384616.2384663},
 acmid = {2384663},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {crowdsourcing, programming languages, quality control},
}
```

Contact information:

Dan Barowy, <dbarowy@cs.williams.edu> Emery Berger, <emery@cs.umass.edu>


# Press

Ooh!  Look at us!

Press coverage of AutoMan:

1. [New Scientist](https://web.archive.org/web/20180831205221/https://www.newscientist.com/article/mg21628945.500-your-next-boss-could-be-a-computer)
2. [Times of India](https://web.archive.org/web/20180831205221/http://timesofindia.indiatimes.com/home/science/Your-next-boss-could-be-a-computer/articleshow/17531158.cms)
3. [ABC (Spain)](https://web.archive.org/web/20180831205221/http://www.abc.es/tecnologia/20121213/abci-jefe-sera-ordenador-201212122152.html)
4. [Naftemporiki (Greece)](https://web.archive.org/web/20180831205221/http://www.naftemporiki.gr/story/354864)
5. [Wired (Italy)](https://web.archive.org/web/20180831205221/https://daily.wired.it/news/tech/2012/12/07/capo-computer-323456.html)
6. [CCC Computing Research Highlight of the Week](https://web.archive.org/web/20180831205221/http://archive2.cra.org/ccc/resources/highlights/past-highlights/239-your-next-boss-could-be-a-computer)


# Bugs / Source / Building

## Reporting bugs <a href="#bugs" id="bugs"></a>

Please report bugs using this repository's [issue tracker](https://github.com/dbarowy/AutoMan/issues).

## Getting the Source / Contributing

AutoMan is available at <https://github.com/automan-lang/AutoMan>.

We are happy to accept pull requests!  AutoMan is licensed under the GPL version 2.  Note that pull requests will require an attribution statement that assigns copyright to the University of Massachusetts Amherst / Williams College.

## Building AutoMan <a href="#building_automan" id="building_automan"></a>

You do not need to build AutoMan yourself, as it is available via Maven as a JAR. However, if you want to hack on AutoMan, or if you just like building stuff, the AutoMan source code includes an SBT build script. The build script builds the AutoMan JAR for you, including downloading all of AutoMan's dependencies. The build script can also build the sample applications that are located in the `apps` directory. These applications are the ones used in our papers.

You can build the AutoMan JAR using the following commands:

```bash
$ cd libautoman
$ sbt pack
```

The AutoMan JAR plus all of its dependencies will then be found in the `libautoman/target/pack/lib/` folder.


# License

AutoMan is licensed under the GPLv2, Copyright (C) 2011-2020 The University of Massachusetts, Amherst / Williams College.

## Table of Contents

* [~~GNU GENERAL PUBLIC LICENSE~~](broken://pages/-MKVy0QvlZx3Bni_RMvX#SEC1)
  * [~~Preamble~~](broken://pages/-MKVy0QvlZx3Bni_RMvX#SEC2)
  * [~~TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION~~](broken://pages/-MKVy0QvlZx3Bni_RMvX#SEC3)
  * [~~How to Apply These Terms to Your New Programs~~](broken://pages/-MKVy0QvlZx3Bni_RMvX#SEC4)

## [GNU GENERAL PUBLIC LICENSE](broken://pages/-MKVy0QvlZx3Bni_RMvX)

&#x20;Version 2, June 1991

```
Copyright (C) 1989, 1991 Free Software Foundation, Inc.  
51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA

Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
```

## [Preamble](broken://pages/-MKVy0QvlZx3Bni_RMvX) <a href="#preamble" id="preamble"></a>

&#x20;The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too.

&#x20;When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things.

&#x20;To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it.

&#x20;For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.

&#x20;We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software.

&#x20;Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations.

&#x20;Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all.

&#x20;The precise terms and conditions for copying, distribution and modification follow.

## [TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION](broken://pages/-MKVy0QvlZx3Bni_RMvX) <a href="#terms" id="terms"></a>

&#x20;**0.** This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you".

&#x20;Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does.

&#x20;**1.** You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program.

&#x20;You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee.

&#x20;**2.** You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: **a)** You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. **b)** You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. **c)** If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.)

&#x20;These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it.

&#x20;Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program.

&#x20;In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License.

&#x20;**3.** You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: **a)** Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, **b)** Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, **c)** Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.)

&#x20;The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable.

&#x20;If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code.

&#x20;**4.** You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

&#x20;**5.** You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it.

&#x20;**6.** Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License.

&#x20;**7.** If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program.

&#x20;If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances.

&#x20;It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice.

&#x20;This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License.

&#x20;**8.** If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License.

&#x20;**9.** The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.

&#x20;Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation.

&#x20;**10.** If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally.

**NO WARRANTY**

&#x20;**11.** BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

&#x20;**12.** IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

## END OF TERMS AND CONDITIONS

## [How to Apply These Terms to Your New Programs](broken://pages/-MKVy0QvlZx3Bni_RMvX) <a href="#howto" id="howto"></a>

&#x20;If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms.

&#x20;To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found.

```
one line to give the program's name and an idea of what it does.
Copyright (C) yyyy  name of author

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
```

&#x20;Also add information on how to contact you by electronic and paper mail.

&#x20;If the program is interactive, make it output a short notice like this when it starts in an interactive mode:

```
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details
type `show w'.  This is free software, and you are welcome
to redistribute it under certain conditions; type `show c' 
for details.
```

&#x20;The hypothetical commands \`show w' and \`show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than \`show w' and \`show c'; they could even be mouse-clicks or menu items--whatever suits your program.

&#x20;You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names:

```
Yoyodyne, Inc., hereby disclaims all copyright
interest in the program `Gnomovision'
(which makes passes at compilers) written 
by James Hacker.

signature of Ty Coon, 1 April 1989
Ty Coon, President of Vice
```

&#x20;This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the [GNU Lesser General Public License](https://www.gnu.org/licenses/lgpl.html) instead of this License.