Tutorial: Write and Run Your First App
An AutoMan app always has at least four parts.
As is typical in most programming languages, we have to tell Scala that we want to use AutoMan. Paste the following
import statement into the top of your source file:import org.automanlang.adapters.mturk.DSL._
Specifically, we're telling Scala that we want to use AutoMan to manage Amazon Mechanical Turk (
mturk) jobs using its MTurk domain-specific language (DSL).We also need to create the main class for our application. Scala makes this a little easier than Java. Paste the following into your editor below your
import statement.object MyFirstAutoManApp extends App {
println("Hello world!")
}
An
object in Scala is just a special kind of class that does not need to be instantiated. It is also sometimes referred to as a singleton class, since you can only have one of them.extends App tells Scala to augment the MyFirstAutoManApp with a public and static main method that takes a String[]. Any code you write inside the class will be interpreted as being a main method implementation. Arguments are available via a variable called args. If you come from Java, this is a lightweight version of the public static void main(String[] args) which you have probably used many times.The code above is a complete Scala hello-world app. As a little sanity check, before we dig into AutoMan, let's just try it out. Run the following in your terminal:
$ sbt run
The first time you run
sbt, it will likely download large numbers of libraries, printing huge amounts of diagnostic information along the way. Do not be alarmed. Subsequent runs should produce substantially less output.You should see output that looks a bit like this:
[info] welcome to sbt 1.3.13 (N/A Java 12.0.2)
[info] loading project definition from /home/dbarowy/my-first-automan-app/project
[info] loading settings for project my-first-automan-app from build.sbt ...
[info] set current project to my-first-automan-app (in build file:/home/dbarowy/my-first-automan-app/)
[info] Compiling 1 Scala source to /home/dbarowy/my-first-automan-app/target/scala-2.12/classes ...
[info] running SimpleRadioProgram
Hello world!
[success] Total time: 4 s, completed Oct 24, 2020, 3:34:12 PM
Notice, buried in all that output, that your program printed out
Hello world!. If you see Hello world! in your output, move on to the next step. If not, read the sbt output carefully to diagnose and fix the problem.Paste the following into your
MyFirstAutoManApp class.implicit val a = mturk (
access_key_id = args(0),
secret_access_key = args(1),
sandbox_mode = true
)
The code above defines a variable
a that stores an instance of mturk. AutoMan needs what we call a platform adapter in order to know which service to connect to. In this case, we are connecting to Mechanical Turk.When you run this code, you will need to supply your MTurk access key and secret key on the command line.
args is an argument array, and args(0) is the first element of that array (if you come from Java, note that arrays in Scala use () instead of []). These are the same credentials you downloaded in an earlier step of this tutorial.Never embed your
access_key_id or your secret_access_key in your source code! Doing so makes it easy to accidentally push your code to a public site like GitHub where they can be stolen and abused.Mechanical Turk comes equipped with a sandbox mode. The sandbox lets you run test code without worrying about real MTurk workers or real money. When
sandbox_mode is set true in the mturk adapter, you will run in sandbox mode.Be sure that you really want to run your program before setting
sandbox_mode to false. It will run real jobs and it will spend real money!Paste the following code below your platform adapter code.
def which_one() = radio (
budget = 5.00,
text = "Which one of these does not belong?",
options = (
choice('oscar, "Oscar the Grouch", "http://tinyurl.com/qfwlx56"),
choice('kermit, "Kermit the Frog", "http://tinyurl.com/nuwyz3u"),
choice('spongebob, "Spongebob Squarepants", "http://tinyurl.com/oj6wzx6"),
choice('cookiemonster, "Cookie Monster", "http://tinyurl.com/otb6thl"),
choice('thecount, "The Count", "http://tinyurl.com/nfdbyxa")
)
)
The above defines a human function called
which_one that takes no arguments. It is important to note that which_one is just an ordinary function in Scala, although it does behave in some special ways that we will describe in the next section.This function creates a "radio button question" on Mechanical Turk by calling the
radio constructor. "Radio button questions" allow MTurk users to select one of n options.AutoMan has constructors for numerous question types. Refer to the AutoMan API Reference for more information.
The constructor above has many parameters which are set to "sane defaults," so that you do not need to specify a task in great detail. These defaults are designed to minimize your surprise.
The key elements in the question function above are:
budget: This parameter specifies the maximum amount of money AutoMan will spend on this task. AutoMan always tries to spend less. If the cost of a task exceeds the budget you supply, AutoMan will shut down the task and return a "low-confidence answer."text: This parameter supplies the text of the question. You describe what you want workers to do here.options: This parameter supplies the valid options. Since this is a radio button question, each option will produce a radio button. Thechoiceconstructor takes three parameters:- A
labelof typeSymbol. You can think of aSymbolas a special string designed for easy comparison. This parameter is not visible to MTurk workers. - A
name, which is visible to MTurk workers.
confidence(not shown): This parameter stands for the statistical confidence level and is a floating-point number between0and1(exclusive). A number approaching zero tells AutoMan that virtually any answer is fine. A number approaching one tells AutoMan that you want to be very certain that it is correct. Although theconfidenceparameter is not shown above, it is set to the default of0.95, which is something of a standard threshold across empirical science.
Now that you have a human function defined, you can call it. First, paste the following AutoMan block below your function definition.
automan(a) {
// your code will go here
}
We are going to call our function inside that block. The purpose of an AutoMan block is to delineate when you are done using MTurk. AutoMan needs this information so that it knows you are ready to shut down your program.
Failing to tell AutoMan to shutdown will cause your program to hang.
Now we can call
which_one() inside our AutoMan block.automan(a) {
println("Answer is: " + which_one())
}
Here is the complete source code for our first AutoMan program (with the hello-world bit removed). You can find a copy of this program in the sample applications directory of AutoMan's GitHub repository, along with many other examples.
import org.automanlang.adapters.mturk.DSL._
object MyFirstAutoManApp extends App {
implicit val a = mturk (
access_key_id = args(0),
secret_access_key = args(1),
sandbox_mode = true
)
def which_one() = radio (
budget = 5.00,
text = "Which one of these does not belong?",
options = (
choice('oscar, "Oscar the Grouch", "https://tinyurl.com/y2nf2h76"),
choice('kermit, "Kermit the Frog", "https://tinyurl.com/yxh2emmr"),
choice('spongebob, "Spongebob Squarepants", "https://tinyurl.com/y3uv2oew"),
choice('cookiemonster, "Cookie Monster", "https://tinyurl.com/y68x9zvx"),
choice('thecount, "The Count", "https://tinyurl.com/y6na5a8a")
)
)
automan(a) {
println("Answer is: " + which_one())
}
}
Notice that the above program does not store any private access keys in the source text! Instead, they must be passed in using command line arguments. For additional examples with more sophisticated command line parsing, see the sample applications directory.
Back on the command-line, we can now run our app with:
$ sbt "run <your access key> <your secret key>"
Assuming you have not changed the
mturk initializer, this program will post jobs to the MTurk sandbox. When you run this program in the sandbox, no work will get done, because only the live production site has active workers. You must simulate the job yourself. We describe the process of simulating a job in the section titled Pro Tip: Use the MTurk Sandbox.After posted jobs are completed, somewhere in AutoMan's voluminous output, you should see something like:
Answer is: 'spongebob
If you want to see AutoMan's quality control algorithm in action, use the sandbox to supply worker responses that disagree. You should observe that after AutoMan obtains all of the responses for a given round, it will decide whether disagreement is strong enough to warrant asking the crowd for more responses.
The program above works, but it's a little simplistic. Without some special configuration, MTurk will not let you post tasks with fewer than 10 HITs. For an easy task like this, which may only need a few responses, this program is a little inefficient. Also, if something goes wrong, this program hides that fact from you.
We provide additional example programs that work around these issues in the
apps directory of our GitHub repository. An efficient, error-handling program that defines essentially the same task can be found here.If something goes wrong as your job runs, be sure to read AutoMan's (voluminous) output. AutoMan provides a great deal of detail to make diagnosing problems easy.