radio
checkbox
freetext
estimate
radios
radio
, except that it returns the entire distribution.checkboxes
checkbox
, except that it returns the entire distribution.freetexts
freetext
, except that it returns the entire distribution.def
) to pass the imgUrl
parameter through to the estimate
constructor. See our sample apps for additional examples.Outcome
. Although you can call toString
on such return values to obtain a simple, printable string, you should probably pattern-match on the result value returned by calling answer
(or answers
, depending on the question) on the returned Outcome
object. Each question type has a different set of possible return values. We describe them in the next section.answer
(or answers
, depending on the question type) will block execution until the task is done running, which may be a substantial amount of time. Be sure that you want blocking behavior when you call answer
.toString
method for Outcome
calls answer
internally, which means that it blocks!mturk
DSL here.options
are the selection options seen by the user, along with optional images. Options can be created using one of the following choice
constructors:choice(key: Symbol, text: String)
or choice(key: Symbol, text: String, image_url: String)
key
denotes a stable identifier for a choice (e.g., kermit
) not shown to the worker, text
is the text label shown to the worker, and image_url
is a url of an image shown beside the text label.text
is the text of the question shown in a HIT
and, by default, also as the task title. You can override the title by setting the title
parameter.Answer[Symbol]
: An object that represents a selected radio button, where each possible Symbol
was defined with the key
parameter of the choice
constructors described above. This object has the following fields:value
: the answer (Symbol
).cost
: the total cost (BigDecimal
)confidence
: the final confidence value (Double
).distribution
: raw sample responses (Array[Response[Symbol]]
)LowConfidenceAnswer
, which has the same fields as Answer
but which indicates that a quality-controlled response has a confidence
lower than the desired threshold.OverBudgetAnswer
, which indicates that a specified task cannot run at all due to insufficient funds. This object has the following fields:need
: the funds needed to start a job (BigDecimal
)have
: the funds at hand (BigDecimal
)NoAnswer
, which indicates that an unexpected runtime error occurred.options
are the selection options seen by the user, along with optional images. Options can be created using one of the following choice
constructors:choice(key: Symbol, text: String)
or choice(key: Symbol, text: String, image_url: String)
key
denotes a stable identifier for a choice (e.g., kermit
) not shown to the worker, text
is the text label shown to the worker, and image_url
is a url of an image shown beside the text label.text
is the text of the question shown in a HIT
and, by default, also as the task title. You can override the title by setting the title
parameter.Answer[Set[Symbol]]
: An object that represents a set of selected checkboxes, where each Symbol
was defined with the key
parameter of the choice
constructors described above. This object has the following fields:value
: the answer (Set[Symbol]
).cost
: the total cost (BigDecimal
)confidence
: the final confidence value (Double
).distribution
: raw sample responses (Array[Response[Set[Symbol]]]
)LowConfidenceAnswer
, which has the same fields as Answer
but which indicates that a quality-controlled response has a confidence
lower than the desired threshold.OverBudgetAnswer
, which indicates that a specified task cannot run at all due to insufficient funds. This object has the following fields:need
: the funds needed to start a job (BigDecimal
)have
: the funds at hand (BigDecimal
)NoAnswer
, which indicates that an unexpected runtime error occurred.pattern
is a COBOL-style picture clause pattern that states what inputs are valid. AutoMan uses this pattern to perform probability calculations. A
matches an alphabetic character, B
matches an optional alphabetic character, X
matches an alphanumeric character, Y
matches an optional alphanumeric character, 9
matches a numeric character, and 0
matches an optional numeric character. For example, a telephone number recognition application might use the pattern 09999999999
.text
is the text of the question shown in a HIT
and, by default, also as the task title. You can override the title by setting the title
parameter.freetext
-specific:allow_empty_pattern
means that the empty string is a valid worker response.
default: false
before_filter
is not currently used.pattern_error_text
is a helpful message that is displayed to the user when their input does not match a pattern. It is not mandatory but it is highly recommended that you use this setting.pattern_error_text
for freetext
questions as the default MTurk help message is not helpful. This parameter gives you an opportunity to provide an error explanation in non-technical terms.Answer[String]
: An object that represents a response string. This object has the following fields:value
: the answer (String
).cost
: the total cost (BigDecimal
)confidence
: the final confidence value (Double
).distribution
: raw sample responses (Array[Response[String]]
)LowConfidenceAnswer
, which has the same fields as Answer
but which indicates that a quality-controlled response has a confidence
lower than the desired threshold.OverBudgetAnswer
, which indicates that a specified task cannot run at all due to insufficient funds. This object has the following fields:need
: the funds needed to start a job (BigDecimal
)have
: the funds at hand (BigDecimal
)NoAnswer
, which indicates that an unexpected runtime error occurred.confidence_interval
lets you denote the confidence interval of an estimate. The options are:UnconstrainedCI()
which will only even perform one round of tasks using the default sample size, returning the SymmetricCI(err: Double)
which returns the err
with confidence
level confidence.AsymmetricCI(lerr: Double, herr: Double)
which returns the -lerrr
and +herr
with confidence
level confidence.text
is the text of the question shown in a HIT
and, by default, also as the task title. You can override the title by setting the title
parameter.Estimate
: An object that represents a "best estimate". This object has the following fields:value
: the estiamte (Double
).low
: the low bound of a confidence interval's estimate (Double
).high
: the high bound of a confidence interval's estimate (Double
).cost
: the total cost (BigDecimal
)confidence
: the final confidence value (Double
).distribution
: raw sample responses (Array[Response[Double]]
)LowConfidenceEstimate
, which has the same fields as Estimate
but which indicates that a quality-controlled response has a confidence
lower than the desired threshold.OverBudgetEstimate
, which indicates that a specified task cannot run at all due to insufficient funds. This object has the following fields:need
: the funds needed to start a job (BigDecimal
)have
: the funds at hand (BigDecimal
)NoEstimate
, which indicates that an unexpected runtime error occurred.checkboxes
constructor here, but freetexts
and radios
are similar. There is also a buggy multiestimate
constructor that should probably not be used at the moment.sample_size
is the size of the sample.options
are the selection options seen by the user, along with optional images. Options can be created using one of the following choice
constructors:choice(key: Symbol, text: String)
or choice(key: Symbol, text: String, image_url: String)
key
denotes a stable identifier for a choice (e.g., kermit
) not shown to the worker, text
is the text label shown to the worker, and image_url
is a url of an image shown beside the text label.text
is the text of the question shown in a HIT
and, by default, also as the task title. You can override the title by setting the title
parameter.budget
is the total amount of money to be spent by a given human question function call. Note that this means that each function call has its own budget.
default: $5.00dont_reject
, when set to true
, will always accept completed assignments and pay workers for their work. This is useful when work is difficult and errors are likely, or when you just don't want to deal with the hassle of reputation management.
default: false
dry_run
, when set to true
, will not actually post jobs on MTurk.
default: false
image_alt_text
adds an HTML ALT
annotation to the IMG
tag created by the image_url
parameter.
default: none (null
)image_url
adds an image to a question. Such images should be hosted someplace publically-accessible, such as Amazon S3 or a personal website.
default: none (null
)initial_worker_timeout_in_s
is the amount of time permitted to a worker in the initial round of tasks. Note that the actual time permitted depends on the number of rounds and is determined by the quality control policy. The default policy uses the formula initial_worker_timeout_in_s
, minimum_spawn_policy
states what the smallest number of assignments for a given HIT
are on MTurk. This is necessary because MTurk has two totally boneheaded policies:UserDefinableSpawnPolicy(n)
where n
is the number you want.
default: 10
note: I am actively unhappy about this and am thinking of ways to simplify it. Suggestions welcome.mock_answers
sets AutoMan to be used in mock mode for testing purposes. This is used interally by AutoMan for testing. You should not change this.
default: null
pay_all_on_failure
controls whether workers are paid when a task runs out of money. Setting this to false
means that workers will not be paid when an OverBudget
result is returned, which generally makes workers unhappy.
default: true
question_timeout_multiplier
controls how much time a HIT exists on MTurk before it is timed out. Note that this is a distinct timeout from the amount of time a worker is given to complete a task, which is controlled by the initial_worker_timeout_in_s
parameter. A HIT's total time is determined by the formula initial_worker_timeout_in_s
, question_timeout_multiplier
.wage
controls the base wage for a worker. The actual reward paid depends on how much time a worker is given to do a task. The default policy uses a maximum likelihood estimate of the probability that a task is accepted in order to compute a wage that disincentivzes wage gaming behavior. It is complicated enough that if you want to know its inner workings, you should read our 2016 CACM article. Generally you should think of the reward as "probably doubling."
default: the U.S. Federal Minimum wage, or $7.25/houra
is an initialized AutoMan platform adapter. Typically this will be an implicit
variable that you return from a platform initializer expression like mturk
. When marked implicit
, you do not need to pass the parameter yourself; Scala will find it in the environment and pass it, simplifying human function calls. AutoMan needs this information in order to bind a human function call to a given crowdsourcing platform.database_path = "/path/to/your/database"
. The format of the database is H2.apps
directory. Apps can also be built using pack
. E.g.,apps/[the app]/target/pack/bin/
.