Pro Tip: Use the MTurk Sandbox

Although AutoMan is designed to make programming crowdsourcing jobs easier and more reliable, simulating MTurk jobs during development is slightly more complicated. The reason is that AutoMan utilizes the MTurk Qualification system to ensure that a sample of completed assignments is i.i.d.

Multiple MTurk sandbox accounts = multiple email addresses

From a practical perspective, the presence of AutoMan's quality control algorithm implies that you will need multiple MTurk sandbox accounts to simulate work in the sandbox. You will also need a unique email address for every sandbox account that you set up. We recommend creating at least five sandbox accounts; for estimates, you will need many more (the minimum sample size is 12 for estimates).

An easy way to obtain unique email addresses that all forward to the same mailbox is to use GMail's task-specific email address feature.

Debugging with multiple accounts = multiple browser sessions

While debugging, we find it useful to stay logged into multiple MTurk sandbox accounts simultaneously. Until recently, doing so was not supported by default in browsers. Fortunately, the Firefox web browser now ships with an addon called Multi-Account Containers that makes managing multiple sessions easy.

After you have set up containers, click and hold on the + sign on the browser's toolbar: it will give you the option of creating a new tab in a new session. Signing in to a website in that tab will limit session state (e.g., cookies) to that one tab, which means that you can open another tab using another container and login using a different account. This feature is a life saver for us!