Like a lot of social scientists, I use Amazon’s Mechanical Turk for recruiting participants for some of my experiments. It’s cheap, fast, and the data quality is about the same as what I’ve seen with student subjects and professional online recruiters. In fact, I use Mechanical Turk for a variety of crowdsourcing tasks, such as data entry and proofreading. It’s an amazingly flexible resource, and I’m a big fan.
However, as of this writing in January 2013, Amazon has recently made a change that I need to post a warning about. In essence, Amazon has changed their requester interface to make “master” workers a secret, opt-out option. If your MTurk project isn’t working well, this might be the reason.
This requires some background explanation. Amazon charges a much higher rate if requesters use workers who have been awarded a “master” designation. Amazon has awarded a number of workers this “master” designation, which is supposed to indicate the best workers. On the face of it, the “master” worker thing sounds like it would be good for requesters. However, it has serious problems.
For one thing, I follow the worker community closely (e.g., Turker Nation), and many good, devoted workers have not earned the “master” designation. There is a lot of angst about “masters” in the worker discussion space. Much of the uncertainty among workers is caused by Amazon’s decision to not reveal the requirements for earning the “masters” designation. No one really knows what it means or how many “masters” there are.
As a requester, I’ve tried using “master” workers, and I’ve had terrible luck. Perhaps most importantly, there are far fewer of these people, so projects don’t get done as quickly. They want higher rewards to entice their participation; that’s not the end of the world, but it’s certainly something that impacts a project. Amazon takes a much larger fee, and that’s also something that impacts a project. Furthermore, I have found that “master” workers are really flighty. They’re paranoid about having any task rejected, and that’s understandable — if Amazon considers rejection rates in their “master” designation, a rejection might mean the worker is demoted from being a “master.” I’ve had projects languish while “master” workers passed over them because the workers weren’t absolutely certain they could avoid rejections. I’ve even had big-time rewards like $15 for a two-hour task that weren’t claimed by workers. As soon as I released the HIT to “regular” MTurk workers, that task was completed quickly and accurately. Personally, I’ve had only terrible luck with “master” HITs.
With all that as background, here is the new problem I’m highlighting. Amazon has recently made a change to their requester interface that makes “master” workers not only an opt-out option, but an option that is hidden from normal view. When a requester completes the “New Project Create” form, it’s not at all obvious that “master” workers are now the default. In the screen shot below, I’ve noted the four places that must be clicked to drill down through the interface to reveal the “master” requirement and remove it.
Frankly, I view Amazon’s change in their interface as unseemly trickery. I assume Amazon did this because their margins are much higher for master HITs. Overall, I’m cheering for Amazon, but not even notifying requesters that they’re asking for masters?!? That’s not impressive. The other day, I was creating a new task from scratch, and I only caught this issue a few screens later when I was looking at the pricing and I saw a surprise surcharge for “master” workers.
I’m sure “master” workers are great for some types of HITs. However, it would be a shame if requesters are tricked by Amazon into unknowingly using “master” workers.
– Eric DeRosia