I’ve found some limitations in Mechanical Turk’s API for Create Qualification Type with an auto-graded qualification test. As of this writing in 2015, the limitations are undocumented, and they cause the API request to fail for no apparent reason. In this post I describe the undocumented limits.
When using Amazon’s Mechanical Turk, creating a qualification that involves a qualification test (i.e., a list of questions that determines whether the worker is qualified to accept a HIT) cannot be accomplished from the MTurk web interface. Accomplishing this advanced task requires accessing MTurk using the API. Creating qualifications with automatically graded qualification tests is documented here, here, here, and here.
However, there are some undocumented limitations in this procedure in terms of string lengths. Even requests that are properly formatted XML will fail if these character limits are exceeded. Unfortunately, the manner in which MTurk’s API fails is not the least bit informative; the API simply fails to return anything whatsoever. Even worse, these character limits don’t seem to be documented anywhere in MTurk’s API documentation. I learned from sad experience that when any of these undocumented character length limitations are violated, the API fails to respond for no identifiable reason. Rude.
With a little trial and error, I’ve found the limitations on string length that are missing from the documentation:
- The qualification’s name cannot be more than 80 characters
- The test’s title (i.e., the header that is displayed at top of the test) cannot be more than 128 characters
- The question text cannot be more than 82 characters
- The selection identifier text (i.e., the labels for the radio buttons) cannot be more than 44 characters
By the way, some simple XHTML tags are allowed in questions text (see here), but they are not allowed in the selection identifier text. Putting XHTML in the selection identifier text yields — you guessed it! — the API to fail without any error message. Bad.
– Eric DeRosia