This document is replaced by the 2014 FAQ.
This document contains answers to frequently asked questions about Turkopticon. Roughly, it covers: current meanings and policies; current "how-to" or technical questions; possible future features; and the people and motivations behind Turkopticon.
This document was last updated 30 December 2013.
1. Using Turkopticon: meanings and policies
1.1. What are the requester "attributes" ("comm", "pay", "fair", and "fast")?
1.2. How can I write a good review?
1.3. What are appropriate uses of flags and comments?
1.4. This person is reviewing wrongly. Can you stop them?
1.5. I am a requester. Can you delete or censor a review of me?
1.6. I am a requester and I changed my name on Mechanical Turk. Can you change my name on Turkopticon?
2. Using Turkopticon: technical
2.1. How do I post a review?
2.2. Can I delete my review?
2.3. My display name is based on my email address (e.g., s...@g...). How do I change it?
2.4. Can I delete my account?
2.5. How does flagging work?
3. Possible future features: smaller things
3.1. Can you send me email alerts when someone flags or comments on my review?
3.2. I am a requester. Can you send me email alerts when someone reviews me?
3.3. Deleting reviews
3.4. Deleting comments
3.5. Deleting your account
3.6. Backend / other
4. Possible future features: bigger things
4.1. Moderation is broken. Will you fix it?
4.2. Will you change the "generosity" attribute to something more specific, like estimated hourly wage?
4.3. Will you accept reviews for specific HITs?
5. Turkopticon people and background
5.1. Who runs Turkopticon?
5.2. Who has been involved with Turkopticon in the past?
5.3. Is Turkopticon associated with Amazon?
5.4. Why was Turkopticon built?
5.5. Do you make money from Turkopticon?
5.6. Can I give a donation?
The "attributes" used to describe requesters answer the following questions:
Give the ratings you feel best describe your experience.
If one of the fields is not relevant, leave it blank.
For example, if you didn't need to communicate with the requester, leave that field blank.
Don't include or link to profanity, racial slurs, or ethnic slurs.
Don't personally insult or attack requesters or other workers. Calling a requester "cheap" is okay, but anything stronger is subject to being hidden.
If you're a requester, don't review yourself. Comment instead.
Reviews that are profane or insulting or appear to be requester self-reviews may be hidden.
The most up-to-date version of these guidelines will always be posted on the review page (Turkopticon login required).
The review is a way for workers to describe their work experience.
The worker has a right to their review.
Flag a review if and only if:
The review contains profanity, racial/ethnic slurs that weren't automatically filtered, or incitements to violence, or
The review personally insults or attacks requesters or other workers, or
You have a good reason to believe the review was written by the requester.
If you think the review was written by the requester, you must explain why in your comment. "Bogus review" is not a useful explanation for moderators.
Do not flag a review just because you disagree with it, or because you had a different experience. Two different workers might have completely different experiences with the same requester. A review might be suspicious, but that doesn't mean it was written by the requester or a shill. Requesters do sneakily self-review, but it's rare.
Flags not following these guidelines will be ignored by the moderators.
If you're not sure if you should flag a review, don't.
The most up-to-date version of these guidelines will always appear right above the text box for flagging reviews.
At present our only moderation tool is flagging. If you have an issue with a review, you may flag it. But please note that moderators and administrators are likely to ignore flags that do not follow the flagging guidelines (see above, Section 1.3, "What are appropriate uses of flags and comments?").
At the same time, we realize not everybody agrees on the "right" way to review. This has caused tensions and frustrations in the past. Addressing this issue fully is likely to require a major overhaul of the moderation system. Unfortunately this is not likely to happen soon. For more on this topic, see below, Section 4.1, "Moderation is broken. Will you fix it?"
Delete? No.
Censor? Almost certainly no.
With the exception of technical errors (e.g., a review posted twice), we do not delete reviews, even at the request of the author. In the case of doubly posted reviews, we delete the duplicate only at the request of the author. (As of 5 August 2013, in almost five years of operation and out of over 88,000 reviews, we have done this exactly once.)
In the future, we may allow review authors to delete their reviews (see below, Section 2.2, "Can I delete my review?", and Section 3.3, "Deleting reviews").
In general, requesters are encouraged to comment on reviews rather than flag. But if a review clearly falls within the flagging guidelines (see above, Section 1.3, "What are appropriate uses of flags and comments?"), feel free to flag it.
We have had several cases of particularly angry workers posting contact information about requesters. At present it is our policy not to censor this information if it is discoverable via a simple Google search. As of 6 August 2013, we have had exactly one case (Turkopticon login required) in which we agreed to censor the information. This case was fairly extreme, involving harassment of the requester at their personal email address (which was not discoverable via simple Google search of the requester name) for an extended period of time. If your case is similar, we may consider censoring the contact information.
Yes. Email us. Normally we will change your name on Turkopticon to
Your New Name (was Your Old Name)
In exceptional cases, we may remove the old name entirely. Generally we try to avoid doing this, as it could cause significant confusion. Factors that make it more likely for us to do this include:
Even if we remove the old name entirely, we will keep a private record of it in our database.
The easiest way to post a review is to click the link "Report your experience with this requester" in the add-on drop-down.
However, you can also review any requester by going to turkopticon.differenceengines.com/report (Turkopticon login required). You'll need to know their Mechanical Turk requester ID, though (e.g., A1BC2DE34FGHJ5). If you don't include a Mechanical Turk requester ID, your review will not help anyone.
For now, no.
If you managed to post a review of the same requester twice (i.e., there was a technical error), email us and we will delete the duplicate, as long as nobody else has flagged or commented on it yet.
The main reason we don't allow deletion of reviews is accountability. The world around Mechanical Turk, including Turkopticon, is generally one of mistrust and suspicion. If we allowed untraceable deletion of reviews, people might worry that we were deleting reviews at the request of requesters. We don't do that (see above, Section 1.5, "I am a requester. Can you delete or censor a review of me?"), and we don't want to jeopardize the usefulness of Turkopticon by giving anyone any reason to imagine that we might. So we don't delete any reviews, even at the request of the author. (Duplicates are the only exception.)
A second reason is that if we were to delete a review that had comments or flags on it, those comments or flags would disappear. This could be confusing, and confusion causes frustration and suspicion, which we try to avoid.
Sometimes people post a review they later regret posting. This often happens when other people flag the review or leave mean comments. The review author often wants to delete the review in order to delete the comments or flags. In our view this is unfair to the comment authors or flaggers, even if they said mean things. We do not have a policy of censoring something if it is merely mean—only if it is profane. In the long run, usually the person who said the mean thing is more embarrassed than the person they said it to.
All that said, in the future, we may allow review authors to delete their own reviews (see below, Section 3.3, "Deleting reviews"). If we do this, we will find a way to keep a record that the review existed, and keep comments and flags accessible.
For now, if you have the urge to delete a review you posted, the best thing to do is edit it instead, or leave a comment under it.
In general, once your display name is set, it is supposed to stay as it is. But if there was a mistake and somehow your display name was set to the default display name (which is based on the email address you used to sign up for Turkopticon), email us and we will "unset" it, after which you can pick a different display name.
If you already have a "real" display name, we will not unset it. The point of this is to create a little bit of accountability.
For now, the best we can do is "close" your account.
We realize this can be frustrating. "I own all of my reviews!", you might say, "I have a right to delete them!" In principle, we agree. Unfortunately there are technical issues. The main one is that if we actually deleted your account—deleted the entry in the Turkopticon database that represents your account—we would have a bunch of reviews left over associated with a nonexistent account. When the web site tried to display those reviews, it would choke. "So delete them!", you might say. If nobody else had commented on or flagged any of your reviews, we could do that without any technical issues—although we would still have the accountability issues discussed in Section 2.2, "Can I delete my review?". But this is not usually the case. And if we delete a review that someone else has commented on, we are effectively disappearing their comment, which is not fair to them (see above, Section 2.2, "Can I delete my review?", for more on this topic).
So, for now, the best we can do is "close" your account. A closed account cannot be logged into. In the future it is possible we will hide reviews from closed accounts in the same way we hide reviews that violate civility guidelines, although for the moment we see no real need to do this.
As of 5 August 2013, we have not closed many accounts (in fact we have closed exactly one), so our policy on this topic is not very well developed. We don't understand fully why people are motivated to delete their accounts, so we don't know exactly what else we should do when an account is closed. We are open to discussing ways to improve how we handle it.
To close your account, email us. In the future, we may set up a way to do it automatically (see below, Section 3.5, "Deleting your account").
When you flag a review, it appears on the list of flagged reviews. Moderators go over this list occasionally. If a moderator agrees with your flag (which is much more likely if your flag is in accordance with the flagging guidelines), they will also flag the review. Once a review has at least two flags, at least one of which is from a moderator, it will be hidden. This means:
It won't come up in the main list of reviews for the requester (although it can still be seen by clicking the link to hidden reviews at the bottom of the review list for that requester; see for example here), and
The scores in that review won't be included in the averages displayed for the requester in the browser add-on.
If somebody flagged your review spuriously, the best thing to do is usually to ignore them.
We've had this question a couple of times. This shouldn't be too hard.
This shouldn't be too hard. If we implement this, we will probably just let anyone sign up for email notifications about new reviews of a particular requester, because verifying that you are who you say you are is hard over the internet.
In the future, we may allow review authors to delete their own reviews. There are a number of technical and social issues with this, though, which is why we don't allow it right now (see above, Section 1.5, "I am a requester. Can you delete or censor a review of me?" and Section 2.2, "Can I delete my review?" for more on this topic).
In the future, we may allow comment authors to delete their own comments. As with reviews, however, we don't currently allow it for accountability reasons (see above, Section 1.5, "I am a requester. Can you delete or censor a review of me?" and Section 2.2, "Can I delete my review?" for more on this topic).
In the future, we may allow full deletion of accounts rather than simply closing accounts, as we do right now. At present however we don't allow it for accountability reasons (see above, Section 1.5, "I am a requester. Can you delete or censor a review of me?", Section 2.2, "Can I delete my review?", and Section 2.4, "Can I delete my account?" for more on this topic).
Here are some other things people have asked us about, or we have considered doing. These also may happen in the future, if we have time.
We know that some veteran Turkopticon users are unhappy about new people showing up who have different ideas about how to review. The flagging system used to cover some of these cases, but it no longer does, because some people don't like to have their reviews hidden when they feel like they have done their best to post a useful, informative review.
Unfortunately for us, we see the merit in both views.
In our view the issue is not so much about who is right and who is wrong, but, assuming that everyone is more or less on the same team, about communicating about norms and standards. What does a 1 mean? What does a 5 mean? When is it okay to post a review without doing the HIT? and so on. These are important issues that need to be discussed among the community, not decided by administrators or even moderators. It is possible that the current review format is not even adequate—but if this is so, we don't know what exactly would be better.
So, unfortunately, the short answer, for now, is "no." We won't fix moderation because we don't know how, and because the problem may be deeper than the moderation system. But it does seem like broader discussion among Turkers about some of these issues would be helpful. We are open to ideas about how to support such a discussion; if you are excited about this, please email us.
We like this idea.
At the same time, making the "generosity" attribute more useful is probably not as simple as setting specific values for the generosity attribute, because people have different ideas about what is good pay. As a result, we want to have a broad discussion about this before we make any significant changes.
This is probably a good idea, because sometimes requesters have some good HITs and some bad (broken, cheap, etc.) ones. Unfortunately Turkopticon is not set up for this right now. It could be in the future, but it would be a fairly significant change. We have no plans to do this right now.
With the help of moderators (in alphabetical order) Anne M, Honuagal, and taintturk, Turkopticon is maintained by Lilly Irani and Six Silberman. Lilly is an assistant professor at the University of California, San Diego. Six is a graduate student at the University of California, Irvine.
Note: Answering this question involves two things. First, acknowledging people who have helped us and tried to help us. Second, acknowledging that some of those efforts have ended badly or have raised suspicions among Turkers. It's all here, with the exception of one omitted name, because he's paid enough for the mistakes we made together. If you have questions or concerns about any of this, please feel free to email us.
Turkopticon was motivated by the responses to our 2008 Workers' Bill of Rights survey; the first thanks have to go to the workers who responded to that survey.
Second, in late 2008 and early 2009, Dolores Labs helped us collect initial reviews and spread the word about Turkopticon by linking to us at the bottom of some of their HITs. They wanted to support Turkopticon because they liked the idea of a website that let workers review requesters. Awkwardly for us, Dolores Labs eventually changed its name to CrowdFlower, and is now famous for being a huge requester of dubious quality and low pay—and for being sued for not paying workers minimum wage.
In 2009 and 2010, Dahn Tamir of Techlist helped us with some code maintenance.
In 2010 and 2011, we worked with an economics professor who wanted to give Turkers exact wage data about HITs. We added code to collect this data to the extension. This freaked people out and was taken out.
No. It is not and it has never been.
Interestingly, however, Amazon appears to have bought the domain turkopticon.com. We don't know why they did this, or even if the information on that page is correct.
Turkopticon started in 2008 as a class project to raise awareness about the unaccountability of Mechanical Turk requesters. We saw requesters and researchers celebrating how cheaply they could get work done on Mechanical Turk and we felt this was wrong. We didn't understand why Amazon didn't build in a requester reputation system, like Amazon Marketplace and eBay have. Building Turkopticon was our way of making this point. We never expected it to be used by so many people, or to become a fixture in the Turker community.
No. We do not, and never have.
We do not charge for access to Turkopticon, sell ads or information, or have any other income from Turkopticon. We have no plan change this.
This could change, but at present, we have no way to accept donations.