IdolHands.com :: Days in the Life of an Alpha Geek
Game Theory and Google Image Labeler
I didn't completely waste my morning on Google Image Labeler or anything. (Found via Boing Boing... thanks for ruining my morning, guys.)
But after spending some time on it, I reflected on what they were trying to accomplish-- motivating the masses to improve their image metadata-- and what they were getting, and I realized that from a game theory perspective, there are serious flaws in Image Labeler.
How It Works
After clicking a button to start, you are paired up with a random partner for 90 seconds. You and your partner are shown an image, and you start entering labels for that image. You'll also see a list of "off limits" labels that seem to represent labels that have already been applied to the image in previous rounds by other participants. You don't see the labels that your partner has entered, but you are told the number of labels that they have put in. Whenever you and your partner enter a matching label, or if you mutually decide to 'pass', you move on to the next image.
At the end of the round, you get to see the Google user name of your partner, and you are presented with a list of the images and labels as well as a link to the sites that the images came from:
The brilliant thing about this is that it's set up as a game. You get 100 points for each label that you and your partner manage to agree on, and you also see your highest ranking partnership:
This brings a competitive spirit to bear on what is essentially a volunteer effort to improve Google's image search metadata.
Problems with Image Labeler:
Aside from the fact that the images that are displayed are often too small to actually make out, and that they are displayed without any context, the more serious flaw in Image Labeler is the conflict between the mechanism for attracting participants and the objective of their participation.
The structure of the game rewards you for guessing what your opponent will enter as a keyword, not for entering the best keyword. The result is that as you play, you end learning that the winning strategy involves entering labels that represent the "lowest common denominator", because these keywords have the higher probability of having been entered by your partner. This means that Google is going to end up with a large number of uselessly short keywords (e.g. "man") rather than truly descriptive keywords (e.g. "Tony Blair"). The time pressure imposed by 90-second rounds only increases the odds of low-quality keywords being applied.
Unless this conflict is addressed prior to keywords from Image Labeler being applied to Google's database, it could contribute to a serious "metadata poisoning" problem for Google, where they have large quantities of low-quality keywords rather than small quantities of high-quality keywords.
Suggestions for Improvement:
- Pair up high-scoring participants for better results.
- Have follow-up rounds where partners (or other participants) can see their unmatched labels and apply them.
- Provide some sort of context for the image that is presented. Some of the images were illegible scanned documents, that at the end of the game you would discover were pictures of handwritten notes by Freud.
- Hire someone with a background in game theory to redesign the process to produce higher quality results.
Google Image Labeler is a fun time sink, but I'm not sure that it's actually going to provide the value that it's intended to without some serious retooling.
I have to go, um, do something now. Something that has nothing to do with improving my ranking (currently #706 overall, #61 today). No, of course not.
Comments