The Dirty Little Secret About the "Wisdom of the Crowds" - There is No Crowd
Written by Sarah Perez / September 17, 2009 7:58 AM / 31 Comments
Recent research by Carnegie Mellon University (CMU) professor Vassilis Kostakos pokes a big hole in the prevailing wisdom that the "wisdom of crowds" is a trustworthy force on today's web. His research focused on studying the voting patterns across several sites featuring user-generated reviews including Amazon, IMDb, and BookCrossing. The findings showed that a small group of users accounted for a large number of ratings. In other words, as many have already begun to suspect, small but powerful groups can easily distort what the "crowd" really thinks, leading online reviews to often end up appearing extremely positive or extremely negative.
Small Groups, Big Impact
To conduct the research, Kostakos worked with a large sample of online ratings. As MIT's Technology Review reports, the researcher and his team studied hundreds of thousands of items and millions of votes across all three sites. In each and every case, they discovered that small numbers of users accounted for the largest number of ratings. For example, on Amazon, only 5% of active Amazon users ever cast votes on more than 10 products but a small handful of users voted on hundreds of items. Said Kostakos, "if you have two or three people voting 500 times, the results may not be representative of the community overall."
This is hardly the first time that the so-called "wisdom of the crowds" has been called into question. The term, which implies that a diverse collection of individuals makes more accurate decisions and predications than individuals or even experts, has been used in the past to describe how everything from Wikipedia to user-generated news sites like Digg.com offer better services than anything created by a smaller group could do.
Of course, we now know that simply isn't true. For one thing, Wikipedia isn't written and edited by the "crowd" at all. In fact, 1% of Wikipedia users are responsible for half of the site's edits. Even Wikipedia's founder, Jimmy Wales, has been quoted as saying that the site is really written by a community, "a dedicated group of a few hundred volunteers."
And as for Digg.com, a site whose algorithm is constantly tweaked in attempts to democratize the votes of its users, it still remains a place where a handful of power users can make or break getting a news item to the site's front page.
Attempts to Address the Issue
It's not surprising then to discover that, when it comes to review sites, it's again small groups that are in control there too. Some sites, including Amazon, attempt to address this discrepancy by allowing users to vote on the helpfulness of reviews - a much easier process than having to write a review yourself. Also, local business finder and recommendations site Yelp implemented ways for business owners to respond to what they feel are inaccurate reviews by way of an owner comments feature. Unfortunately, despite these efforts, the small groups still remain in control of these so-called "popular opinion" features.
According to the article, another professor at CMU, Niki Kittur, suggested that sites create new tools for transparency. For example, there should be an easy way to see a summary of a user's contributions which would quickly reveal any bias. He also suggested removing overly positive and negative reviews.
Earlier this year, we looked at another user-generated review site which attacked this problem from another angle. Lunch.com, a new Yelp competitor, uses something they call their "Similarity Network" which matches you to other site users who share your interests. That way, instead of looking at a list of reviews which could originate from anyone with an agenda or axe to grind, you're focused on reviews from others like you.
Still, there is yet to be a perfect solution to the problem. Perhaps it's time we give up the idea that the "wisdom of the crowds" was ever a driving force behind any socialized, user-generated anything and realize that, just like in life, there will always be active participants as well as the passive passerbys.
Posted inNews, Social Networks, Social Web, Trends and tagged with
groups, ratings, research, reviews, wisdom of crowds
Print This Digg Share Related Entries
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
I think your headline is misleading and Vassilis Kostakos should read the book before poking holes.
Surowiecki is very clear about the conditions necessary for a wise crowd to prevail and those conditions are:
1. Diversity of opinion
2. Independence
3. Decentralization
4. AggregationIf your crowd possesses those qualities then it is wise and then it will be better at making decisions under Surowiecki's paradigm. The crowds used in the research (and the crowd in general) doesn't possess those qualities and therefore is an unfit data set. We should be trying to create the ideal crowd before we can obtain superlative results and not try to get good results from any random crowd.
Posted by: Muhammad
| September 17, 2009 8:18 AM
yep there is no crowd listen to yourself wikipedia and digg is manipulated by their owners itself
http://Evolvhealthy.com
Posted by: evolvhealth | September 17, 2009 8:20 AM
You mean to say, of course, that Wikipedia is edited by a minority of unaccountable zealots
Posted by: Bruno | September 17, 2009 8:34 AM
Wisdom of Crowds is a crypto-fascist idea; there is no objective truth, there are no facts, truth is what "the crowd" decides it is. You get these unhealthy echo chambers of "activists" setting the agenda.
This article said it best, over three years ago:
DIGITAL MAOISM
The Hazards of the New Online Collectivism
By Jaron Lanier
Posted by: Peter Verkooijen | September 17, 2009 8:43 AM
There are no small groups without a crowd. This is rudimentary logic. And small groups are only representative of something if they are part of a large crowd. Otherwise, the small group does not actually create any impact. Research conducted by prof. Vassilis Kostakos is valid, but does not truly question the vision of Wisdom of the crowd.
Posted by: Alex | September 17, 2009 8:48 AM
it's all about the niche. It's the selection of the relevant from the crowd of the diverse that really drives wisdom.
Posted by: davidcushman | September 17, 2009 8:54 AM
Hi Sarah,
Nice summary. For those running UGC sites, it's no secret that a small percentage of the user base creates the vast majority of the content.
At RateItAll (www.rateitall.com), we are trying a number of things to improve the trustworthiness of our content:
- We offer a newsfeed format that populates your review stream with a combination of reviews from your friends, and reviews of topics you are interested in. We give you the option of moving out in degrees of network separation (e.g. "show me reviews of friends of friends of friends") with the idea that networks are self policing. Presumably, people you know in real life won't deliberately steer you the wrong way.
- We provide "compatibility tests" that let you find reviewers who most resemble you in outlook on a category by category basic (e.g. music vs. politics). This was the inspiration for Lunch's similarity network.
- We offer a proprietary "confidence score" that helps determine the best product / item in a certain category. This score factors in a number of results - not just rating. See this dog food ranking for an example: http://www.rateitall.com/t-353-dog-food-brands.aspx (Yelp is also aggressive with automated algos to weed out reviews that look shillish)
- We provide total transparency on a reviewer's rating history. Clicking a user's profile name link shows you immediately all of their ratings activity on the site, without making you dig.
- We provide review by review voting.
- We allow commenting on any review - if you shill, you will likely be called out publicly on it by our regulars.
- We encourage people to login using Facebook Connect (real identity is a major factor when talking about trustworthiness of a review - there's a new site called Oyster.com that ONLY will accept reviews from folks logged in via FB Connect)
- As a horizontal rating site, we try and offer something for everyone. The more data and more participants, the more trustworthiness.
- Similarly, we have tried to remove as many barriers to participation without losing identity - for example, we recently launched a tool that lets you post a review and signup just by sending an email to reviews@rateitall.com - the product / place you are reviewing goes in the subject line, and the rating and review goes in the body.
Transparency. The ability to isolate reviews from people you trust in real life. Trust algorithms. People discovery. Reputation. Broad participation. Real identity. Accessibility.
In my opinion, these are the keys to maximizing the trustworthiness of user generated reviews.
Lawrence Coburn
CEO, RateItAll (www.rateitall.com)
Posted by: Lawrence
Sounds like there is a vast conspiracy going on here. I'm sure of it. A small group of impotent, brainless little twit-faced pee-ons are trying to suck the control and credit out of the hands of those greedy, power-hungry, know-it-all experts. Oh yeah. Treachery is afoot.
By the way, I remember hearing somewhere that two’s company, but can’t remember what they call three. It’s right on the tip of my tongue … aww, shucks. I can’t remember.
Nevermind.
Back to your lives, citizens.
Posted by: Eric Wilbanks
Agree with the previous commenter. The situations described in the research do not relate to a crowd as defined in Surowiecki's book.
Posted by: Stephen
Re "Of course, we now know that simply isn't true. For one thing, Wikipedia isn't written and edited by the "crowd" at all" see http://www.aaronsw.com/weblog/whowriteswikipedia
Posted by: Dan Brickley | September 17, 2009 12:26 PM
Muhammad, do you think Digg complies with those 4 points in the book?
1. Diversity of opinion [ummmm]
2. Independence [hmmmm]
3. Decentralization [maybe]
4. Aggregation [check, except it aggregates just a handful of sites, like Ars Technica and a few other ones popular with digg's power users]
Posted by: Richard MacManus
Hi Richard,
I don't think so at all. The problem with most of these sites is that they suffer from information cascades and people participating don't always have 'pure motives.'
From Wikipedia:
There are two key conditions in an information cascade model:
1. Sequential decisions with subsequent actors observing decisions (not information) of previous actors.
2. A limited action space (e.g. an adopt/reject decision).What happens on Digg is that a lot of stories get Dugg because a particular user submitted them, or particular users Dugg them, or they were from a particular domain, or simply because someone asks you to Digg something. Therefore decisions aren't made with merit in mind (or Suriwiecki's four conditions in mind) and instead popularity rules the day.
I can't remember the name of the site but I recall there was a foreign version of Digg that didn't show you who submitted a story, that's a step up. Similarly Reddit doesn't show you how many votes a story has for the first hour. Things like that try to prevent information cascades.
Posted by: Muhammad
Great points Muhammad. That would indeed be interesting, if Digg hid the submitter name!
An interesting application of crowdsourcing are decision markets, see http://en.wikipedia.org/wiki/Prediction_market As a sidenote, it's quite ironical that wikipedia highlights the shortcomings of those markets in their entry - but in their defense they also include the criticism on their own bias in their entry http://en.wikipedia.org/wiki/Wikipedia
I followed prediction markets for the US presidential election and they seemed spot on. There are numerous studies showing that they predicted the results more accurately than any http://www.semanticsincorporated.com/2008/11/90-of-an-obama-win-as-per-prediction-markets.html
Limitations in predictions market are well documented (and include Muhammad's points above), and constrain their practical application to a well-defined number of situation.
Crowdsourcing suffers from the same limitations, which is not a problem, as long as you limit its application correspondingly. The problem occur when you stretch it outside the required constraints and yet present the results as "scientific", i.e. as a good proxy for what the crowd thinks.
That's what professor Vassilis Kostakos's theory ultimately comes down to (or should - I don't know, I haven't read his report). Apps like Digg or Amazon's review are not scientific applications of crowdsourcing, and thus their results should not be seen as precise representation of our collective thinking.
On the one hand, I don't think they present themselves as scientific, so attacking them for that is a little controversy-seeking. On the other hand, they also don't advertise how much they rely on such a small group of reviewers, and they probably should - else they should correct their algorithm further to prevent it. Ultimately, I applaud professor Vassilis Kostakos for his role as a watchdog, and helping improve crowdsourcing applications... in a crowdsourcing kind of way.
Posted by: Greg Boutin | September 17, 2009 4:41 PM
In business you have the 20/80 rule.
In social website possibly there is a 1/99 rule or even less.
Meaning a very small number of people actually generate content. From this small group a large part has a different agenda as the content they generate has a self-benefit purpose.
Posted by: LEADSExplorer.com | September 18, 2009 1:18 AM
Hey I found this website called HTcity.desimartini.com , it has a good collection of Page 3 party pics.
Posted by: Page 3 | September 18, 2009 3:50 AM
While I think it always important to question popular sentiment...I also think a crowd is defined primarily by its members participation. If you aren't contributing to some group, then your viewpoint may or may not be represented by others in that group.
I think the interplay between different crowds (i.e., different points of view) has been frequently referred to as politics. Per Mohammed's fine comments (above), when individual viewpoints can be observed without information cascades and the influence of identity, you might be able to glimpse the actual wisdom of the majority at a given point in time. Of course accepting or rejecting that wisdom is a highly individual decision.
Posted by: jeff hammond
Richard, Muhammad is right. Suroweicki is adamant that the four rules be followed for the wisdom of crowds to take effect.
In addition to knowing who submitted something, Digg also breaks Rule #2 (Independence) by showing how many diggs something has. This is clearly social bias...Duncan Watts did an amazing study showing how much influence these numbers can have on someone's behavior: http://bokardo.com/archives/social-design-101-aggregate-displays-change-user-behavior/
The idea of wisdom of crowds is that people make decisions independently, and then by aggregation we find out what the crowd thought.
In general, the wisdom of crowds theory isn't as applicable as originally thought...lots of social systems defy the rules in order to drive positive engagement.
Posted by: Joshua Porter | September 18, 2009 6:07 AM
An interesting array of comments, and concerns, which are fascinating in their own right. I've been cognizant of Surowiecki's work, and his four principles for assuring the ability to bring out the "wisdom of crowds" in addressing complex, highly charged topics, while avoiding the "dumbness" and group think so commonly seen. (The Bay of Pigs analogy is a useful one).
In my work, I've considered and used the four principles as we attempt to pull together the types of collaboratives whcih will allow for the truly creative thinking necessary in changing health care. While aware of the controversy around Surowiecki's work (likely suggesting he's hit on some sensitive nerves), I concur with many of the comments, specifically those of Mohammed, and have found this a useful construct.
Of course it is only one aspect of moving collaborative efforts forward. I've also considered the adaptive/technical leadership model of Ron Heifetz, Otto Scharmer's Theory U, and have used aspects of Group Genius as written by Keith Sawyer. An additional useful construct in the work has been incorporating the elements of Fair Process outlined by Kim in the Harvard Business Review in 1997.
Just my take on looking at the convergences and synergies of different concepts and philosophies, and drawing from them. Of course, attention to the appropriate and attentive use of the concepts is essential to prevent the unintended consequences of misapplication of these concepts.....much more to learn.
Posted by: Gary Oftedahl | September 18, 2009 6:18 AM
"Suroweicki is adamant that the four rules be followed for the wisdom of crowds to take effect."
In other words a utopia
Posted by: Don Ulrich | September 18, 2009 6:31 AM
What I'd like to see is non-fakeable metrics on ecommerce sites: return rates or reorder rates (as appropriate), for example. Or for apps, how many times users open the app per day/week or whatever.
Posted by: Lysander Meath Baker | September 18, 2009 7:14 AM
This is not just about TWoC, but human behavior: a small amount of people have, are, and will continue to contribute disproportionately. This is neither a "good" or "bad" thing of itself; simply observed behavior. And it's nothing new, especially when we consider eccentrics (meant in the most complimentary sense) who possess incredibly intense amounts of focused knowledge, and thanks to the Internet, are able to easier share their encyclopedic bearings on retrogaming, weird stuff happening in Asia, or even the quintessential archetype of the sci-fi geek.
Whether it's the proportion of blog commenters vs. blog readers, forum regulars vs. forum lurkers, or content creators vs. content consumers (like in virtual world Second Life), the "actives" are an incredibly devoted subset of the "passives". And someone who's very active on one site may be passive on others depending on how much they're able or willing to contribute. A CHOICE. Again, not a "good" or "bad" thing — many people enjoy watching YouTube videos rather than make them — this is just how it is, and clouding thoughts with an abstract "crowd" isn't helpful when what really matters are individual personalities. Sometimes collaborating, yes, but that's more vivid, peaky grouping (and certainly consistent with some of what Surowiecki suggested).
Not everyone has an interest in being heard, but severe actives often possess a clear proclivity towards self-expression and being rewarded for it — not necessarily in cold cash, but peer recognition and feeling special. Maybe swag. After all, from academia studies to urban art, having your ideas supported is nourishing! An early trickle of reward, even encouragement from the creators or earlier stars may very well snowball and turn a casual hobby into being a major voice on a site like Wikipedia or Amazon's reviews. I've grown to this position multiple times in varied online communities, so through experience, I can relate to how the process develops. And THOSE are the stories I'm most interested in.
As described in ambient "social media" books about reaching out to influential individuals instead of appealing to a faceless mass: power users (hence the name) will continue to wield wide impact over others, such as lesser-ranked currying their favor (like linking) to climb up the social ladder and learning how to ascend in the process.
Posted by: Torley
Not to mention the fact that the "wisdom" of crowds does not bring out any objective truth, just a collective subjectivism.
Posted by: Nathan Ford | September 18, 2009 9:50 AM
Joshua Porter's comment ++
In a live talk I saw Surowiecki give shortly after the book's release, he began with the statement that while ants as a crowd become more "intelligent", humans--as a crowd--become dumber. The unfortunate legacy of his book's misleading title is that those who did not read it have still used it to support a "None of us is as smart as All of us..." approach based on collaboration and consensus that--for the most part--the book demolishes.
Posted by: Kathy Sierra | September 18, 2009 2:02 PM
That's why I dont read books on technology and society trends written by journalists... they are all abou selling a good story, and miss out on the substance all to easily. Go for Duncan, Barabasi, Tapscott, people who actually have experience and research to back their opinions.
I mean, the wisdom of the crowd actualy says the opposite of what you find on most of the material published on social networks: small world phenomena, centrality, hubs, power law distribution of connections, etc.
Posted by: Filipe | September 18, 2009 3:30 PM
You miss the other key point: the Crowd isn't very clever http://www.itskeptic.org/folly-crowd
Posted by: The IT Skeptic | September 19, 2009 1:09 PM
The wisdom of crowds does work. The best example I have seen on the web is http://www.rateyourmusic.com - it's not a site with any bells and whistles, but if you are a music fan go search for your favorite band, look at the list of albums and "crowd ratings" that come up, and see if the crowdsourced ratings match what you (or the critics at large) think are the best, and worst, records by a given band. I bet you'll agree.
Similar things could be said about IMDB's user-voted films list. The system can work - though I agree wholeheartedly with the other posters who caution against some users with strong agendas polluting the results. Algorithms need to be a moving target. It's possible that rateyourmusic's accuracy has to do with it not being a very professional site and therefore off the radar of those who would try to game it.
We've just launched an ambitious beta startup at http://www.ranker.com that will also harness the wisdom of crowds, though that feature has not yet been made user-facing.
Posted by: Clark Benson | September 19, 2009 4:18 PM
The Guardian (uk newspaper) had online an article about people claiming their editing was undone on wikipedia, and the comments posted in response included eyewitnesses having their stuff deleted, and apparently you get higher ranked the more edits you make, many people compete, and the quickest edit is 'delete' so to get points fast, these types just delete every edit as soon as they find it. :(
Posted by: aiammaia
Interesting article. However, I'd like to point out one fundamental necessary condition for "The Wisdom of Crowds" that is often overlooked (also in this article): Crowds are only wise when the members come up with their individual opinions INDIVIDUALLY, that is, when they are NOT influenced by what other people already said, wrote etc.
For example, if twenty experts predict next year's inflation rate, then the average of their predictions will be more accurate when they DID NOT talk to each other about their predictions (as compared to when they did). In the latter case, their errors will become more similar because of the discussion and thus their errors are less likely to cancel out. The same logic also applied to voting procedures.
Surowiecki's (2004) book "The Wisdom of Crowds" (and other work) clearly spells out this boundary conditions of "The Wisdom of Crowds". He (and other people in this area) don't simply claim that having a bunch of people do something will be inevitably wise. It's the independence of opinions that's key!
So please stop bashing a strawman and actually criticize "The Wisdom of Crowds" as it actually was proposed.
Posted by: Stefan Herzog | September 20, 2009 11:45 PM
Relying on crowd opinions can be helpful, up to a certain point. Sites like Yelp are usually pretty accurate when it comes to group consensus about local businesses. Voting on the helpfulness of a particular comment can help weed out (or at least minimize) reviews that don't contribute to the community in a significant way.
Posted by: Michelle Barnette | September 21, 2009 10:50 PM
Yep, agree with all those who state that the research does not fit with Surowiecki's definition of the crowd. However, the research is interesting if linked to ideas of unrepresentative or illiberal democracy, as posited by Fareed Zakaria that suggests small interest groups can hijack democratic systems.
Posted by: Rob Godden | September 22, 2009 3:40 AM
RWW SPONSORS
FOLLOW @RWW ON TWITTER
follow @rww on Twitter
- Could Wowd Be the Skype of Real-Time Search? Private Beta Invites http://bit.ly/1ardPD about 6 hours ago
- Replying to writer applications. We're particularly looking for someone keen on Enterprise 2.0 topics. Email editor@readwriteweb.com about 6 hours ago
- Woopra Aims to Monetize Real-Time Analytics From Outside Silicon Valley http://bit.ly/72ulX about 6 hours ago
- Google Earth Tracks Marine Exploration http://bit.ly/L4xin about 8 hours ago
- Wanted: Writers for ReadWriteWeb. We're looking for more daily news writers. Email editor@readwriteweb.com if interested. about 10 hours ago
ReadWriteWeb on Facebook
RECENT JOBS
SAS Programmer
Irving, TX
