Rating Scales for Collective Intelligence in Innovation Communities

We all rate user-generated content on the web all the time: videos, blog posts, pictures. These ratings usually serve to identify content that we might like. In online innovation communities such as Dell’s IdeaStorm or My Starbucks Idea those ratings serve a different purpose: identify the “best” idea to be implemented by the host organization (there is a whole other argument that it is only about marketing and not about the “ideas” but let’s not go down that road). So the question arises: how can we best design collective intelligence mechanisms for idea selection in innovation communities?

In a paper published at the last ICIS conference, a group of researchers from TUM (Germany) compared three different rating mechanisms in a field experiment (n=313)  against a base-line expert rating.

To test the collective intelligence assumption, they also tested for a moderating effect of user’s expertise. Here is the research model:

They find that a multi-attribute scale works significantly better regarding rating accuracy than the simpler scales, but, as expected, there are some drawbacks regarding user satisfaction. Surprisingly, the most simple scale scores worst both on rating accuracy and satisfaction. They also find that user expertise has no moderating effect. Under the hood of the paper is an interesting method contribution: the authors present a way to judge an individual user’s rating accuracy (they term it “fit-score”). This could serve as a basis for future investigations in the design of collective intelligence mechanisms.

Christoph Riedl is currently a post-doctoral fellow at Harvard University researching innovation competitions and online communities. His focus is on open innovation, crowd sourcing, and collective intelligence.

Christoph Riedl, Ivo Blohm, Jan Marco Leimeister, Helmut Krcmar (2010): Rating Scales for Collective Intelligence in Innovation Communities: Why Quick and Easy Decision Making Does Not Get it Right. Proceedings of Thirty First International Conference on Information Systems (ICIS’10), St. Louis, MO, USA. SSRN