Told by an idiot

[I have a Muse on Nature News about the perils and benefits of recommender systems. Here’s the pre-edited  version.]

Automated recommender systems need to put some jokers in the pack, if we’re not going to end up with narrow-minded tastes.

Medieval monarchy might not have much to recommend it compared to liberal democracy, but here’s one: today our rulers have no Fools. Even if the tradition was honoured more in literature – Shakespeare’s King Lear – than in reality, how often now will a national leader employ someone to laugh at their folly and remind them of bitter truths? More often, cabinets and advisers seem picked for their readiness to confirm their leader’s judgements.

Some people fear that the information age encourages this tendency to spread to the rest of us. The Internet, they say, is a series of echo chambers: people join chat groups to hear others repeat their own opinions. Climate sceptics talk only to other climate sceptics (and accuse climate scientists of doing likewise, perhaps with some justification). DailyMe.com will supply you with only the news you ask to hear, realising the vision of personalized news championed by Nicholas Negroponte of MIT’s Media Lab. The ‘Daily Me’ is now often used in a pejorative sense to decry the insularity this inculcates.

Now it seems you can’t make an online purchase without being recommended other ‘similar’ items. Music browsers such as Search Inside the Music, developed at Sun Labs, find you songs that ‘sound similar’ to ones you like already. But who’s to say you wouldn’t be more interested in stuff unlike what you like already?

That’s the dilemma addressed in a paper in the Proceedings of the National Academy of Sciences by Yi-Cheng Zhang, a physicist at the University of Fribourg in Switzerland, and his coworkers [1]. They point out that most data-mining ‘recommender’ systems such as those used by Amazon.com focus on accuracy, measured by testing whether they can reproduce known user preferences. This emphasizes the similarity of recommendations to previous choices, and can lead to self-reinforcing cycles fixated on blockbuster items [2].

But, say the researchers, the most useful recommendations may not be the most similar, but ones that offer the unexpected by introducing diversity. Like Lear’s Fool, they challenge what you thought you knew. Zhang and colleagues show that a judicious blend of algorithms optimized for accuracy and for diversity can actually offer more diversity and accuracy than any of the component algorithms on their own.

The researchers compare this effect with the value of ‘weak ties’ in our friendship networks. While we tend to seek advice from close friends – typically people sharing similar views and preferences – it is often comments from people with whom we have a more limited connection that are the most helpful, because they offer a perspective outside our regular experience.

The same is true in scientific research: scientists from disciplines outside your own can spark new trains of thought, while your fellow specialists trudge along the same track. Without fertilization from outsiders, disciplines risk stultifying. (One recent study implies that astronomy could be in danger of that [3].)

But it seems we instinctively gravitate towards the echo chamber. Networks expert Mark Newman at the University of Michigan has uncovered the stark division in purchases of books on US politics through Amazon [4]. He studied a network of 105 recent books, linked if Amazon indicated that one book was often bought by those who purchased the other. Newman found a pretty clean split into communities containing only ‘liberal’ books and only ‘conservative’ ones, with just two small bridging groups that contained a mixture. There was a similar split in links between political blogs. This clear division, Newman says, ‘is perhaps testament not only to the widely noted polarization of the current political landscape in the United States but also to the cohesion of the two factions.’ Recommender systems that offer ‘more of the same’ can only encourage this Balkanization of the ever-growing universe of information, opinion and choice.

Not everyone agrees there’s a problem. In an essay on Salon.com, David Weinberger disputed the notion of the Internet as an echo chamber [5]. He argues that some unspoken common assumptions – among liberals at that time, that George W. Bush was a bad president – allow online conversations to move on to more constructive matters, rather than becoming, say, a tedious litany of Bush-baiting. ‘If you want to see a real echo chamber’, said Weinberger, ‘open up your daily newspaper or turn on your TV.’

If people truly want more of the same, it’ll always be hard to make them hear the Fool’s wisdom. But most recommender systems do want to find what people will like, not just what they think they like. Throwing diversity into the mix is a good start, but the bigger challenge is to figure out how preferences are formed. What are the coordinates of ‘preference space’ and how do we negotiate them? There might, say, be something about the melodic contours or timbres in Beethoven’s music that a fan will find not in other early nineteenth-century composers but in twentieth-century modernists. Some music recommender systems are examining how we classify music according to non-traditional criteria, and using these as the compass directions for navigating music space. Understanding more about such preference-forming structures will not only improve the choices we’re offered but might also tell us something new about how the human brain partitions experience. And we could be in for some delicious surprises – just as when we used to browse through record stores.

References

1. Zhou, T. et al., Proc. Natl Acad. Sci. USA doi:10.1073/pnas.1000488107.
2. Fleder, D. & Hosanagar, K. Manag. Sci. 55, 697-712 (2009).
3. Guimerà, R., Uzzi, B., Spiro, J. & Amaral, L. A. N. Science 308, 697-702 (2005).
4. Newman, M. E. J., Proc. Natl Acad. Sci. USA 103, 8577-8582 (2006).
5. http://mobile.salon.com/tech/feature/2004/02/20/echo_chamber/index.html