I recently signed up for a Netflix trial account and rated around 300 movies. I noticed that I was giving out lots of 3 and 4 stars, a fair number of 5s, but very few 1s and 2s. It reminded me of this article, which says the average rating for products and movies, at lots of different sites, is around 4.3 out of 5. Here’s a post from Youtube showing that most ratings are 5s. Another from Yelp. The article asserts that people are too nice or, for some inexplicable reason, currying favor (from whom? why?). The implications of this are interesting for any collaborative filtering, or “crowdsourcing”, algorithms because if ratings are skewed high than you need to factor that into your recommendation system.
Since I am not a nice person nor am I sucking up to faceless corporate entities, why are my ratings generally favorable? The reason is that I’ve already filtered out the movies I know I’m going to hate by simply not ever watching them. I hate chick flicks, most goofball comedies, most horror movies, all musicals, many action movies and most epic films. Since I won’t watch any of these movies in the first place, I can’t give them the 1 star rating they deserve. The few times I really hate a movie is when I go simply because the movie is popular. I hate all the Harry Potter movies, but I watch them because everyone else seems to like them. (You are all idiots!) Same goes for books and products and restaurants. I don’t go to random restaurants hoping for a good meal. I read the reviews on Yelp and Chowhound and check their menus online before walking in the door. I buy electronic products after reading the reviews on Newegg and studying the specs. I read reviews and watch interviews with the authors before picking up a book. Most people use a similar suite of filters to pick a movie, book, or product; therefore, they will generally be pleased with what they get. A user’s average rating should be much higher than 3 stars (average). In fact, a user’s average rating is really a measure of his filtering scheme: the better his filters the more pleased he will be with his final choices.
Since the average ratings will be skewed too high, how can you get useful information from collaborative rating systems? For Netflix, they should more aggressively ask if you are even willing to see a particular movie (yes/no) and feed that info back into the recommendation system. The Netflix dataset does not contain this information. Yelp could do the same: does this restaurant sound interesting (yes/no)? For Amazon, it might help to allow customers to save a few choices so they can do a side-by-side comparison. For example, I could select 5 40” LCD TVs that meet my requirements. Amazon could have a slick interface that helps me compare and eliminate them one-by-one, leaving me with my final choice which implicitly gets a high rating because I chose to buy it. Apparently, eBay has a good solution of primarily looking at the percentage of negative sellers’ ratings because those are angry customers. Too many angry customers and you are booted from the site. All these systems can be improved by determining which choices a customer discarded on the way to making a final, usually satisfying, decision.