Software Reviews: Read beyond the ratings

Vinay Bhagat
November 23, 2013

Software Reviews: Read beyond the ratings

It may seem odd for the CEO of a review site to take a stance against star ratings, but I believe they receive far too much emphasis. Don’t get me wrong. They have their place. In an era of content scanning and low attention spans, star ratings give a quick visual representation of sentiment. They do, however, need to be considered in context. 

For example, we find that simpler, less expensive products tend to average higher scores than more complex, expensive products on TrustRadius. This doesn’t necessarily mean they’re better. If your needs are more complex, you’re probably going to be better served by a more robust product, even if it receives a lower rating. I’m not 100% certain why we see this phenomenon but my guess is that users of more expensive products have higher expectations and are hence harsher critics.

More importantly, we find that there’s far more information value in the text of what people say to accompany their ratings than the ratings themselves. The average rating on TrustRadius is 3.96/5.00 and more than 75% of reviews score products a 4 or more out of 5. As a benchmark, per a March 2012 article in New York magazine, the average rating on Yelp is 3.8, and in 2009 the Wall Street Journal reported that Internet wide, the average consumer rating was 4.3.

tr-star-rating-distribution

Ratings are most useful when they represent extremes, helping you eliminate products or include them on your long-list. However you should never select one software product over based upon small rating differences. It’s critical to read the text that accompanies reviews. We’ve found that even raving fans are able to express where a product or service can be improved and what use cases it best serves and those insights can be very telling. For a terrific example check out this IBM Cognos review.

That’s why at TrustRadius, we’ve placed a huge emphasis eliciting in-depth reviews. Today, our average review is ~500 words and our longest over 2,500. According to Bazaarvoice, a software company that powers review forums for over 2,000 retailers/brands, the average consumer product review is under 100 words. Other business software review sites like the AppExchange tend to more closely mirror the consumer review approach of focusing on ratings with short comments.

If you’re making a critical product decision, the depth and quality of reviews is key.

About the Author

Vinay Bhagat
Vinay Bhagat, founder and CEO of TrustRadius, is an entrepreneur passionate about tackling big problems. Vinay conceived TrustRadius after experiencing challenges when buying enterprise solutions at his last company. In 1999, Vinay founded Convio, the leading Software as a Services platform for nonprofits. In April 2010, Convio became a public company, and was acquired in May 2012 for $325 million. Prior to Convio, Vinay was at Trilogy Software and Bain & Company. He holds an MBA from Harvard Business School where he graduated as a Baker Scholar, an MS Engineering Economic Systems from Stanford University, and a MA Engineering Information Sciences from Cambridge University with First Class Honors. When he’s not working, Vinay loves spending time with his family, playing squash and racing cars.