What Makes a Good Online Review?
When Zagat launched its first survey of New York City dining, aggregating restaurant reviews by ordinary people was a novel idea. Today, the user-generated review has moved online and is a major influence on commerce. But models for collecting and presenting opinions continue to evolve.
By Ben Mattison
The user-generated online review is ubiquitous. Just about every e-commerce site provides a mechanism for users to evaluate their purchases, and research shows that reviews have a measurable impact on purchasing decisions. For companies like Amazon and Google, getting users to write reviews—and buyers to read them—has an impact on the bottom line.
When Google purchased Zagat Survey in 2011, and later incorporated Zagat ratings into Google Maps, it marked a small but significant shift for the online giant. Most of Google’s products depend on scale—sophisticated algorithms looking for patterns in massive amounts of data. Similarly, review sites like Yelp—which Google reportedly courted before buying Zagat—depend for their authority on a large number of reviews, to smooth out the effect of outliers and provide something like a consensus. Zagat starts with a different assumption: that knowledgeable reviewers will provide better data than a poll of everyone with an opinion.
This philosophy dates to Zagat’s origins, as a pen-and-paper restaurant survey conducted as a hobby by founders Tim and Nina Zagat, first in Paris, then in New York. At the time, the only published evaluations of restaurants came from the lone, disguised reviewers of the New York Times or the Michelin Guide. The Zagats assembled a broader, but still expert, view, first by collecting the opinions of friends with a passion for good restaurants, then adding friends of friends. “We thought if you had a large number of people you would be inherently more reliable,” says Tim Zagat, “particularly if you had people who were avid and therefore knowledgeable, you’d be more likely to get good results.”
As Zagat Survey grew into a commercial entity, added surveys on shopping, nightlife, spas, and other businesses, and began surveying electronically, it retained careful control over the pool of reviewers. “I think we get better data because we focus on people who are really passionate about the subjects,” says Nina Zagat. The company now solicits reviewers online, but it has a screening process to confirm that they are knowledgeable—and that they are amateurs, not business owners trying to influence the results.
Judith Chevalier, the William S. Beinecke Professor of Finance and Economics at Yale SOM, has written a series of papers examining aspects of competition in online commerce. She says that the Zagats anticipated one of the great debates of online reviewing: between “closed” platforms like Zagat and Expedia (where users must have booked a stay to review a particular hotel), and “open” sites like TripAdvisor or Yelp, where anyone can post a review.
“There are tradeoffs,” Chevalier says. An open platform tends to have more reviews, and that means more information and the ability to filter it more finely. “On Yelp I can read the reviews from people who went with their kids, or I can search only for people who mentioned the dessert, which is what I really care about.” But that density comes at a price—the likelihood that the reviewers are inexpert, or even fraudulent. “You never really know when you read a review, what’s the evidence that my tastes would be in line with this person’s? So I think that’s the advantage of a platform like Zagat’s.”
Another distinctive element of Zagat’s data is its 30-point scale. The scale also dates from the Zagats’ time in Paris, where existing surveys, like the Michelin Guide, had a three- or four-star scale; seeking to create a more nuanced measurement, the Zagats simply multiplied by 10. These days, of course, Yelp and other online reviewers typically have a five-star scale. “If you look at any five-point scale,” says Tim Zagat, “you’ll find that almost everything tends to be three and a half to four and a half. I don’t think that helps people make a decision.”
To Chevalier, the five-point scale on sites like Yelp reflects their orientation toward collecting as many reviews as possible. “It makes sense if you're trying to encourage people to review and you want people to be able to review and understand what they're doing fairly quickly.” Zagat’s aim, by contrast, is to recruit and retain a pool of expert reviewers with the time to internalize a finer scale. “The Zagat strategy is to create this barrier to reviewing and have semi-professional reviewers, in a sense.”
The 30-point scale allows for more precision, and gives the Zagat reviews a distinctive flavor that allows for comparison between restaurants. In New York City, the most elite restaurants, including Bouley and Le Bernadin, have a 29 for food—no restaurant has a perfect 30. A somewhat larger group of still high-end restaurants get a 28. A 25 might signal an excellent pizza place or sushi restaurant, while a 22 is a solid neighborhood spot. “Those distinctions, I think, are really meaningful,” Tim Zagat says. “Once you know that 30 is perfect and 0 is rotten, it doesn’t take you more than a minute, or a couple of restaurants, to understand the scale.”
And, the Zagats point out, their restaurant survey doesn’t have one 30-point scale, but three: food, décor, and service; the top-line data also includes the cost of a meal (including a drink and tip). “Those are different things and they play different roles, depending on what you’re doing,” says Tim Zagat. “If you’re taking the young kids out and they’re going to spill, you want a place that is inexpensive and the décor better not be too fancy.”
On Yelp or Amazon, the written reviews may be as important as the numerical rating; an evocative and detailed rave or pan can be decisive for a reader trying to make up his or her mind, even if it is an outlier. On Zagat, on the other hand, the words of individual reviewers are boiled down to a summary, shaped (in oft-parodied fashion) into one quote-laden sentence. The editors, Nina Zagat says, look for the “the best of the quotes, the most amusing, the most perfect of them.” But the quotes must conform to the numerical rating, she adds, removing, for better or for worse, the power that dissenters have on Yelp or Amazon.
Since joining Google, Zagat has expanded its surveying, but its ratings will still appear on only a fraction of the businesses and other points of interest in Google Maps, alongside reviews from users. This hybrid approach is perhaps a reflection of an industry that remains in flux.
“I think we have seen over the last few years just an explosion of different models of ways in which companies use reviews and the way in which they generate review platforms,” Chevalier says. Like Zagat and its successors, many of these companies are seeking the ideal balance between comprehensiveness and reliability. One recent innovation, Chevalier notes, is to leverage the user’s social network, which, like the Zagats’ original survey, could give all of us the ability to consult with our friends and friends of friends. “I think those efforts haven’t yet achieved a lot of density. But I think this idea of linking with social media is a way of providing a layer of trust.”
Related: Can Online Reviews Be Trusted?