Video game reviews: A symposium – In the Press

Shawn Elliott, Dan Hsu, and N’Gai Croal –all respected former or current members of the video game press– have coordinated a fascinating conversation with quite a few notable contributors in the on-line and print publishing industry regarding video game reviews. As a hack who is just now dipping his toe into those waters, I have found this first portion of their conversation intensely thought provoking and educational.

The debate invokes visions of politics– one side representing the idealistic and the other side representing the pragmatic. The idealist believes that review scores are harmful to the industry, prodding and pulling on the honest opinion of the reviewer and also negatively tainting the retail and marketing aspects of the industry. They don’t like review scores, and they hope they can go away altogether, no matter the odds against it. The pragmatic may not necessarily like review scores, but believe that the industry needs them to give their readers what they ultimately want, which in turn is what allows these companies to do business via much needed advertisement revenue.

The psychology behind review scores has been an unusual interest of mine recently, first triggered by Gamespot’s overhauling of their scoring metric in mid-2007. Jeff Gerstmann (now of Giant Bomb) had recently assumed the duty of Editorial Director for the site and was to my knowledge the instigator in their rating overhaul.

In a nutshell, the change brought simplification to their rating process, removing a weighted average score derived from the rating of multiple aspects of a game, and dropping from a 100 point scale (1.0 – 10.0 with .1 increments) to 20 (1.0 – 10.0 with .5 increments). To add some nuance to the new system, “badges” were added to describe at a glance what the reviewer like and disliked about the game being reviewed, such as “great story,” or “shallow game play.”

This particular change isn’t too interesting by itself, but is very interesting in context with what Gerstmann then went on to do after leaving Gamespot to form Giant Bomb, and 1UPs similarly timed move from a numerical system to letters, a la a high school or college grading system. For Gerstmann, with complete editorial control of Giant Bomb’s operation along with close fellow associates that followed from Gamespot, the review process that was settled upon was even further simplified into a 5 point scale represented as stars. In Metacritic terms, this means a game can get a 0, 20, 40, 60, 80, or 100 — nothing in-between. The goal of this, as I understand it, is to rid their review scoring of the fuzzy science behind what makes a game an 85 instead of a 90 (read: there is no legitimate difference) and eliminating the unspoken perception by readers that 70 is average game, not 50. This ultimately went towards better communicating one thing to the reader, and that is, “did I like this game or not,” give or take a “hell yeah” or “fuck no.” I think this approach –in a world where review scores are a necessary evil– is a pretty good one.

Adding to the debate of the worthiness of the latest Gamespot system, 1UP mixed things up during a similar hand-off in editorial control from John Davison to a Dan Hsu/James Mielke team this last February, whom assumed the roles of Editorial Director for the 1UP network and Editor-in-Chief of EGM and 1UPs non-PC video games division, respectively. They originally used a 20 point scale no different than Gamespot’s new system, but apparently came to view it as a flawed one. The rating system was revised to address the perceived confusion by readers of “what is an average game score.”

In a February interview with N’Gai Croal’s Level Up column, Hsu explained, “we switched to letter grades because we felt that it’d be more universally understood than our previous numeric scale. You see, in the past, we insisted on using the entire 0-10 range, with 5 being in the middle, rather than an “8-10 is good, everything below that is bad” scale most other outlets seem to use. But just because we insisted on it didn’t mean all our readers bought into it. So while a 7 meant “not bad, still good” to us, it looked like it was on the verge of failing to some in our audience. We tried it for a few years and decided it wasn’t working, so we switched to a letter-grading scale that we knew would be crystal clear to everyone. “

Granted, the 1UP review score metrics were deeply rooted in it’s print magazine heritage, particular Electronic Gaming Monthly, which had been using the 20 point 0-10 rating system since the dawn of time (well, a couple decades at least). Their standard was realistic, in which –to restate Hsu’s explanation– a 5/10 represented an average, acceptable game, but the perception of this score was influenced by other publications that reinforced the idea that a 7/10 or 70/100 has become average and a 5/10 or 50/100 meant  “the game was not very good.” To combat this, they took an unorthodox route and adopted the widely understood analogy of school grades, where a 70% score did indeed represent the average, acceptable game. Although not as distilled of a system as Giant Bomb’s, in my opinion, this again is a more effectively communicated review metric than the standard 1-10.

Anyhow, with all of this hullabaloo, I began thinking about what review scores meant to me. Is a simple or intricate rating system more effective? Would I buy a game scored below a certain number? Would I read a publication that avoided assigning scores to their reviews? Could I be retrained as a consumer to interpret these different metrics with different pivot points between good, average, and bad?

I will address these questions next week. In the mean time, I encourage you to read through the thoughts of these industry leaders, and ask yourself the questions I’ve posed and write in with your own thoughts.

Additional Reading

2006 Gamasutra reviews discussion with John Davison (1UP), Greg Kasavin (Gamespot), and Greg Vedermin (PC Gamer):
http://www.gamasutra.com/view/feature/1760/gamasutra_podcast_transcript__.php

February 2008 Level Up interview with Dan Hsu:
http://blog.newsweek.com/blogs/levelup/archive/2008/02/19/1up-network-editorial-director-dan-hsu-explains-his-companys-recent-overhaul.aspx

More information about Giant Bomb’s review policies:
http://en.wikipedia.org/wiki/Giant_Bomb#Review_system
http://www.giantbomb.com/help/

More information about Gamespot’s review policies:
http://en.wikipedia.org/wiki/GameSpot#Reviews_and_rating_system
http://www.gamespot.com/misc/reviewguidelines.html

More information about 1Up and EGMs review policy history:
http://en.wikipedia.org/wiki/Electronic_Gaming_Monthly#Review_philosophy
http://www.joystiq.com/2008/02/27/1ups-letter-grade-conversion-explained-analyzed/