Rating systems...opinions wanted

_Paul · 07/29/2007, 06:19 AM

I'm mulling over changing the rating system for the updated PCE Software Bible. Marking out of 5 as it is seems alright, but I'm not completely satisfied with it. I've always believed that the text itself is the key to how good you think a game is and not the final rating, where 'middle' scores are always misinterpreted as meaning poor games. Ratings out of 100 or even 10 I think are next to useless; not only is it impossible to accurately rate something out of 100, but again anything below 75 or so is generally regarded as poor.

One idea is to have a simple rating of NO - MAYBE - YES represented by something like smily faces or thumbs up/down or somesuch. As most games would fall under the 'maybe' this would give a quick indication of what to avoid and what to go for.

Am I barking up the wrong tree if I simplify things? Do you think rating out of 3 or 5 is too restricting, and is 10 far too inaccurate?

Turbo D · 07/29/2007, 06:21 AM

I like the "out of 5" rating that you use and I've always felt your ratings were fair. I love how you use bonk meats as rating

SuperDeadite · 07/29/2007, 07:22 AM

Personally I don't like ratings and never use such systems. I think a simple text review is the best way to go, as it forces people to actually read your hard work and know what the game is actually about.

Also it helps prevent people from directly comparing different games. For example, lets say Dracula X gets a 5/5, and Cotton gets a 4/5, so Dracula X must be a better game then Cotton right? Platformers and shooters should never be directly compared like that because they are totally different genres and have nothing in common besides being PC-Engine games. Its like saying concrete bricks are better then tongue depressors. They are both excellent building materials depending if you want to build a house for yourself or your G.I. Joes.

I hate reviews that say "Graphics look like game X, music sounds like game Y." Such information is absolutely useless unless the reader has actually experienced game X and Y themselves. Its much better to say "I like this game's music because...." or "This game looks like crap because..."

If you want to use a rating system, the simple "Thumbs Up, or Thumbs Down" is the best. The main focus of a review should always be is the game fun or not and nothing more.

Just my 2 cents, either way I've always loved your site.

_Paul · 07/29/2007, 09:37 AM

Quote from: SuperDeadite on 07/29/2007, 07:22 AMPersonally I don't like ratings and never use such systems. I think a simple text review is the best way to go, as it forces people to actually read your hard work and know what the game is actually about.

This is true. However, if I had no ratings, then people would just keep asking for them. Thumbs up/down is a little extreme I think as there are games that are decent, but neither exceptional or poor. I would have a hard time chosing between the two, particularly on things like puzzle games (which I'm not a big fan of) without being unfair and unrepresentitive of how good the game actually is for its genre.

esteban · 07/29/2007, 09:37 AM

Quote from: SuperDeadite on 07/29/2007, 07:22 AMPersonally I don't like ratings and never use such systems. I think a simple text review is the best way to go, as it forces people to actually read your hard work and know what the game is actually about.

Also it helps prevent people from directly comparing different games. For example, lets say Dracula X gets a 5/5, and Cotton gets a 4/5, so Dracula X must be a better game then Cotton right? Platformers and shooters should never be directly compared like that because they are totally different genres and have nothing in common besides being PC-Engine games. Its like saying concrete bricks are better then tongue depressors. They are both excellent building materials depending if you want to build a house for yourself or your G.I. Joes.

I hate reviews that say "Graphics look like game X, music sounds like game Y." Such information is absolutely useless unless the reader has actually experienced game X and Y themselves. Its much better to say "I like this game's music because...." or "This game looks like crap because..."

If you want to use a rating system, the simple "Thumbs Up, or Thumbs Down" is the best. The main focus of a review should always be is the game fun or not and nothing more.

Just my 2 cents, either way I've always loved your site.

I totally agree. Rating systems fail because they really aren't a summary of the text (the content of the review)... unless you have a horribly written review.

Personally, though, I think that "Thumbs Up, Thumbs Down" is too constraining, so if you (Sunteam) were going to use a rating system, I think the "Yes, No, Maybe" ranking is the way to go. I honestly think that is the best way to use rankings, since I would put tons of games in "Maybe" and then explain (in the review) why the game might appeal to some niche audience despite the fact that it is universally panned by most folks (like China Warrior and Timeball, you bastards).

"Yes, no, maybe" converts to "buy, avoid, rent" with contemporary games, so I think it is a useful framework that folks can relate to.

spenoza · 07/29/2007, 09:38 AM

I do think moving away from numbers would help. I think the problem with people thinking 75 or less is a bad game comes from school system grades, where less than a C (70 or 75 or so) is "no credit". I think the best "rating" system would involve a scale of 1-5 but using words. Horrible - Poor - Fair - Good - Excellent. Fair is therefore a passing grade.

esteban · 07/29/2007, 10:02 AM

Quote from: guest on 07/29/2007, 09:38 AMI do think moving away from numbers would help. I think the problem with people thinking 75 or less is a bad game comes from school system grades, where less than a C (70 or 75 or so) is "no credit". I think the best "rating" system would involve a scale of 1-5 but using words. Horrible - Poor - Fair - Good - Excellent. Fair is therefore a passing grade.

Hmmmm, I had never considered a "Horrible - Poor - Fair - Good - Excellent" scale before. That sounds pretty good, and much better than "5 stars" and other such poppycock.

Yes, poppycock is the word for the weekend

.

GUTS · 07/29/2007, 02:55 PM

I like the school grading system way of rating, its quick and easy. No need to re-invent the wheel, just go with what already works.

NecroPhile · 07/29/2007, 03:33 PM

Quote from: guest on 07/29/2007, 09:38 AMI do think moving away from numbers would help. I think the problem with people thinking 75 or less is a bad game comes from school system grades, where less than a C (70 or 75 or so) is "no credit". I think the best "rating" system would involve a scale of 1-5 but using words. Horrible - Poor - Fair - Good - Excellent. Fair is therefore a passing grade.

I like the 'Horrible, Poor, Fair, Good, Excellent' idea, but it needs to relate to the bonk meats - they're too damn cool to abandon. Maybe rank 'em from Inedible to Delicious.

CrackTiger · 07/29/2007, 07:00 PM

The reviews for my site (which are still currently hidden), score Virtual Console games out of 10, in increments of .5, like EGM does. I score regular Turbo/PCE games out of 100.

I also have thumbs up/down type system and describe 'who' would like to play each game.

I recommend most of the games with mediocre scores overall and explain why.

Turbo D · 07/29/2007, 07:09 PM

why must you hide it from me

CosMind · 07/29/2007, 07:12 PM

i dig your concept of "no - maybe - yes", but would also add a final "must have" to the mix.

no - maybe - yes - must have

no = don't even bother
maybe = give it a try, you might enjoy it
yes = a definite play, possibly even a purchase
must have = a definitely must own

Keranu · 07/29/2007, 07:35 PM

Personally I prefer to avoid score numbers and such all together. A review is best "scored" from the text itself and adding a score number or something just seems to cause difficulty and contradictions with other game scores.

spenoza · 07/29/2007, 10:34 PM

I agree that numbers can be problematic because everyone interprets them according to their own scale, but at the same time, if the review is only in the text, it can be hard to extract an overall picture. I think there does need to be a one-work to one-sentence sum-up that covers, overall, whether the game is worth 2 beans or not.

esteban · 07/29/2007, 10:39 PM

Quote from: guest on 07/29/2007, 10:34 PMI agree that numbers can be problematic because everyone interprets them according to their own scale, but at the same time, if the review is only in the text, it can be hard to extract an overall picture. I think there does need to be a one-work to one-sentence sum-up that covers, overall, whether the game is worth 2 beans or not.

Yeah, I think you're right. A pithy sentence or two would go a long way in helping people, accompanied by the short-hand score.

D-Lite · 07/29/2007, 11:10 PM

For me, I don't mind so much the scale itself in any form, although I do think an A, B, C, D, F works nicely as everyone knows what that means. But.

I think breaking up the score by category is infinitely more useful. Meaning a rating for Graphics, Gameplay, Sound, Replay, and Overall. Sure, it's more work, but it says a lot more.

Take a game like R-Type. It's very nice graphically and the sound is well suited to the game. Gameplay is solid if not very hard and that can be reflected in the Gameplay score. And a hard game may not have too much replay value for some people. Now I would appreciate that type of scoring breakdown. It helps people draw conclusions based more on what's important to them. Graphics whores are going to favor that category and those looking for the longterm satisfaction will more heavily rely on the replay mechanic.

CrackTiger · 07/30/2007, 08:00 AM

Quote from: Keranu on 07/29/2007, 07:35 PMPersonally I prefer to avoid score numbers and such all together. A review is best "scored" from the text itself and adding a score number or something just seems to cause difficulty and contradictions with other game scores.

Thats why I put up a disclaimer/review guide explaining how to 'use' a review. Basically how the text is the real deal and the scores aren't to be used to pit games versus each other.

Quote from: turbo D on 07/29/2007, 07:09 PMwhy must you hide it from me

One review section is on the verge of being revealed.

Keranu · 07/30/2007, 12:34 PM

Quote from: guest on 07/30/2007, 08:00 AM
Quote from: Keranu on 07/29/2007, 07:35 PMPersonally I prefer to avoid score numbers and such all together. A review is best "scored" from the text itself and adding a score number or something just seems to cause difficulty and contradictions with other game scores.
Thats why I put up a disclaimer/review guide explaining how to 'use' a review. Basically how the text is the real deal and the scores aren't to be used to pit games versus each other.

Hopefully it's a bold and noticeable disclaimer or else people are going to be like "OMG!" if you gave a game a 5/10 when really that just means it's completely average.

Spector · 07/30/2007, 06:27 PM

I would say definitely keep a rating system as it's a great shorthand. My favourite way of marking is that used by Rolling Stone magazine for their album reviews which they've had for 40 years. Mark it out of 5 stars including half marks, giving you ten grades in total.

People say "if it's one to five using half divisions, why not just mark it out of ten?" The reason why is that three stars sounds more positive than six out of ten. Marks out of ten seems to always make 7 the average score, which I don't like. I always think of games using the star system. Vigilante is a ** 1/2. Pacland is a ****. Space Harrier is a ****. And Paul, Xevious is a ***1/2

So put my vote in for that method, cause it's my favourite.

CrackTiger · 07/30/2007, 07:28 PM

Quote from: Keranu on 07/30/2007, 12:34 PM
Quote from: CrackTiger on 07/30/2007, 08:00 AM
Quote from: Keranu on 07/29/2007, 07:35 PMPersonally I prefer to avoid score numbers and such all together. A review is best "scored" from the text itself and adding a score number or something just seems to cause difficulty and contradictions with other game scores.
Thats why I put up a disclaimer/review guide explaining how to 'use' a review. Basically how the text is the real deal and the scores aren't to be used to pit games versus each other.
Hopefully it's a bold and noticeable disclaimer or else people are going to be like "OMG!" if you gave a game a 5/10 when really that just means it's completely average.

Seperate from the disclaimer(s), I also have a breakdown of what the scores generally mean. To me, even a game that earns a 4 overall isn't garbage.

runinruder · 07/30/2007, 08:05 PM

That Bonk system rules.

D-Lite · 07/30/2007, 08:06 PM

Quote from: guest on 07/30/2007, 07:28 PM
Quote from: Keranu on 07/30/2007, 12:34 PM
Quote from: guest on 07/30/2007, 08:00 AM
Quote from: Keranu on 07/29/2007, 07:35 PMPersonally I prefer to avoid score numbers and such all together. A review is best "scored" from the text itself and adding a score number or something just seems to cause difficulty and contradictions with other game scores.
Thats why I put up a disclaimer/review guide explaining how to 'use' a review. Basically how the text is the real deal and the scores aren't to be used to pit games versus each other.
Hopefully it's a bold and noticeable disclaimer or else people are going to be like "OMG!" if you gave a game a 5/10 when really that just means it's completely average.
Seperate from the disclaimer(s), I also have a breakdown of what the scores generally mean. To me, even a game that earns a 4 overall isn't garbage.

I love you.

Keranu · 07/30/2007, 09:00 PM

Quote from: guest on 07/30/2007, 07:28 PMSeperate from the disclaimer(s), I also have a breakdown of what the scores generally mean. To me, even a game that earns a 4 overall isn't garbage.

Ahh that's very good, that's how I think of a out-of-10 score bar too. I love the Bonks!

esteban · 07/30/2007, 10:08 PM

PRETTY FRIGGIN' SWEET scoring guide!

spenoza · 07/30/2007, 11:12 PM

If Paul doesn't use your Bonk ratings system, we'll just have to beat him up (virtually, of course, so as to leave no bruises).

NecroPhile · 07/31/2007, 01:28 AM

Quote from: guest on 07/30/2007, 11:12 PMIf Paul doesn't use your Bonk ratings system, we'll just have to beat him up (virtually, of course, so as to leave no bruises).

The Bonks are cool, yet I still slightly prefer the meat scores. Anyway, never wimp out on a beating. A bag full of ripe oranges won't leave any marks, but will leave a wonderfully refreshing citrus scent.

_Paul · 07/31/2007, 03:35 AM

Quote from: guest on 07/30/2007, 11:12 PMIf Paul doesn't use your Bonk ratings system, we'll just have to beat him up (virtually, of course, so as to leave no bruises).

I think Black Tiger would have something to say if I ripped off the scoring system for his site wholesale.

From a visual standpoint, the new design is a bit cleaner and smoother so I'm not sure using sprites would fit in any more.

The extra score for exceptional games is one I've already considered, but as a special medal or something in addition to a thumbs up (kind of making 4 scores in total). I'm still not sure if I'll change things but this discussion is very interesting.