I'm known to have a bad attitude about the rating systems used by spirits competitions. Their basic flaw is that they give a sheen of objectivity to something that is inherently subjective.
I had a long conversation about this with Jim Murray once and his conclusion was, "we owe it to people to give them some kind of guidance." I concede his point, but what's the best way?
Usually when you are asked to rate, say, customer service in a survey, 5 is considered 'acceptable' or average. Anything below 5 is considered some degree of less-than-acceptable, anything above 5 is considered some degree of exceptional. Below average, average, better than average. A classic bell curve.
Here is how the major competitions do it. This is a 10-point scale but since they allow tenths, just move the decimal point to make it a 100-point scale.
0 -5 FAULTY - There is something technically wrong with the product.
6 - 7 POOR - The product has little character or complexity and lacks balance.
7 - 7.5 AVERAGE - The product is okay but nothing special.
7.6 - 7.9 GOOD - The balance is good and there are elements of complexity.
8 - 9 VERY GOOD - The product is well balanced and complex.
9+ EXCEPTIONAL - The product is very complex, deep and rich, with lots of character.
Note that 0-7 is products that probably shouldn't be sold let alone entered into contests. 7 is 'acceptable.' Only above 8 does it get competitive. That's the way the scoring goes in most competitions. Nothing below 7, only a few 7 to 7.9, everything else 8 to 9.9. There are no tens because nothing is perfect.
The rationale is that, indeed, anything less than 5 is unacceptable, and while 5 to 7.9 might be acceptable, only 8 to 9.9 is award-worthy. The purpose of the competition is to determine which of the award-worthy products is best.
I'm neither attacking this system nor defending it, just explaining it. What do you think?
Chuck, here's my system. Please critique any way you see fit. But in short it's similar to what you mention. No rocket science here, but I think it helps people. Would love your critique.
ReplyDeletehttp://sourmashmanifesto.com/rating-scale/
Classic grade inflation.
ReplyDeleteNobody wants to be a 3 on a 1-10 scale, so it ends up not ever being used.
Grade compression on the upper end of the scale is deceiving. An 87 is 2 points better than an 85, but those 2 points are on a fake 1-100 scale. This system benefits retail sales, because a consumer sees an 87 and 89 side by side and thinks they're almost the same. Are they? Who knows. Rating a whisky with a single number is also crazy. Sadly, most people don't care, so a single number distillation is about as good as the lowest common denominator gets.
I dislike and distrust all liquor competitions. I find them to be less about rating the spirits, and more about boosting the status and lining the pockets of the hosts and judges. When a contest comes along where any spirit can enter, free of fee, then I may take notice. But even the, as you say Chuck, it's subjective at best.
ReplyDeleteIn my opinion numerical ratings don't really mean vey much. Unless a whiskey is faulted the scores reflect only the taste of the rater and little else. Once a whiskey moves beyond the faulted range everything else is rather subjective.
ReplyDeleteFor example why is complexity important? What is even meant by balance and why does the definition seem to vary from person to person?
Was a poll taken of the taste on most whiskey consumers and the tastes of the preponderance noted so that the raters could calibrate their taste to give an objective rating corresponding to the average consumer?
A review is different. A review tells me about a product and how it compares to things I might be familiar with. It puts a whiskey in context. More reviews and less rating would be a good thing for me.
I can sympathise with what you guys are saying, I am always impressed with Jim Murrays ratings as he scores, reviews with specific tasting notes and there is always a little extra thrown in to make it all a bit more readable. I think we will just have to put up with numerical systems as a necessary evil particularly in competitions, though whether they should be released or just placings given is another idea, as surely the numbers become more worthless comparing year upon year
ReplyDelete