Friday, December 4, 2015

Putting Whiskey Rankings, Ratings, and Awards in Perspective.



You've seen the headlines. "Crown Royal Northern Harvest Rye Named Whiskey of the Year." Then the follow-up headlines. "Crown Royal Northern Harvest Rye Flying Off the Shelves."

In other news, the New York International Spirits Competition has taken to giving awards such as "Kentucky Rum Distillery of the Year" and "South Carolina Vodka Distillery of the Year." The facial absurdity of these awards, not to mention their sheer gall, seems to go unnoticed. Exactly how did Adam Levy and company assess and evaluate every vodka distillery in South Carolina and every rum distillery in Kentucky, and where exactly is evidence of the public's demand for that particular breakdown of information? The competition named Kentucky's best rum distillery but is silent about the state's much more prominent whiskey distilleries.

Just today, Whisky Magazine announced the results of the "Icons of Whisky Scotland 2016," "Hall of Fame Scotland 2016," and "Independent Bottlers Challenge 2015." Whisky gives an almost uncountable number of different awards every year, including its World Whiskies Awards, announced in the spring.

Whisky Advocate Magazine is dropping its annual awards now, one every few days, and this morning named Dad’s Hat Pennsylvania Rye as its Craft Whiskey of the Year. They are announcing award winners from eleven different whisky segments one day at a time, on their blog, from today through December 14. The Whisky Advocate awards will culminate with the naming of a Lifetime Achievement honoree on December 15 and Distiller of the Year on December 16.

Winners are inevitably proud to win and as the Crown Royal experience shows, a well-publicized win in the right competition is rewarded at the cash register. The normally circumspect Canadian whisky writer Davin de Kergommeaux is treating Crown's score as a matter of national pride. Competition results are news, after all. They are an automatic publish for many information outlets hungry for content that is timely, safe (uncontroversial), and free.

A part of every issue of Whisky Advocate and Whisky Magazine is devoted to rating new whiskey releases. Whisky Advocate uses a 100-point scale. Whisky uses a ten-point scale finished to one decimal point, so a 100-point scale. The content of most whiskey blogs is product reviews, usually with ratings. If you want to give your own whiskey awards there is nothing to stop you. All you need is a platform and a good PR agency.

Excluding hobbyists who rate spirits for their own amusement, most people involved in this sort of thing are selling something. Competitions typically charge a substantial entrance fee. Some are very profitable. Their 'product' is the awards themselves. If you win an award, you can publicize it. The publicity names the product and the award, but never talks about methodology. What was the basis for the win? Who was the competition? Who did the judging? You never hear or see any of that. It's just "Whiskey Name Wins Prestigious Award."

Since awards are the product, award givers maximize revenue by giving lots of them. Here's an interesting statement from the folks in New York. "While other renowned competitions prize up to 85% of entrants with awards, the discerning panel and ethos of the NYISC is to honor the brands that are most deserving among their peers. This year, NYISC prized only 46% of its entrants." So instead of almost everyone winning a prize, only about half do. That's integrity!

Crown Royal's big prize, which certainly sounds universal and definitive, is actually just the opinion of one person, Jim Murray, who writes and publishes an annual whiskey buying guide called, with unapologetic hubris, The Whiskey Bible. When one is disseminating the inspired word of God, no more explanation is needed.

Just as Murray's awards are designed to sell his book, awards given by publications such as Whisky Magazine and Whisky Advocate Magazine are intended to sell magazines and advertising. In each case there is a pool of voters, typically comprising the magazine's staff and maybe some of its freelance writers too.

Sometimes there are blind tastings, though often not. Sometimes the judging panels include producer representatives and sometimes awards go to that producer's product. Yes, really.

The problem with all of this is simple. The award givers are engaged in a ruse to convince you that something inherently subjective and personal can be rendered objective and universal just by how you describe it. One way to test the validity of these exercises would be to look at how often they agree with each other, which is almost never.

The target of all this folderol is you, the whiskey buyer. If they influence your buying decisions they perhaps provide a service by weeding out the really bad products, which rarely win awards. But are the products blessed by these self-proclaimed taste-makers really 'the best' in any sense?

Let's compare whiskey awards to something really important, the naming of Best in Group and Best in Show at the annual Westminster Kennel Club Dog Show. One obvious difference between whiskey awards and doggie awards is standards. The 'perfect' dog of each type is rigorously defined by a body recognized as the authority on that particular breed. While a judge's opinion is still subjective, it is based on comparing the candidate animal to a standard, an ideal. Learning and internalizing those standards is literally the life's work of the competition's judges.

Is there a comparable universally-agreed-upon standard for whiskey? No, not even close. Even the top distillers can't agree on what would constitute a perfect whiskey. Do judges at whiskey competitions have organoleptic training? Some do, some don't, and among those who do some have more than others. Are whiskey judges screened, are their organoleptic abilities evaluated? In a word, no.

So what? Well, if you understand all this and still find some or all of these declarations useful, then no harm done. Do lazy people use them as a cheat sheet, both for buying decisions and personal pontifications? Clearly, that's the downside.

23 comments:

Unknown said...

Well said, Chuck! There are a couple of email spammers who sell spirits and several times a day try to sell me something based on an award, and when I casually look into the pedigree of that award, it's clearly an award-marketing scheme like you describe.

Unknown said...

More and more respect to Chuck... he says exactly what I am always thinking about the industry. Thank you for a transparency, honesty, and bullshit-free perspective.

Bikram Singh said...

compelling article.

Unknown said...

When I first started to enjoy whiskey, i was a bit of a novice on the subject. So I read the awards from a few respected sources, and let it influence by purchases. After I bought some of the award winners and tasted them, did I learn the meaning of awards. What one person thinks is the best in category is not what I think. Today, I read the reviews, but I learned to let my pallet make the decision for me.

nathan kaiser said...

We get solicited everyday by different competitions. Who, for a "nominal" fee provide a great trophy / award to add to our collection. Unfortunately, we are bootstrapped and don't have the cash. Also, I am against these pay-to-play schemes across the board. The best is where if you advertise with said organization, the chances of winning increase significantly...

Anonymous said...

This is the new societal norm. Everyone gets a "participation" award. We don't want anyone going home unloved. Now, please excuse me while I go to my "safe room." It is where I stock all my booze.

Adam Levy said...

Chuck

Allow me to clarify a few things

-Yes we do take pride that less than 50% of those who submit win a medal
in our competition when compared to those on the West Coast or London
where over 84% win a medal. Lew Bryson article in the “Malt Advocate”
many years ago gave me the inspiration to start my first competition when
he asked in his article why do Malt Advocate results differ from all these competitions and
referenced San Francisco International Spirits Competition and IWSC.

-We do not accept any advertising dollars from any liquor producers on all of our
sites including the www.alcoholprofessor.com so consumers and trade will
know that a producer who buys a 5 page ad will not win a Gold Medal

-In fact in our last spirits competition only one spirit won a Double Gold
Medal. We again take great pride in that. So we are again not a medal
factory or a “3rd Grade Soccer Competition or I guess your check
cleared…here is your medal competition.

-My philosophy from the beginning is that the “Self Proclaimed
Taste-Makers” don’t know what consumers truly want. That is why all of
our judges are real trade buyers judging the liquid by its category and
actual price. I refer to this as the “Real World” So the judges will
know for example if it is a American Single Malt that sells for $45 or $42
or $58 they are judging. Consumers buy based upon price why should you not judge based
upon price.

-The judges sitting on our panels are top buyers in the New York Metro
Area and are award winning mixologists, restaurant owners, retail store
buyers and more. They judge and award medals after a consensus.

-Individual awards are awarded if a producer who already won a Bronze,
Silver, Gold or Double Gold Medal stands out compared to their peers. So
no I will confess we have not judged every Vodka from South Carolina…only
the ones that submitted to our competition. Though we would gladly
accept every commercially produced spirit around the world into our competition to be
judged by our trade buyers.

Overall I do believe we are the only international competition where all
the judges are real trade buyers judging the liquid by its category and
price. I also know that our growth has been hampered because we are not a
competition where over 84% win a medal but I am in this for the long haul
and will keep my integrity intact.

Cheers
Adam Levy
Founder
New York International Spirits Competition
Alcohol Professor

Doctor Tarr said...

I was driving through Ontario twenty years ago on the day the Tragically Hip was to perform on Saturday Night Live. Every radio station I tuned in was talking as if a Canadian spaceship was about to land on the Moon.

Treating any accomplishment by any Canadian as a point on national pride is what they do.

Luis Troccoli said...

Chuck - Great piece and could not agree more on the business of awards. Although several winners every year deserve praise - I´ve had my fair share of terrible whiskey experiences with some tag on the bottle claiming "Award Winning". With so many products in the market - I personally believe the answer is MORE truly independent reviews and specifically from consumers. We launched Craft Spirits Exchange this year (www.drinkCSX.com) as a discovery platform for craft and limited-release spirits and included a robust user-generated review platform to frame reviews and taste scores with experiences from actual drinkers. We also have "Featured Reviews" from a rotating panel of contributors but again - their score is just one more in the mix. We know there are risks with this model but at the end - we feel this is where the industry is going with great individual contributors leading the way (Jim, Paul, Yourself) and the growing community of passionate drinkers helping each other make smart choices about what you drink. Hope to see you on the site soon. :)

Cheers,
Luis Troccoli
Founder & CEO
CRAFT SPIRITS EXCHANGE
www.drinkCSX.com

Chuck Cowdery said...

So, basically, Yelp for booze.

DavindeK said...

Hehehe. Reaction by some Canadians on the whisky-web teetered between venom and hysteria. I hope I brought some balance.

The Malt Maniacs Awards and the Canadian Whisky Awards are each run as non-profits, and are judged blind by a volunteer panel that sees everyone's scores, rotates regularly and includes no brand reps. Awards/medals are based on scores. The better the field, the more medals.

Some of my favourite musicians have never taken a music lesson. There is politics up to the ears in dog shows. Now back my circumspection.

Davin

Luis Troccoli said...

Basically - although we don´t like to be compared with Yelp as they have been flagged for paid reviews and other such scandals. We do not pay for reviews and brands cannot pay us to remove reviews. All reviews must be "arms length" to avoid self-promotion. We use TripAdvisor as our comparison as they´ve done a much better job remaining a true community. We also focus on great story-telling and getting the stories behind great spirits. Happy to share more as we´d love to get your thoughts on the whole thing. Send us a note at team@drinkCSX.com if interested. We actually met two years ago during the first days of the project at Jen Massolo´s CRAFT event in Miami. Two years later - here we are!

Anonymous said...

Sadly, you are right on the mark with this commentary. I say sadly, because I enjoy reading about what is happening in the industry, yet everything I read seems more and more misguided and based on "awards." When I read all the pending information on the soon to be released Harvest Rye (last year), I wouldn't wait to try it. Then, when I got the chance to try it, I was disappointed. I am a rye whiskey fan all the way! I love the spice of a Monongahela style as well as the smoother sweeter Maryland style, and everything beyond, but was taken back by the off tasting Harvey Rye. Now I see how it was "selected" as Whiskey of the Year and just can't believe my eyes. To make that issue more perplexing, I am seeing how every one of the newer Craft Distilleries (which I prefer to call Micro Distilleries since even the corporate distilleries have crafting in their blood) seems to keep claiming "awards," and "more awards than any other craft distillery," as well as all the "first statements" such as "we are the first legal distillery in the state since...," or how about the "we are the first in the state to make bourbon" when there are clearly others in the state who have made bourbon, just not a company who chose to get on the "first" band wagon. As all this comes to a head and more and more brands hit the shelves, I fear new followers to whiskey may be chased away by misinformation, dissatisfaction in a product after reading that a product is great, only to find they dislike it. This could be a plus for me in the end as demand may slow and excess quantity could bring about lower prices to make my collection grow. Sadly, all this seems to be under the control of the "price point" in the retail world and may lose sight of the heritage and appreciation. Once again I find myself settling back to watch the chaos unfold. Get ready as the "awards & first" appear to be the new lie similar to what "age statements" were just a short time ago!

Bob
Pittsburgh, PA

Julian Van Winkle said...

Nice job Chuck! All these ratings make my head spin.
Cheers,

Julian Van Winkle, III
President
Old Rip Van Winkle Distillery

Chuck Cowdery said...

Comments from Chris Middleton:

‘Everybody wins a prize for showing up’ is a dismissive industry saying on the abundance of competitions and the recent proliferation of awards. Some comps do have worthy competition, of the small sub-set of distillery brands which to enter their product/s.

Of the two dozen or so major whisky competitions worldwide, thousands of whisky awards are given out.

Take the top ten whisky competitions where there are many hundreds of awards waiting to be allocated: from world’s best, gold to bronze and special mentions, craft, blended, Asian, European … the categories are exhaustive so everyone has a chance and regional manufacturing biases are minimised. If you wanted to enter a couple of categories in most of these awards (i.e. $250 entry fee, plus cartage, etc.) your need a large budget. Some the larger distillery conglomerates must spend a small fortune knowing there is a payoff in the game: ‘You’ve got to be in to win it’. In all probability, they will be rewarded with plenty of shiny metallic stickers to put on the packaging and forward to their PR company for bragging rights.

And we have not even questioned the calibre of the judges in some competitions (bar tenders, media personalities, celebrities, retailer staff, etc.). Some are curious choices; some can recruit the cream of the tasters. Even diehard whisky experts have different sensory preferences, scoring the same whisky numerically different points and attributing different written descriptors to other judges on the panel. Sensory evaluation can be very idiosyncratic in the minutia.

The more you scrutinise the competition game, the more inconsistencies and capriciousness you encounter e.g. different scoring systems, the law of numbers, distillers who pick one cask from thousands to enter in comps, to some competitions matching a $30, 5-year-old whisky up against 25-year-old $250 label. Old Olympians against primary school athletes.

I understand it’s now a part of modern whisky business. It makes money, often generates some awareness and hopefully sells more whisky; more likely causes committed whisky drinkers to change brands or to up-scale their expenditure. All good, healthy stuff.

Unlike the Olympics where superior athletes earn their place through elimination and empirical achievements, whisky competitions are subjective beauty contests. We should not take them too seriously.

Unknown said...

Great article , Well said !
I remember an article a few years back in the Wall Street Journal about wine judging in a similar vein which then went on to analyze what it took to get a gold medal - the short form - 10 competitions no matter how bad your product was, it would get one.
With all those competing panels out there there are plenty of panels to chose from.
My usual advice to a consumer is to compare them to movie critics, find the people who most closely mirror your personal tastes and you probably can't go too wrong.
Also as a side bar, don't forget Grey Goose Vodka , who won an award around 1984 or so as the "Worlds Best Tasting Vodka" way back then , and have since trademarked the phrase (Hence you will notice the ™ symbol at the end of that statement, making it a trademark, not an objective fact,using a long ago award.

Chuck Cowdery said...

The Grey Goose thing wasn't even an award, as such. It was a subjective tasting conducted by the New York Times, probably on a slow Friday afternoon.

Chuck Cowdery said...

"Preferred by New York Times staff writers" just doesn't have the same zing. Actually, the fact that you can trademark it shows how meaningless the phrase "World's Best-Tasting Vodka" really is. At law it's considered 'puffery,' something not capable of being proved, something that no reasonable person would take seriously.

Erik Fish said...

While I'm sure that there are some awards that have considerably more merit than others, and don't doubt the sincere efforts of some, as evidenced in some prior comments, to have standards and imbue their prizes with some meaning, ultimately they are all judging something quite subjective, and the result, whether bought or genuinely earned, is a marketing tool. To borrow a thought from Ralfy, who does scotch reviews online, most consumers buy whiskey not for its taste and quality, but to celebrate, commiserate, or just get pissed. For an overwhelming majority of whiskey buyers, those gold medal stickers on the bottle or in an ad are the equivalent of Colgate putting another "New and Improved!" on the toothpaste box: no more than a little psychological nudge for the buyer standing in front of the shelf eyeing the selection. Nothing much to get excited about.

Chuck Cowdery said...

Ralfy is the wisest of us all.

Anonymous said...

Great question in the beginning from Chuck and a great response from the NYISC on what things meant and how distilleries are awarded. Looking at the awarded distilleries, it seems old brands and new brands alike entered, so that's something to be said about the level of competition.

I'm with Juilan, I mean how does Pappy Van Winkle go from the number one whiskey in the world to not even making the top 20 whiskey list?? I'd say your head is spinning!

Unknown said...

I yell at the TV commercials at my house all of the time. Bud is a world class beer, Chevy fools customers into thinking they are selling a Mercedes, McDonald's sells health food, the NFL cares about breast cancer, a pink ribbon raises money for "awareness," it never ends. My wife just rolls her eyes at me, but she is weakening and coming to my side!

Cary Dice said...

One of your all-time classic posts.