In the process of getting Roast Ratings organized and ready to go, one the most essential things we had to address was what system we wanted to use for actually rating the coffees. Since our goal is to be able to rate any coffee that’s out there being consumed by the general public,  there was a lot that we had to take into consideration. When it comes to how they take their coffee, people definitely have their own ways and preferences. Roast level, the adding of ingredients, preparation methods, hot v. iced, blends v single origin coffees- the list could go on forever.

We needed something capable of rewarding excellence, as well as honor coffee’s general use in society. Now, to create a system that can handle the demands of such a task, that was just the beginning of the adventure…

HOW TO TASTE?

Before we could get into the creation of the actual score sheet, we first had to determine our methods for tasting and evaluating coffee. First, we wanted to utilize the professional technique known as “cupping”. If you have never heard of it check out this video of Roast Ratings own, Pete Licata, giving explorer Mike Rowe a crash course in the subject.

To us as professionals, cupping is the standard method used in the industry to evaluate quality, and it speaks well to the versatility of a coffee through various brew styles. We’ve used these systems a lot in our careers and feel that maintaining the professional aspect of quality assessment is important in determining the how a coffee should rate. However, we knew that this alone was only a part of the bigger picture.

Next, we knew we would need to taste the coffees in brewed form, as that is truly how coffee is intended to be enjoyed.  In deciding on our official brew method, we knew that to be fair and accurate for our consumer audience it would need to be one that was designed for home use. While manual brewing like Chemex or Melitta have resurged in popularity, we opted for an automated brewer, not just because of the consumer point of view, but also to maintain consistent results with the ratings of every coffee. After trying the SCAA (Specialty Coffee Association of America) certified home brewers, we decided on the Bonavita 1900TS. For more information on how we selected that brewer, read more here.

HOW TO EVALUATE?

Once our methodology was settled, we turned our attention to our scoring system. We began by looking at what was already out there for tasting and evaluating coffee like the Specialty Coffee Association of America and Cup of Excellence scoring systems, as well as competitive scoring criteria (World Barista Championship and the like). All of these systems have been tried and used by the coffee industry with success, so we wanted to let them be an influence on us. If you’re unfamiliar with these systems and would like to know more here are some links to the scoring criteria, as well as a little breakdown including what makes them great alongside why they weren’t good fits for what we were going for:

• Specialty Coffee Association of America Criteria

The SCAA scoring system is mainly designed to designate whether or not a coffee’s quality measures up as “Specialty Grade”, ie- the final score must be over 80 points out of 100. It takes into account qualities like roast degree, consistency of the flavor and experience in multiple cups, and accounts for differences in regional flavor characteristics (where it comes from and how it is likely to taste as a result). While some of the best coffees out there will certainly fall in the high scoring range for this system’s criteria, it isn’t necessarily designed to be used in scoring many of the everyday retail coffees out there available to the masses.

• Cup of Excellence Criteria

The Cup of Excellence scoresheet serves to evaluate the quality of green coffee (unroasted), created to be the official scoring system of farming competitions in coffee producing countries. With that in mind, it’s geared towards what we in the coffee industry call “sample roasted coffee”- a small, lightly roasted batch of coffee that allows us to taste the character of the coffee itself without the interference of roast flavors.  Needless to say this particular roast profile is not what most of us encounter at the coffee shop, or the coffee aisle, on a regular basis. While this score is great for finding the best raw product, it not ideal for the varied final roasted product we find on store shelves.

• World Barista Championship Sensory Scoresheet

The WBC sensory score sheet is a tremendously helpful tool for defining aspects of flavor and balance (sweet, acidic, & bitter), but it is clearly geared towards espresso beverages. While it has been a very useful tool for both of us and shaped some of our thoughts on what coffees can and should be, it isn’t designed to rate filter brewed coffees for home consumption.

THE NEED FOR SOMETHING NEW

All of these systems serve their individual purposes well and are in fact very necessary in the coffee industry, but when it comes to evaluating coffee for everyday consumption we came to the conclusion that none of them quite fit the way we envisioned. After all, we aren’t rating the ability for people to source good raw coffee, we’re rating the flavor of the finished product as it is sold to the masses. You see, if we were to send 50 samples of one green coffee to 50 different roasters across the country, we would likely get 50 different interpretations of that coffee based on differences in and equipment and roasting philosophy. The experience of the coffee is what we want to hone in on and give a closer look. After all, at the end of the day (or beginning rather) that’s what coffee is all about. 

We needed something that could be fair in accounting for coffees, from the best to the worst and everything in between. 100 points is plenty high for such use, but the existing systems didn’t seem appropriate to rate the full spectrum of coffees out there. In other words, we wanted a system where 100 was theoretically possible, but so was 0. We knew that in order to do this we needed to take things to a different level. We decided that we wanted to give each tasting (cupping and brew) equal weight in the final score. We also knew that we wanted our score to be arrived at completely organically (ie- no automatic “raw score”). After several drafts and discussions, both amongst ourselves and with other professionals, we narrowed our criteria down to 7 main points that we wanted to evaluate, divided into three categories. 

PALATE

The ‘Palate’ section contains qualities that, while not the core of a coffee drinking experience, enhance or detract from it. ‘Aroma’ for the smells that create anticipation of the experience, ‘body’ that fills out the mouthfeel, or texture, that the flavors ride on, and ‘aftertaste’ which is the echoed impression left behind by the coffee. This section is worth 10 points per scoring session (cupping and brew) and 20 points of the overall score. 

FLAVOR BALANCE

Moving on to ‘Flavor Balance’, we come to the heart of the Roast Ratings score. Weighted by a multiplier (the total of this section is multiplied by 2) this section makes up 30 points in each scoring session and account for 60 points in the total score. Through our research, as well as taking into account our own preferences, we found that most people fundamentally want the same thing from their coffees: balance. In this section ‘Acidity’, ‘Sweetness’ and ‘Bitterness’ all have the same potential value, creating balance. Needless to say, the coffees that get the best scores on Roast Ratings will have pleasant, articulate and balanced versions of these qualities that play well together in the cup. 

DRINKABILITY

Last, but definitely not least we have ‘Drinkability’. On a scale from 0-10 we rate the whole of the experience, asking ourselves ‘how much did I enjoy this coffee?’ and ‘how likely am I to drink it again?’ Professionals would call this section ‘cuppers points’, or the one spot on the score sheet where the person gets to assert their own opinion on the experience, taking into account the other scores given. For more in depth information on how each attribute is scored, read the article Dual Rating System.

COMPARISON

So, how does our system differ from those that people are familiar with? Here are some of the basics laid out for easy comparison.

While there are definitely some similarities in all three systems, there are some obvious differences. Aside from our addition of a brew rating, our final score isn’t presented as a number but rather a star rating (which we’ll get into more here in a second). The other major difference is that our score is arrived at completely organically. We decided, given that our mission to score any coffee out there regardless of raw quality or roast technique, our system had to start at zero with every point earned through the experience of the coffee in both parts of the evaluation. 

Beyond the fundamentals there are other differences in the details. To dig into that a little deeper, here is a breakdown of the scoring criteria for each and how scoring is achieved in each system.

FINAL SCORE PRESENTATION

When deciding how to present the final scores, we took a long look at how people today relate quality to their purchasing decisions. When it comes to beverages in general there are a fair amount of scoring systems out there that are centered around a number from 0 to 100, but we only tend to see things that rate 88 and up. With rating coffees from all walks of life we needed something different that could be more inclusive. Expanding our research beyond food and beverage sites, we found was that there was far more familiarity and affinity for a different rating idea: a star rating from zero to five. Scattered across the internet are numerous examples of this, not least of which is Amazon, assisting consumers with purchasing decisions through the simplicity of stars. Thus the star rating was born. Once we evaluate the cupping and brews of our coffees we then add the scores together to get a composite score that equates to a star rating. Here’s a breakdown of that:

You’ll notice that a five star coffee doesn’t have to be perfect. It just has to be awesome. In the 100 point systems a score of 100 (i.e. an experience that was satisfying and delicious leaving you wanting for nothing) is rare, if not impossible. We want to celebrate great coffees and leave more room at the top of our scale to say- these coffees are truly exceptional, and while some of them may not be 100% perfect they’re pretty freakin’ close.  They’re some of the best we’ve ever tasted….and in our days we’ve tasted a lot of coffees.