Rating Methods

Rating methods vary by product category but typically share the same key criteria—safety considerations, product performance as tested by CR's experts, and where applicable, longer-term reliability and owner satisfaction data from CR Members, data security and privacy.

CR holds webinars to update the industry on ratings and scoring changes, and also provides high-level test protocols and rating methodology documents. Click the link below to view and download.

What Goes Into CR's Overall Score?

Auto

The CR Overall Score for autos includes four key factors:

Road Test Reliability Owner Satisfaction Safety
Rating Criteria

Road Tests

Approximately 50 new cars and trucks each year are run through more than 50 tests at CR’s 327-acre test facility in Colchester, Conn.

Rating Criteria

Reliability & Owner Satisfaction

Data is collected in the CR Member Annual Reliability Survey (fielded in the spring and summer) on hundreds of thousands of cars across 17 trouble spots.

Rating Criteria

Safety

Safety is another major factor in vehicle scores. Safety ratings include crash-test data from the Insurance Institute for Highway Safety (IIHS) and the National Highway Traffic Safety Administration (NHTSA), if they’ve been conducted.

Appliances & Electronics

The CR Overall Score for Appliances & Electronics includes four key factors:

Lab Tests Reliability Owner Satisfaction Data Privacy and Security

Rating Criteria

Lab Tests

More than 2,000 products are tested annually in 63 laboratories at CR’s Yonkers, N.Y., headquarters and partner labs, and ratings for nearly 8,000 products are available on CR.org

Rating Criteria

Reliability & Owner Satisfaction

Data is collected in the CR Member Annual Reliability Survey on tens of thousands of major appliances and electronics products. Kitchen appliances (cooking appliances, dishwashers, refrigerators) surveys fielded in the spring; major electronics surveys (smartphones, televisions) in the summer; and laundry appliances (washers, dryers) in the fall.

FAQ

The Overall Score for Appliances, Home Products, and Consumer Electronics is based on a compilation of data from lab testing, and includes Predicted Reliability and Owner Satisfaction for many product categories, and may also include Data Security and Privacy.

  • Data from lab testing can include performance evaluations, ease of use assessments, and specification and features listings.
  • Predicted Reliability and Owner Satisfaction are based on brand-level data collected from CR member surveys. Models from brands that score Poor or Fair for Predicted Reliability cannot be designated as Recommended.
  • Data Security and Privacy testing is done on some categories with connected (Internet of Things) products, consistent with criteria of the Digital Standard.


What are the weighting factors?

Predicted Reliability can account for up to 20% of the Overall Score, Owner Satisfaction up to 5%, Lab Tests up to 100%, and Data Security and Privacy up to 40%. The percentages are determined on a category by category basis as a result of technical considerations and consumer priorities. For example, consumers are very concerned about the reliability of large appliances given the pivotal role they play in their daily lives, the inconvenience of a breakdown and getting them repaired. Consumers expect them to last a long time. As a result, Predicted Reliability for large appliances has a higher weighting, in general, as opposed to many electronic devices. Electronics tend to be more reliable and may be kept for a shorter period of time. Long-term reliability may be less critical to the consumer, which results in a lower weighting in some categories. The same principle holds for data security and privacy. For example, a router is critical for maintaining home network data security and privacy as all connected devices are linked to it. For this category, data security and privacy would have a higher weighting than, say, for connected coffee makers.

For more information, review the webinars and other rating resources here.

The Overall Score for Autos includes data from road tests, Reliability and Owner Satisfaction surveys of CR members, the availability of frontal crash prevention systems with pedestrian detection, along with blind spot warning; and, if available, results from government and insurance industry crash tests. Where we have insufficient survey responses, or when a model is all new or redesigned, we use our expert judgment based on brand track record and other similar models to predict reliability or owner-satisfaction ratings.

We give extra credit to vehicles that have the advanced safety systems that have been shown to reduce crashes, injuries, and deaths when they are offered as standard equipment across all trim levels of a model:

  • Highway-speed Automatic Emergency Braking (AEB)
  • City-speed AEB with pedestrian detection
  • Blind-Spot Warning (BSW)
  • Forward Collision Warning (FCW)


In order for a vehicle to be a Top Pick, CR requires standard FCW and City-Speed AEB with pedestrian detection. Starting in 2022, the AEB system must be able to operate at Highway speeds too.

CR will be scoring tested models with a combination of Adaptive Cruise Control (ACC) and sustained Lane Keeping Assistance (LKA) based on whether they have an adequate Direct Driver Monitoring System (DDMS). Tested vehicles that have such an automation system with an adequate DDMS system will begin getting bonus points in the overall score beginning in February 2022. Tested vehicles with such an automation system that lack an adequate DDMS will begin receiving a penalty starting with 2024 models.

What are the weighting factors?

While we do not provide our exact formulae or weighting factors, we do publish general information on our methodology and sample size thresholds as well as context on how the data is prepped for analysis.

Our research shows that reliability and safety are the most important factors for car buyers. For this reason, the Reliability Rating and test results that impact safety are the most heavily weighted in the Overall Score. Ratings for performance, comfort and convenience, everyday usability, and fit and finish are directly comparable for every type of car, but each value is weighted differently in the Overall Score for different test groups. For instance, while rear-seat access might receive a very low score for a sports car, the rating will not impact the Overall Score much.  

For more information, review the webinars and other rating resources here.

Depending on the product category (and what makes the most sense for each product we rate), our predicted reliability estimates are based on responses to one of the following three survey questions: 

  • Did this product ever break or stop working as well as it should? (Applicable for all major appliances, many electronic devices, and outdoor power tools.)
  • Have you ever had problems with this product since you’ve owned it? (Applicable for small appliances, including blenders and coffee makers.)
  • Which of the following best describes the reliability of this product during the time you have owned it? (Applicable for printers and smartphones; response choices are “very reliable,” “somewhat reliable,” “somewhat problematic,” and “very problematic.”)


Predicted reliability ratings are based on estimated instances of problems occurring within a given time frame, or “problem rates.” Problem rates within each product category are calculated at the brand level, not for each specific product model. 

For product categories with multiple major subcategories (for example, various types of refrigerators, such as French-door, top-freezer, bottom-freezer, side-by-side, and built-in), problem rates are calculated at the brand level but are subcategory-specific. 

Our goal is to measure product experience over time. Each survey gathers data on products purchased across a longer time frame. The number of years of purchases for which we gather data and then include in our reliability analyses varies by product category and is based on members’ expectations of how long newly purchased products in that category should last (as gathered through separate survey questions). Currently:

  • Ten years is members’ average expected life span for all major appliance categories.
  • Eight years is members’ average expected life span for blenders and coffee makers.
  • Five years is members’ average expected life span for computers.
  • Three years is members’ average expected life span for smartphones.


Advanced statistical models are used to calculate a brand’s Predicted Reliability rating (a reflection of its estimated problem rate) in a particular product category as a function of product age (measured through year of purchase), frequency of use, and extended warranty or service contract coverage. We control for these factors in our models to ensure objective comparisons among brands.

Across all product categories, Predicted Reliability ratings are set at the midlife of a product’s expected life span. We predict to the middle year rather than the last year because estimates that are pinpointed to the middle year are more reliable statistically than those made to the final year of the expected life span, and, because most products will have experienced one or more problems by the final years of the life span, there is less reliability differentiation across brands at that point. 

If we have sufficient data for a brand, we assign it its own Reliability rating for that product category. If we do not have adequate data to give a brand its own rating, we take the following steps:

  • First, we look to see whether we have enough data on other brands in that product category belonging to the same parent company.
  • If we do, we base our prediction for the brand lacking in adequate sample on an average of how the other brands belonging to the same parent company are rated in that category.
  • Barring that, we assign brands lacking adequate sample a category average rating. When this occurs, we display a dash in CR ratings tables to indicate that we did not have adequate data to calculate a reliability prediction for that brand.
  • In these cases, while the predicted reliability rating is the category average, if we do have any sample from that brand, the data is weighted as part of the predicted reliability score factored into the Overall Score.


For more information, review the webinars and other rating resources
here.

Predicted Reliability, also called New Car Prediction, forecasts how well a new model that is currently on sale is likely to hold up based on its recent history. For this Rating, we average a model’s Overall Reliability score for the newest three years, provided the vehicle did not change significantly in that time and hasn’t been redesigned for the current model year. One or two years of data may be used if the model was redesigned within that three-year time frame, or if there was insufficient data for some years. 

We will make a prediction for a brand-new or redesigned model, or a model with insufficient data, based on the manufacturer’s track record, history of the previous generation, or similar models that shared the same components. 

More on individual Reliability Verdicts for Trouble spots

The Model Year Overall Reliability Verdict is based on the gap of all trouble spots’ Problems Per Hundred (PPH) to the industry average. More serious trouble spots (engine major, cooling system, transmission major and drive systems problems) are more heavily weighted.

Separately, each trouble spot receives a 1-5 rating (chevron), based on where the PPH for that specific problem rate falls within the distribution for the industry. These individual trouble spot ratings are shown on CR.org, however they do not “add up” to the Overall Reliability verdict.

For more information, review the webinars and other rating resources here.

For Autos questions, please email gabe.shenhar@consumer.org. For all other product categories, please email the External Relations Group, externalrelations@cr.consumer.org.

Winter Survey (January to March): Cameras, Chromebooks, Cordless Drills, Dehumidifiers, Desktops, Laptops, Leaf Blowers, Mattresses, Tablets, Vacuums

Spring Survey (April to June): Auto Reliability, Cooktops, Dishwashers, Microwaves, Ranges, Refrigerators, Snow Blowers, Wall Ovens

Summer Survey (July to August): Air Conditioners, Auto Satisfaction, Blenders, Coffee Makers, Grills, Headphones, Smartphones, Televisions

Fall Survey (October to December): Advanced Driver Assistance Systems, Chainsaws, Clothes Dryers, Mowers, Printers, String Trimmers, Washing Machines