U.S. News & World Report Rankings Explained
By: Dr. Jeffrey Anderson, Pediatric Cardiologist at Cincinnati Children's Hospital Medical Center (CCHMC)
Dr. Anderson has been a pediatric cardiologist in the Heart Institute at Cincinnati Children's Hospital Medical Center (CCHMC) since 2009. He is a graduate of the University of Utah School of Medicine and recently completed a Masters in Business Administration at the Ross School of Business at the University of Michigan. Dr. Anderson completed his pediatrics residency at the University of North Carolina Children’s Hospital before coming to CCHMC, where he finished his training in Pediatric Cardiology and advanced training in Pediatric Electrophysiology. He currently leads quality improvement and value efforts within the Heart Institute as co-director of the Heart Institute Safety, Quality and Value program. Nationally, Dr. Anderson holds several leadership positions including roles with the National Pediatric Cardiology Quality Improvement Collaborative and the American College of Cardiology.
Each spring the U.S. News & World Report releases their rankings of the best medical centers and clinical programs in the United States. The purpose of these rankings, as noted by the U.S. News & World Report is to “identify the best medical centers in various specialties for the most difficult patients”. Because the methods used by the U.S. News & World Report are complex, understanding and interpreting the rankings can be difficult for patients and families. This post is meant to help families understand some of the detail that goes into these rankings and give some guidance to interpreting them as you make decisions about your child’s care.
History of the U.S. News & World Report rankings
The following is taken directly from the 2015-2016 US News & World Methodology Report (http://www.usnews.com/pubfiles/2014_BCH_methodology_report.pdf). It is important to note that the U.S. News & World Report ranking for Pediatrics initially was entirely based on a reputational survey of physicians. Between 2007 and 2015 the reliance on reputation has decreased and the report has been more and more heavily based on outcomes, providing evidence-based processes of care, and structural (e.g. provider certification, presence of specific programs) elements. In 2015, the Pediatric subspecialty rankings still included a reputational survey, but this survey was only weighted at 16.7% of the total score. While the U.S. News & World rankings are still not perfect, each year additional outcomes measures have been added and the overall methodology continues to improve.
In 1990, U.S. News & World Report began publishing what was then called America’s Best Hospitals. The intent was to identify the best medical centers in various specialties for the most difficult patients − those whose illnesses pose unusual challenges because of underlying conditions, procedure difficulty or other medical issues that add significant risk.
The 2014-15 rankings mark the 25th year of annual publication. The focus on identifying top sources of care for the most difficult patients remains the same.
Pediatrics was among the original specialties in which hospitals were ranked, but until 2007 the pediatric rankings were based entirely on a reputational survey of physicians. Still, hard data to inform the rankings remained absent. Such data are critical because young patients present special challenges. Their small size relative to adults complicates every facet of care, from intubation to drug dosages; they are more vulnerable to infection; they depend on adults to manage and administer their medications, and they are treated for congenital diseases such as spina bifida and cystic fibrosis.
In the absence of databases for pediatrics comparable to MedPAR for Medicare recipients, U.S. News resolved to collect data directly from children’s hospitals. The first rankings that incorporated such data were published in 2007. Those rankings listed the top 30 children’s centers only in General Pediatrics. Data collection was subsequently broadened and deepened, and Best Children’s Hospitals now ranks the top 50 centers in 10 specialties: Cancer, Cardiology & Heart Surgery, Diabetes & Endocrinology, Gastroenterology & GI Surgery, Neonatology, Nephrology, Neurology & Neurosurgery, Orthopedics, Pulmonology and Urology.
The survey uses the framework developed in the 1960’s by Avedis Donabedian, conceptually defining the quality of healthcare using a structure, process and outcome framework. This framework is at the heart of many healthcare quality improvement initiatives and is based on the interrelationship of three key elements:
Structure: Indirect quality-of-care measures related to a physical setting and resources. Examples include staffing ratios, certifications, and patient volume.
Process: Measures that evaluate the method or process by which care is delivered, including both technical and interpersonal components. Examples include providing evidence-based care such as microalbumin screening for children with diabetes, and timely care delivery such as the timing of antibiotic administration for immunocompromised children with fever. Also included in this section is reputation with pediatric specialists.
Outcomes: Outcome elements describe valued results related to lengthening life, relieving pain, reducing disabilities, and optimal performance on disease-specific measures. Examples include bloodstream infections, survival after organ transplantation, and survival after complex heart surgeries.
Where does the information come from that determines the U.S. News & World Report rankings for Cardiology and who decides what questions to ask?
Information that is used to determine U.S. News & World Report rankings comes from two sources: 1) a survey that is completed and submitted by participating hospitals s and 2) a reputational survey that is sent to pediatric cardiologists and pediatric cardiac surgeon. Each year, programs are asked to submit data that are used to determine the rankings
Outcomes at the program (33% of the score)
Surgical mortality for various levels of complexity (STAT 1-5).*
Survival to first birthday after Norwood or hybrid Stage I palliation.*
One and three year survival after heart transplantation.
Prevention of ICU infections and pressure ulcers.
Processes at the program (16.7% of the score)
Does the program’s hospital use appropriate infection prevention measures?
Does the program formally monitor for surgical site infections?
Does the program have a single ventricle interstage monitoring program?
Does the program refer complex congenital heart patients for neurodevelopmental evaluation and intervention?
Does the program participate in known safety control measures?
Does the program participate in national registry and improvement collaboratives (NPC-QIC, IMPACT, STS, PHN)?
Structure of the program (33% of the score)
What is the surgical volume of the program for various levels of complex surgery (STAT 2-5)?*
What is the catheter procedural volume of the program?
Does the program have a cardiology and cardiac surgery fellowship?
Does the program have a cardiac transplant program?
Does the program have full-time subspecialists available (cardiac intensivists, cardiac anesthesiologists, adult congenital heart specialists?
Does the program have adequate nursing coverage?
Is the program committed to quality improvement and clinical research?
Is the program committed to engaging families?
* It is important to note that the surgical volume and outcome measures cover a four year period and lag for one year in the U.S. News & World Report rankings. For example, in the 2015 report surgical data from calendar year 2010-2013 was used to determine surgical volume and outcomes.
Reputation of the program (16.7%)
Each year a survey is sent to practicing pediatric cardiologists and pediatric cardiac surgeons. This survey asks participants to list 10 U.S. hospitals that they believe “provide the best care in Pediatric Cardiology for patients who have the most challenging conditions or who need particularly difficult procedures”.
The average of the previous four years of responses is used to score this part of the ranking. For example, in the 2015 survey the responses from 2012-2015 were used to determine the reputation score.
A question raised over the years has been the absence of patient and family experience scores as part of the rankings. This is obviously an important component of quality with patient/family responses providing a well-rounded and different perspective from which to inform improvement. Nationally standardized patient experience tools do exist for Pediatrics. Despite this, the tools are not broadly implemented across all reporting hospitals. Because of this variation, the ability to use these scores as part of a ranking methodology is not possible. Currently, hospitals are free to select the tool and method that best meets their informational and improvement needs. As long as this situation remains and hospitals are not required to report using a specific instrument, the use of patient and family experience performance data will not work for the rankings.
Continuous Survey Improvement
Each year, The U.S. News & World Report organizes a committee of pediatric specialists who review survey content and measures and make recommendations for improvement. This process helps ensure cross-hospital representation and involves content experts in the deliberations around measure selection and definitions. Although core outcomes that are nationally reported elsewhere (i.e. STAT survival) remain on the survey from year to year, changes that occur through this process can make year to year comparisons somewhat difficult. The makeup of the committee determining these questions not released by U.S. News & World Report. Each year, clinical programs are asked for input to improve the questions and that input may or may not be used to make changes the following year.
So how do I interpret and use these rankings?
First, there are problems with any system that attempts to compare clinical programs and rank them, the U.S. News & World Report is no exception. However, the U.S. News & World Report has made efforts over the last several years to place more of a focus on outcomes than on reputation. In addition, it is important that we have some way for parents and families to make informed decisions about programs and currently, the U.S. News & World Report provides a platform for this comparison.
One of the challenges with the U.S. News & World Report is the fact that much of the data, especially data about processes and structure of the programs, are self-reported. While the U.S. News & World Report does have the right to audit programs to determine the validity of their data, this is not a common occurrence. Another consideration is the lag associated with surgical volume and outcome data. Because the rankings use data from a four-year period ending the year prior, changes and improvement in the surgical program will not be reflected through the data for quite some time.
Despite the challenges described, the U.S. News & World Report rankings is a great way to begin the conversation about quality. After all, getting started is often the most difficult step. From that point, continued momentum for improvement around common measures that matter to patients and families can follow. If nationally standardized quality measures for a specific specialty or condition exist, then it is important that these measures be reflected in the survey. Viewed from this perspective, the survey becomes a way to both reinforce and make visible, quality measures that truly matter and make a difference in the lives’ of patients and families. Above all, probably the most important thing when using the U.S. News & World Report to make decisions about your child’s care, is that it serves as an important way to begin and continue the conversation about quality. From there, discussions with your cardiologist or cardiac surgeon about the structure and outcomes of the program at their center can continue, and then segue to a transparent conversation about program outcomes and efforts in place to continually improve.