Skip to main content

Under Competitive Pressure, Nursing Homes Appear to Game Rating System

Research co-authored by Yale SOM’s Amandine Ody-Brasier shows that when nursing homes increased reported staffing in response to a change in Medicare ratings, care didn’t improve, especially in competitive markets. The study suggests that ratings based on self-reporting may be unreliable, and offers a solution: hide the thresholds for jumping to higher ratings.

An elderly man in a nursing home

Marlena Sloss/The Washington Post via Getty Images

By Dylan Walsh

Roughly 1.3 million Americans live in nursing homes, which means roughly 1.3 million families have grappled with the difficult question of which facility suited their loved one best. To help simplify this process, Medicare announced in 2008 that it would publish online quality ratings for all of the nursing homes in its network: five stars if the nursing home ranked “much above average,” four stars for “above average,” three stars for “average,” and so on.

Consumers would also be able to look beyond the overall rating to three sub-ratings that, also on a scale of one to five stars, explored distinct aspects of a nursing home’s performance: patient outcomes, results from onsite inspections, and staffing levels. This final variable was reported to Medicare by the nursing homes themselves.

In a new article with Amanda Sharkey at the University of Chicago, Ody-Brasier took advantage of a tightening of regulations to study how closely star ratings actually track a patient outcome known to be highly correlated with quality of care. In April of 2012, the Centers for Medicare and Medicaid Services increased the level of staffing required to maintain each star rating. If 0.5 nursing hours per resident per day originally corresponded to five stars, then the new regulation might require, say, 0.6 hours to maintain this rank. Nursing homes that were on the threshold of acceptability thus needed to boost their staffing hours to maintain their star rating.

Read the study: “Under pressure: Reputation, ratings, and inaccurate self‐reporting in the nursing home industry”

“One would expect that raising the bar in terms of nursing hours required by regulation would lead to better outcomes for residents in nursing homes,” Ody-Brasier said—an expectation backed by reams of research in public health and nursing. “But we see an increase in staffing levels does not seem to move the needle based on one very basic health outcome: bedsores.” Instead, where staff hours were reportedly increased, bedsore rates remained the same.

This core finding raised a troubling question: why would additional staff not lower bedsore rates? Are nursing homes gaming the system by misreporting staffing levels? Ody-Brasier and Sharkey ran a series of analyses to test alternate possibilities: Did nursing homes simply hire cheaper labor with less experience, meaning more staff hours didn’t lead to fewer bedsores? Was more time needed in the wake of the staff increase to demonstrate improvements in bedsore rates? The answer to both questions was no.

“We find that regulating with information provided by market actors themselves, unless that information is checked, can be quite problematic.”

What the two researchers did find, however, was that in competitive markets, where consumers have a lot of options, nursing homes were more likely to report staff increases without a related jump in quality of care. This result is consistent with the possibility that nursing homes are gaming the system: in crowded markets, where reputation is particularly important, companies may be more likely to care about maintaining their star rating.

The results point to a larger problem with ratings based on self-reporting. “We find that regulating with information provided by market actors themselves, unless that information is checked, can be quite problematic,” Ody-Brasier says. This is particularly true in the case of healthcare, where patients’ lives are on the line. And yet there is undeniable value in self-reported data: it’s cheap for regulators to collect.

One way to navigate these competing priorities, Ody-Brasier speculates, is keeping the precise threshold between star ratings private. If companies don’t know the number of nursing hours per patient that separates five stars from four stars, then gaming behaviors may be hindered.

Another recommendation falls on the shoulders of consumers, who should view star ratings as loose guidance, not a singularly informative variable on which to base decisions. In the case of nursing homes, consumers would ideally visit different locations and talk to current residents and staff.

Since Ody-Brasier and Sharkey began their research, the Centers for Medicare and Medicaid Services updated their rating process, now requiring nursing homes to pair reported staffing data with payroll information, but Ody-Brasier still offers a word of caution. “You can imagine an instance where you see a five-star rating for staffing and one star for quality outcomes,” she says. “This is cause for prompting a potential red flag to really get under the hood and find out what’s going on.”

Department: Research