In this month’s issue of the New Yorker, best selling author and surgeon Atul Gawande pens a lengthy article
with the thesis that different physicians have different levels of ability; the physicians then achieve different levels of results because of their disparate talents. Dr. Gawande goes so far as to suggest that physicians’ skills may actually resemble a bell curve, a theory that chillingly implies there’s a surfeit of dreadful and below average doctors out there.
To illustrate his point, Dr. Gawande looks at two different Cystic Fibrosis (CF) clinics, one that achieves extraordinary results and one that achieves sub-mediocre outcomes. Regrettably, Dr. Gawande’s article is at best logically weak, at worst perhaps recklessly irresponsible. By handling the data he received from the CF community in a flawed manner, there exists the possibility that Dr. Gawande has done that community a grave and unwarranted disservice. There is a real chance this article will profoundly harm people who don’t currently want for hardships. The doctor should be ashamed of his effort, and the New Yorker should be ashamed of publishing it.
Cystic Fibrosis is a genetic lung disease whose average victim succumbs to the ravages of the disease at the age of 33. That represents great progress from a few decades ago when CF was considered solely a children’s disease because so few sufferers survived into adulthood. There are roughly 30,000 CF patients in the United States. Although the CF gene was discovered roughly 15 years ago, effective treatments for the disease and indeed a comprehensive understanding of how the disease works remain maddeningly out of reach. There are reasons for that, though, which I will get to in a minute.
Fortunately, the CF community is blessed with a wonderful and progressive national Foundation. Over the last several decades, the succession of visionaries in charge of the Foundation have been responsible for extending tens of thousands of lives. One of the Foundation’s initiatives in recent years has been to compile data from all of the country’s 117 CF treatment centers so the centers can learn from each other and treat the disease more effectively.
This initiative is what apparently attracted the attention of Dr. Gawande. Like I said, Dr. Gawande had a thesis that physicians of differing skills achieve different results and that physicians’ abilities as a whole represent a bell curve. Finding evidence to support such a thesis would, however, be difficult.
Physicians and hospitals are loath to track and compare such things for fear that “apples” will be compared to “oranges”. For instance, a gifted oncologist who willingly takes on all the tough cases will doubtlessly have a lower success rate than one who eschews seeing the “hopeless” patients. The oncologist who takes the hard cases would naturally not want his “success” rate compared to the other guy’s, and thus such numbers are hard to come by when they’re compiled at all.
But thanks to the CF Foundation’s initiative in comparing the country’s 117 CF treatment centers, Dr. Gawande hit the mother-lode. Here were reams of data he could use to support his thesis. He found that the results (the patients’mortality rates and lung functions) of the 117 centers resembled a bell curve. To further bolster his case, Dr. Gawande zeroed in on two centers, the one in Minneapolis which has the lowest mortality rate in the country and the one in Cincinnati which has a below average success rate. After spending time at both centers, he found the staff and doctors at Minneapolis to be a lot more skilled at what they do than their Cincinnati counterparts. Dr. Gawande spends roughly 3,000 words suggesting that this skill difference accounts for the longer lives of the Minneapolis patients. At no point in his article does he suggest ANY other factor might contribute to the Minneapolis center’s superior record.
Here’s what’s so incredibly sloppy about Dr. Gawande’s study: Throughout his article, he treats Cystic Fibrosis cases as if they’re all fungible. In over ten thousand words Dr. Gawande never reveals that there are over a thousand variations (remember, out of a population of 30,000) of the Cystic Fibrosis gene. That fact means that most Cystic Fibrosis patients have different “diseases” from one another. There are some CF patients born with such a virulent genetic mutation that they have little chance of living to their tenth birthday. There are other CF patients born with such a mild variation of the disease that they live almost completely normal young lives and avoid being even diagnosed with CF until they’re well into their 20’s. Any CF doctor, or for that matter any doctor, would tell you that BY FAR the biggest determinant in a CF patient’s outlook is the nature of his genetic mutation. Again, this is a fact Dr. Gawande omits from his article. Gawande’s entire thrust is that the physician’s skill is the great variable in a patient’s prospects. Indeed, at no point does he even acknowledge that any other variable might be a contributing factor to a patient’s well-being. For the CF community, this is risible nonsense.
In comparing the Cincinnati and Minneapolis centers, Dr. Gawande never tells us whether or not he’s looked into whether the populations have comparable distributions of the less virulent and more virulent genetic mutations of CF. Given the fact that he never so much as mentions the fact that there are different genetic forms of CF and that those forms present their holders with radically different “diseases” and outlooks, I see little reason to have faith in Dr. Gawande’s thoroughness.
Furthermore, given that there are 117 CF centers throughout the country with an average patient population of around 200, it would be expected that some centers would have unusually large clusters of “mild” cases while others would have unusually large clusters of “severe” cases. Given that the centers come by their patients by a “random” process (most of a given center’s patients are at that center by virtue of where they were born), statistically we would expect different centers to have populations of differing disease severity. The “Occam’s Razor” explanation for the discrepancies between the Minneapolis and Cincinnati centers is that they’re dealing with genetically different populations and that Cincinnati got the short end of the stick. I would be shocked if Dr. Gawande didn’t hear this from the folks at the Minneapolis clinic.
Here’s the great irony of Dr. Gawande’s piece. At one point Dr. Gawande concedes that it’s intellectually unlikely that physicians’ skills would be distributed along a bell curve. The explanation for this is that bell curves typically reflect only random populations and physicians are anything but a random population. All doctors are college educated, medical school graduates, self selected, self motivated enough to have made it through residency, etc. If you were to look at society as a whole as a bell curve, physicians would all be at the far end of that bell curve. Dr. Gawande writes that the physicians’ “curve” theoretically wouldn’t be shaped like a bell at all but instead like a shark fin with the biggest clusters being around the best results.
While physicians don’t reflect a random population, the populations at the CF centers ARE in fact random populations. When it comes to CF we’re talking about an overall community of 30,000 with over 1,000 different genetic variations, so no individual CF center (each of which has between a few dozen and a few hundred patients) would fully reflect the full range of the CF population’s disparities or even a significant portion of those disparities. It is therefore unsurprising that the results of the 117 centers composed of random populations reflect a bell curve. To say this bell is due primarily to the varying skill levels of the centers’ physicians instead of the random nature of the centers’ populations is remarkably implausible and indeed somewhat obtuse. All logic dictates that the bell curve Dr. Gawande finds is because of the randomness of the patient populations, not the self selecting physician populations.
It should also be pointed out, for what it’s worth, that Cincinnati is hardly a Cystic Fibrosis backwater. Those of you who are big football fans who have heard of CF might have first heard about the disease when longtime Cincinnati Bengals quarterback Boomer Esiason’s son Gunnar was born with CF. Because of Esiason’s heroic involvement with the Ohio and national Foundations the past decade, Cincinnati has been a veritable hub of the CF community, both well funded and progressive. The Cincinnati center is an unlikely candidate to be an exemplar of sub-mediocre medical practice.
So what’s the big deal about all of this? A lot. The potential pernicious effects of this study are several:
1) Dr. Gawande suggests that the treatment at the Cincinnati center is subpar. To say this will rattle the cages of the center’s patients (and the patients’ parents) would understate things. Indeed, Dr. Gawande interviews the parents of a seven year old patient who are perilously close to concluding they’ll have to move from Cincinnati to preserve their child’s health. Dr. Gawande’s sloppiness is likely to heap hardships on a population that hardly needs more troubles.
2) The Cincinnati center is presumably staffed by doctors, nurses, therapists, social workers and others who heroically devote themselves to treating critically ill children and young adults. Their efforts have been unfairly maligned because of a hopelessly sloppy statistical study. Similarly, the 50% of the nation’s CF centers that find themselves in the bottom 50% are likely to find themselves on the receiving ends of similar opprobrium. Again, this article is likely to bring a lot of grief to people who simply don’t deserve it.
3) When people have misfortune in their lives, they look for someone to blame even if their misfortune is just a stroke of bad luck. If you suffer from CF, you’ve had a rough break. Dr. Gawande has now served up a scapegoat for Cincinnati’s CF sufferers. Indeed, I wouldn’t be surprised if CF patients around the country begin clamoring to know where their center “ranks”. But given the crudeness of these rankings and the fact that even a putatively gifted physician like Dr. Gawande was unable to perceive their limitations, one must wonder how much good having this information generally available will do. One can also imagine this effect mushrooming across the entire medical landscape. One can picture gravely ill cancer patients blaming their misfortunes on the oncologists who as a rule take the hard cases (instead of avoiding them) and thus have a “below average” success rate. And one can picture, as a result, the medical community running from the hard cases instead of seeking them out.
4) As Dr. Gawande writes, “One small field in medicine has been far ahead of most others in measuring the performance of its practitioners: cystic fibrosis care.” And he gives the CF medical community its reward – to have its numbers distorted and reported out of context so a magazine writer can bolster his thesis. The people at the CF Foundation doubtlessly know that comparing the raw results of one center to another is comparing “apples to oranges” if one hasn’t weighted the results to reflect the different genetic mutations of the given populations. The readers of the New Yorker, however, do not. What kind of effect do you think this article will have on similar fields of medical endeavor that would like to track such things? Pretty damn chilling, I’d imagine. Since Dr. Gawande is such a fan of tracking physicians’ results, you’d think he’d be concerned about such a thing.
I should say clearly that if Dr. Gawande did check the different genetic mutations of the Cincinnati and Minneapolis centers and made sure he was comparing “apples to apples” none of the above is valid. However, given the fact that he hasn’t shown us any of his methodology beyond looking at the different centers’ mortality rates and lung function tests, I’m not too hopeful. Furthermore, Dr. Gawande is making some dramatic disparagements of the Cincinnati center; it’s appalling that he would make such a commentary without supplying more information regarding the methodology that led him to such dramatic conclusions.
Pending proof regarding the quality of his methodology, I can’t help but conclude that Dr. Gawande’s article is irresponsible journalism of the worst kind. Its author posits an expertise that he doesn’t possess and then proceeds in a recklessly sloppy manner. Worse still, he has made the lives of some CF patients and their families harder than they already were.
I agree with Dr. Gawande that different physicians have different abilities and talents and that the better doctors practice better medicine and, all other things being equal, will therefore achieve better results. But to support his case Dr. Gawande has marshaled inappropriate data paired with a handful of anecdotes.
Most readers of the New Yorker piece will probably be convinced of Dr. Gawande’s case in its entirety. That’s unfortunate. Unless quickly debunked this article is likely to have several pernicious effects on the medical community at large starting with the Cystic Fibrosis community. If Dr. Gawande’s methods were indeed as flawed as I fear, I earnestly hope he’ll do whatever is necessary to unring this bell that he has so irresponsibly and unnecessarily clanged.
Responses? Thoughts? Please email them to me at firstname.lastname@example.org
James Frederick Dwight