This is top-notch stuff! And damn, I think you are right. I’ve suspected so for a while but didn’t quite know how to articulate it. I’ve been thinking about this problem in recent years from the angle of “non-specific psychopathology” as discussed in a 2023 paper by Peter Zachar: https://onlinelibrary.wiley.com/doi/10.1002/wps.21043
Your post adds new consideration to this. I’ll try to articulate my own thoughts on how we can navigate this mess. I love your idea of assigning confidence/probability ratings to diagnoses. This is something I’ve doing informally when I’m giving an official diagnosis for various reason but I am highly uncertain or have low confidence in it’s applicability.
Thank you, I appreciate that. The non-specific psychopathology concept (neuroticism?) is something I have thought about for a long time. Rereading the Ioannidis paper supplied a framework that I think is valid here.
As I read this, I was thinking of restacking (? Is that the word) a paragraph, but then I read more and I wanted to restack that, and that feeling kept going throughout this piece. Definitely one of the best I have read here.
It has been something I have been sensing more and more as I have more years under my belt, but you articulated it much better than I ever could have - my sense that, even as a psychiatrist, most psychiatric diagnoses are false.
You hinted at many of the biases leading to that result, from individual (clinician inexperience, or “pet diagnoses”), to systemic (needing a diagnosis for billing reasons). Ultimately the whole system is set up to diagnose, which compounds the problem.
Great job explaining sensitivity/specificity and PPV/NPV, especially considering you did a nice job explaining prevalence for the latter. Regardless of specialty, many clinicians will describe sensitivity/specificity fine but then go into PPV/NPV without contextualizing the measures within prevalence beyond a passing blow. I always try to give the clinician-authors the benefit of the doubt (I’m an ID epidemiology trainee) with epi methods, but even JAMA was publishing stuff up til the relatively recent JAMA statistics and methods editorial shift that any well-trained, competent epidemiologist could smell a mile away. The ADHD Dx and Tx literature gets this stuff right maybe 50% of the time. (With the disclaimer that I'm not a psychiatric epidemiologist; this stuff is more of a hobby research interest of mine 🥲)
This is an outstanding piece that's rigorous, clear-eyed, and soberingly honest about the limits of our tools. Thanks for laying it all out with such precision. I especially appreciated the framing of diagnostic interviews as underpowered tests run in biased conditions, often with unacknowledged thresholds and poor predictive value.
Reading this, I found myself thinking less about the technical implications (which are important, of course, but already familiar to me) and more about the *existential* ones. Particularly, the way the psy-disciplines (e.g., psychiatry, psychology, neuropsychology, psychotherapy) increasing seem to be coping with the terror of diagnostic uncertainty by doubling down on diagnostic essentialism.
That is, as our understanding of what a diagnosis *means* complexifies, we more fervently assert that having a diagnostic label is THE key to understanding a person. We've trained our clients to twist their complex selves and multifaceted pain into the constricted question of “Why do I have symptoms X and Y?”, which we then answer even more narrowly with “Because you have Symptom X and Y Syndrome.” The tautology is rarely acknowledged. Nor is the fact that the entire process reflects not just symptoms, but the patient’s worldview, the clinician’s worldview, and the social meanings attached to different diagnoses.
What most worries me is that the diagnostic “if” (if the person has a disorder) becomes a stand-in for more perceptive and generative questions we could be asking. When “if” becomes the central question, it subtly conveys that diagnostic legitimacy is the only path to being taken seriously --especially if the answer is “yes” to a popular diagnosis (internalizing, neurodevelopmental, or trauma-related), and “no” to an unpopular one (personality disorder, psychosis, low IQ, externalizing). This maps disturbingly well onto cultural scripts about whose suffering is understandable, whose is dangerous, and whose is inconvenient.
In my field of psychological and neuropsychological assessment, this narrow frame is often even more entrenched. We specialize in lengthy and complex evaluations, so we have the time, the tools, the training. And yet, we chase the “if” with near-religious intensity, rationalizing our preferred answer even when the data wavers. We sidestep more fruitful questions, such as:
-- Why is this (usually longstanding) problem surfacing so acutely now, and in this form?
-- When is the problem better or worse, and what helps?
-- Where in this person’s life are the conditions more favorable for growth or ease?
-- Who is this person, and how does that self-understanding shift the landscape of intervention?
When we chase diagnosis as destiny, we forget to ask how the person can engage with long-standing human needs: for meaning, self-compassion, challenge, responsibility, rest, and contribution. We don't ask, how is the balance for this person? Are they experiencing too much adversity (perhaps too much challenge, too little meaning, and too little rest), or are they instead adrift, avoiding responsibility and meaning (perhaps mistaking accountability for blame and desirable difficulty for suffering)? What small actions could shift them toward a life where they feel more alive, authentic, accomplished, or adored?
None of this is easy. The pressures (though they shift with setting) are real: time constraints, reimbursement demands, sociocultural dynamics around self-diagnosis, and the sheer inertia of systems. But I do wonder, have we lost the art (and nerve) of case conceptualization because we feel it doesn’t matter anymore? Do we feel that the treatments are the same regardless (statistically, the person is probably already on antidepressants and stimulant medication and in therapy anyway), and we're worried that without something more to say, some new diagnosis to add to the client's chart, we risk becoming irrelevant? Do we feel like our own value as clinicians is unearthing the most 'invisible' diagnoses, since we often feel powerless to shift the structures of a person's life? Is this because we can barely touch the cultural malaise making so many of us feel less attentive, less awake, less able, and less attached to meaning and community? Or are we afraid to recommend anything that involves real effort, loss, responsibility, or hope?
It’s a strange time to be in this field. Many of us sense that something isn’t working. But instead of returning to the harder questions (why, what now, how else), I fear we're increasingly clinging to the shallow comfort of “if.” For a while, this may keep us afloat, supporting our ability to feel useful in the face of complexity we can’t resolve. But what happens when AI can answer the 'if' questions as "accurately" as we can, with much less time and expense? Won't it be essential for us to have more to say?
Thank you again for this thoughtful piece. I will definitely be sharing it with my colleagues.
Thank you! I really feel that psychiatry/clinical psychology has found itself at something of a dead end with the current nosology, but we are in too deep to change it. Imagine the public reaction if we came up with some entirely new system and people's favored diagnoses went away.
As a layman who listens to many discussions on ideation, including that of psychiatric diagnoses, it strikes me that much of the time what is described as a disorder is an adaptation of thought to circumstance. In physical terms, a man without fingers (or with two thumbs, for that matter) must adapt to the circumstance he finds himself in, as best he can (and often poorly).
Psychiatric explanations seem to be made up out of thin air for just about everything over the past century. Prior to that, how was behavior explained? The actions of the planets (melancholy, due to the effect of Saturn, for example, resulting in the common adjective, saturnine) or witchcraft or the tenets of a religious belief, even numerology. Psychiatry is, of course, godless, but only to the extent that the self has taken the place of gods (or God) in the minds of many.
You might enjoy Dr. Randy Nesse's work, Good Reasons for Bad Feelings. I think you are right that some psychiatric disorders are best seen as adaptations, or failure to adapt. You could imagine a parallel between what we call chronic depression due to life circumstances and physical over-use injuries that many people have, for example.
Completely agree! Thankyou for your comment, it balances the weight of the 'intellectual' to that of being human. I always wonder where the spiritual comes in within psychiatry. And where this is on all of these 'scales' of madness.
This is a good article. Far better than most on this topic. A few critiques.
Critique 1:
Firstly, you don’t distinguish sharply enough between two claims:
(1) Most patients who receive a psychiatric diagnosis are diagnosed with at least one condition they don’t have
(2) Most of the time when someone is diagnosed with a specific psychiatric condition, they do not have this condition
Claim (1) is what your models actually show. They do not show claim (2). In particular, the ‘hitting a bullseye…’ section would not apply if you were trying to show (2). And the ‘typical evaluations are underpowered’ section would have to be modified as well.
Several commentors have clearly mixed up claims (1) and (2) and you haven't corrected them. Suppose I'm reading someone's chart and I see 'schizophrenia' . It's one thing to say 'this person probably has at least one incorrect diagnosis'. It's quite another to say 'this person probably does not have schizophrenia'. (1) does not imply (2).
Critique 2:
By focusing on (1), what you mean by an ‘incorrect diagnosis’ is that at least one of the conditions the patient has been diagnosed with is a condition that they don’t have. That’s not necessary completely unreasonable. But I think you should clarify why you’re focusing on this.
Personally, I wonder if it would make more sense to say that a patient has been incorrectly diagnosed if the most severe condition they have been diagnosed with is one they don’t have. Arguably, it’s this issue that would raise concerns about undermining trust in the psychiatry profession.
Critique 3 (a minor point):
Technically, the multiple comparisons issue that you point to can also increase the probability of a *correct diagnosis.*. You don’t make this explicit but it looks like what you’re modelling as a correct diagnosis is:
the patient receives at least one psychiatric diagnosis, and all of the diagnoses they receive are for conditions they have.
Now consider a patient who actually has conditions A, B, and C. And suppose multiple testing means that the patient gets tested for B and C instead of just A. This will increase the probability of a correct diagnosis. I don’t think this would seriously undermine your argument for (1) but it should be factored into your model.
But finally, note: you could derive (2) from a much simpler model, at least for certain conditions. Suppose almost everyone that shows up looking to get diagnosed with ADHD gets their diagnosis. And suppose almost no one who doesn't look for the diagnosis gets it. In this case, the probability an ADHD diagnosis is true will be roughly equal to the base rate of ADHD amongst people who show up looking for that diagnosis
So, I imagine that most clinicians couldn't tell you the Bayesian likelihoods of MDD (what I'm familiar with, I'm certain other diseases are also predictive for other psych disorders) on cancer versus "Metabolic Syndrome" (heart disease). If a doctor can't even estimate "what's the probability" based on the known diseases, of course you're a lot of false diagnoses.
(By the same token: do you diagnose someone with MDD if they have a "contributory illness"? If they have IBS, which is significantly more of a cause than a contributing factor? If you don't diagnose, are you still allowed to treat?)
Two comments: 1. Curious in these clinical gold standard interviews, when there is lower reliability, how far off the mark so to speak the diagnosis is. It’s one thing if someone who has schizophrenia is being diagnosed as just ptsd or GAD, but it’s another if they are “off” by diagnosing schizoaffective, schizophreniform, schizotypal, substance induced psychosis etc. If people are diagnosing their diagnostic neighbors, then I don’t think it is necessarily as bad of a problem. I for example have almost no interest in appropriately diagnosing MDD versus PDD versus GAD, versus primary panic instead of a panic subtype - I think of it as general internalizing and neuroticism, and don’t want to falsely reify these diagnoses to patients. Typically the treatment each of these follows in the same way.
2. I worry about prototype matching as an explicit approach (since we do this anyways as clinicians) - this would work if our diagnoses were not so heterogeneous and frequently have poor construct validity, and as you mention largely the absence of diagnostic hierarchy. I’ve seen, for example, residents say a patient doesn’t have schizophrenia because the patient wasn’t hallucinating or disorganized; of course you only need 2 out of 5 core symptoms to meet criteria, and so you can have totally non-overlapping pictures for someone meeting criteria for schizophrenia but if you anchor to a certain prototype you will miss this. You would neglect lots of types of schizophrenia. Not to mention prototype matching biases towards an inertia in practice, and whatever you saw/conceptualized early isn’t necessarily right. Given that we have no blood test or image that can validate your early suppositions were right, you can easily veer off in psychiatry to continue to see what you want to see.
I don't know all the details re inter-rater reliability. It seems like, from the DSM-5 field trials paper ( https://psychiatryonline.org/doi/10.1176/appi.ajp.2012.12070999 ) that the raters were provided a menu of diagnoses at each site, so for instance one site had PTSD, MDD, AUD, mild TBI, and BPD, and other diagnosis. Others had schizophrenia, Bipolar 1, etc. So the diagnoses are in the same realm, but also the options were limited. Overall this seems like it would inflate the kappa values.
I don't think prototype matching is a "solution," but I do think it's an improvement. A benefit of the prototype system is that it's easy to define several subtypes of a disorder, so I would think that helps the heterogeneity problem. Also, it would be useful to have a way to distinguish "strong" matches vs "weak" matches. I don't think severity quite does this.
I agree that many of these diagnoses are very similar/overlapping and that traits capture a lot of it. Something like an unspecified neurosis would go a long way...
It would be interesting if patients knew that their depression diagnosis was likely wrong... Especially if they got it from a GP. I admit I come from a place of superiority - NGRI is a pretty good place to start from to assert the credibility of my bipolar diagnosis, I'm not sure what else 4 nights of no sleep, literally didn't lie down for 4 nights, bizarre behaviour repeatedly brought to ED by police, in someone who's never abused substances, could be called... Let alone severe psychosis into the bargain... And even I had a senior psychiatrist review me, floridly psychotic, and write in my notes that I didn't have a psychiatric illness (3 forensic psychiatrists disagreed with him)
I'm interested in the pathophysiology of psychopathology. Not hand-waving and BS about unbalanced neurotransmitters, which granted are fading from the conversation. I have epilepsy and OSA, both arising in childhood. Either of these diseases can cause psychological symptoms. The two together would presumably cause more intricate pattern of symptoms as each disease waxes and wanes along various physiological and developmental pathways.
How often do I hear psychiatrists mention epilepsy? Not often. How often do I hear about OSA in psych lit? Even less. Could you guys reconnect with internal medicine and stop navel-gazing over your ludicrously overspecified and byzantine diagnostic manual? You are doctors after all.
This is my primary interest as well! The missing piece of Psychiatry is a physiological model similar to what other medical specialties have. I think about this a lot. It seems that a big barrier to progress is that we "can't say no" to people. If someone has new shortness of breath, we can do an echocardiogram and say "good news, you don't have heart failure." But even if we had adequate brain imaging, it would seem wrong to tell a sad/down person, "good news, your dopamine circuits are functioning normally. You don't have depression!" For example.
There's a difference between "you need help" and "you have this issue." The first is a statement of "care" -- and most doctors are able to say "You have unspecified pain."
"You need treatment" is fine and dandy to say -- but the insurance company may ask for a diagnosis. And, particularly for mental health, they don't want to hear "Situation Unclear, ask me again in 6 more sessions."
People come to doctors because they want someone else's perspective. Unfortunately, 50% of doctors or so will prescribe placebos, rather than saying "you're fine!"
Yes. In other areas of medicine, it is understood that some symptoms correlate to "objective" findings or test results, and some do not, and there is usually a distinction between these. You can imagine something similar in psychiatry, where we ascribe causes to symptoms/syndromes in some cases, but in other cases are not able to do so. This could lead to progress in understanding pathophysiology. Maybe!
I want to suggest a different paradigm. It's weather forecasting, of all things. You generally have a "face" prediction (50% chance of rain today), and then a "technical discussion" where the NWS forecaster talks about all the models, where they differ, and when he's going with his gut.
You kind of want that "technical discussion" to be passed from clinician to clinician.
Ideally, of course, you want predictive "here's what I think will happen" so that you can actually be differentiating between diagnoses. "If he's got anxiety as primary, he will do XYZ in response to going to the anime convention." "If he's got MDD as primary, he will do ABC in response to going to the anime convention."
The way people cope with physical illnesses is more psychology than psychiatry... I'd say there's no evidence for any psychiatric aspects of OSA, it'll all be non evidence based and so small effects they're statistical noise... beyond the obvious consequences of sleepiness...
So what you are saying is that not only should psychiatry be divorced from internal medicine but also psychology? That is the kind of thinking that is the root problem of what I described.
Also OSA causes quite profound effects on the brain through oxygen desaturation and fragmented deep sleep. Whether you label the inevitable symptoms psychiatric or psychological is immaterial to me.
Every so often, that story cycles around about how doctors were unable to apply Bayes' theorem to reason about false positive rates.
That's not really a problem of statistical literacy, it's just basic probability. I'm doubtful that adding adding more *statistics* to the medical curriculum would be very useful at this point in history.
But I think it probably would be good if MDs all knew basic probability in their bones the way a trader or poker player does. Not sure how to make this happen. Having trainees gamble on patient outcomes in the hospital would be very effective but unseemly.
I've been thinking about this. It would be easy enough to put a statistics calculator in an EMR so you could plug in the numbers. I think also changing how we write notes. If you imagine the one-liner/chief complaint as an introduction to the note, part of the expectation could be including at least your initial differential diagnosis, if not numeric probabilities. Some specialties, like Emergency Department, know these numbers pretty well already: 60 yo with this medical probability presenting with chest pain has this likelihood of STEMI, etc.
Having taught FM residents EBM decision making, the issue is actually the attendings. If I am the only attending showing them how to use likelihood ratio to train intuition, and they only work with me once a month they won’t learn it. I am not a big fan of just generating numbers from calculators since people tend to get false certainty from numbers. I used more of green, yellow, red tier system with green being diagnosis certain and treatment clear, yellow diagnosis is somewhat likely but other factors in play so gather information frequently while being cautious with treatment, red diagnosis not clear and need more information before treating or get a consult. Something with a good likelyhood ratio could take from red to green. Intermediate likely hood ratios would need multiple pieces of data pointing the same way. My big problem with the average therapist is lack of critical thinking about the certainty of diagnosis and then with shear amount of time they spend with the patient pigeon hole them into what is going on.
Good point about numbers and false certainty. That seems to be a general pitfall with Bayesian thinking. There also seems to be an issue with EMRs, the "publication bias" I alluded to, where a diagnosis that was intended to be provisional soon becomes "official" when the next provider looks at the problem list.
This is top-notch stuff! And damn, I think you are right. I’ve suspected so for a while but didn’t quite know how to articulate it. I’ve been thinking about this problem in recent years from the angle of “non-specific psychopathology” as discussed in a 2023 paper by Peter Zachar: https://onlinelibrary.wiley.com/doi/10.1002/wps.21043
Your post adds new consideration to this. I’ll try to articulate my own thoughts on how we can navigate this mess. I love your idea of assigning confidence/probability ratings to diagnoses. This is something I’ve doing informally when I’m giving an official diagnosis for various reason but I am highly uncertain or have low confidence in it’s applicability.
Thank you, I appreciate that. The non-specific psychopathology concept (neuroticism?) is something I have thought about for a long time. Rereading the Ioannidis paper supplied a framework that I think is valid here.
As I read this, I was thinking of restacking (? Is that the word) a paragraph, but then I read more and I wanted to restack that, and that feeling kept going throughout this piece. Definitely one of the best I have read here.
It has been something I have been sensing more and more as I have more years under my belt, but you articulated it much better than I ever could have - my sense that, even as a psychiatrist, most psychiatric diagnoses are false.
You hinted at many of the biases leading to that result, from individual (clinician inexperience, or “pet diagnoses”), to systemic (needing a diagnosis for billing reasons). Ultimately the whole system is set up to diagnose, which compounds the problem.
Great job explaining sensitivity/specificity and PPV/NPV, especially considering you did a nice job explaining prevalence for the latter. Regardless of specialty, many clinicians will describe sensitivity/specificity fine but then go into PPV/NPV without contextualizing the measures within prevalence beyond a passing blow. I always try to give the clinician-authors the benefit of the doubt (I’m an ID epidemiology trainee) with epi methods, but even JAMA was publishing stuff up til the relatively recent JAMA statistics and methods editorial shift that any well-trained, competent epidemiologist could smell a mile away. The ADHD Dx and Tx literature gets this stuff right maybe 50% of the time. (With the disclaimer that I'm not a psychiatric epidemiologist; this stuff is more of a hobby research interest of mine 🥲)
This is an outstanding piece that's rigorous, clear-eyed, and soberingly honest about the limits of our tools. Thanks for laying it all out with such precision. I especially appreciated the framing of diagnostic interviews as underpowered tests run in biased conditions, often with unacknowledged thresholds and poor predictive value.
Reading this, I found myself thinking less about the technical implications (which are important, of course, but already familiar to me) and more about the *existential* ones. Particularly, the way the psy-disciplines (e.g., psychiatry, psychology, neuropsychology, psychotherapy) increasing seem to be coping with the terror of diagnostic uncertainty by doubling down on diagnostic essentialism.
That is, as our understanding of what a diagnosis *means* complexifies, we more fervently assert that having a diagnostic label is THE key to understanding a person. We've trained our clients to twist their complex selves and multifaceted pain into the constricted question of “Why do I have symptoms X and Y?”, which we then answer even more narrowly with “Because you have Symptom X and Y Syndrome.” The tautology is rarely acknowledged. Nor is the fact that the entire process reflects not just symptoms, but the patient’s worldview, the clinician’s worldview, and the social meanings attached to different diagnoses.
What most worries me is that the diagnostic “if” (if the person has a disorder) becomes a stand-in for more perceptive and generative questions we could be asking. When “if” becomes the central question, it subtly conveys that diagnostic legitimacy is the only path to being taken seriously --especially if the answer is “yes” to a popular diagnosis (internalizing, neurodevelopmental, or trauma-related), and “no” to an unpopular one (personality disorder, psychosis, low IQ, externalizing). This maps disturbingly well onto cultural scripts about whose suffering is understandable, whose is dangerous, and whose is inconvenient.
In my field of psychological and neuropsychological assessment, this narrow frame is often even more entrenched. We specialize in lengthy and complex evaluations, so we have the time, the tools, the training. And yet, we chase the “if” with near-religious intensity, rationalizing our preferred answer even when the data wavers. We sidestep more fruitful questions, such as:
-- Why is this (usually longstanding) problem surfacing so acutely now, and in this form?
-- When is the problem better or worse, and what helps?
-- Where in this person’s life are the conditions more favorable for growth or ease?
-- Who is this person, and how does that self-understanding shift the landscape of intervention?
When we chase diagnosis as destiny, we forget to ask how the person can engage with long-standing human needs: for meaning, self-compassion, challenge, responsibility, rest, and contribution. We don't ask, how is the balance for this person? Are they experiencing too much adversity (perhaps too much challenge, too little meaning, and too little rest), or are they instead adrift, avoiding responsibility and meaning (perhaps mistaking accountability for blame and desirable difficulty for suffering)? What small actions could shift them toward a life where they feel more alive, authentic, accomplished, or adored?
None of this is easy. The pressures (though they shift with setting) are real: time constraints, reimbursement demands, sociocultural dynamics around self-diagnosis, and the sheer inertia of systems. But I do wonder, have we lost the art (and nerve) of case conceptualization because we feel it doesn’t matter anymore? Do we feel that the treatments are the same regardless (statistically, the person is probably already on antidepressants and stimulant medication and in therapy anyway), and we're worried that without something more to say, some new diagnosis to add to the client's chart, we risk becoming irrelevant? Do we feel like our own value as clinicians is unearthing the most 'invisible' diagnoses, since we often feel powerless to shift the structures of a person's life? Is this because we can barely touch the cultural malaise making so many of us feel less attentive, less awake, less able, and less attached to meaning and community? Or are we afraid to recommend anything that involves real effort, loss, responsibility, or hope?
It’s a strange time to be in this field. Many of us sense that something isn’t working. But instead of returning to the harder questions (why, what now, how else), I fear we're increasingly clinging to the shallow comfort of “if.” For a while, this may keep us afloat, supporting our ability to feel useful in the face of complexity we can’t resolve. But what happens when AI can answer the 'if' questions as "accurately" as we can, with much less time and expense? Won't it be essential for us to have more to say?
Thank you again for this thoughtful piece. I will definitely be sharing it with my colleagues.
Thank you! I really feel that psychiatry/clinical psychology has found itself at something of a dead end with the current nosology, but we are in too deep to change it. Imagine the public reaction if we came up with some entirely new system and people's favored diagnoses went away.
As a layman who listens to many discussions on ideation, including that of psychiatric diagnoses, it strikes me that much of the time what is described as a disorder is an adaptation of thought to circumstance. In physical terms, a man without fingers (or with two thumbs, for that matter) must adapt to the circumstance he finds himself in, as best he can (and often poorly).
Psychiatric explanations seem to be made up out of thin air for just about everything over the past century. Prior to that, how was behavior explained? The actions of the planets (melancholy, due to the effect of Saturn, for example, resulting in the common adjective, saturnine) or witchcraft or the tenets of a religious belief, even numerology. Psychiatry is, of course, godless, but only to the extent that the self has taken the place of gods (or God) in the minds of many.
You might enjoy Dr. Randy Nesse's work, Good Reasons for Bad Feelings. I think you are right that some psychiatric disorders are best seen as adaptations, or failure to adapt. You could imagine a parallel between what we call chronic depression due to life circumstances and physical over-use injuries that many people have, for example.
Completely agree! Thankyou for your comment, it balances the weight of the 'intellectual' to that of being human. I always wonder where the spiritual comes in within psychiatry. And where this is on all of these 'scales' of madness.
I know the way it is evaluated but i believe that this is just another form of oppression unfortunately.
This is a good article. Far better than most on this topic. A few critiques.
Critique 1:
Firstly, you don’t distinguish sharply enough between two claims:
(1) Most patients who receive a psychiatric diagnosis are diagnosed with at least one condition they don’t have
(2) Most of the time when someone is diagnosed with a specific psychiatric condition, they do not have this condition
Claim (1) is what your models actually show. They do not show claim (2). In particular, the ‘hitting a bullseye…’ section would not apply if you were trying to show (2). And the ‘typical evaluations are underpowered’ section would have to be modified as well.
Several commentors have clearly mixed up claims (1) and (2) and you haven't corrected them. Suppose I'm reading someone's chart and I see 'schizophrenia' . It's one thing to say 'this person probably has at least one incorrect diagnosis'. It's quite another to say 'this person probably does not have schizophrenia'. (1) does not imply (2).
Critique 2:
By focusing on (1), what you mean by an ‘incorrect diagnosis’ is that at least one of the conditions the patient has been diagnosed with is a condition that they don’t have. That’s not necessary completely unreasonable. But I think you should clarify why you’re focusing on this.
Personally, I wonder if it would make more sense to say that a patient has been incorrectly diagnosed if the most severe condition they have been diagnosed with is one they don’t have. Arguably, it’s this issue that would raise concerns about undermining trust in the psychiatry profession.
Critique 3 (a minor point):
Technically, the multiple comparisons issue that you point to can also increase the probability of a *correct diagnosis.*. You don’t make this explicit but it looks like what you’re modelling as a correct diagnosis is:
the patient receives at least one psychiatric diagnosis, and all of the diagnoses they receive are for conditions they have.
Now consider a patient who actually has conditions A, B, and C. And suppose multiple testing means that the patient gets tested for B and C instead of just A. This will increase the probability of a correct diagnosis. I don’t think this would seriously undermine your argument for (1) but it should be factored into your model.
But finally, note: you could derive (2) from a much simpler model, at least for certain conditions. Suppose almost everyone that shows up looking to get diagnosed with ADHD gets their diagnosis. And suppose almost no one who doesn't look for the diagnosis gets it. In this case, the probability an ADHD diagnosis is true will be roughly equal to the base rate of ADHD amongst people who show up looking for that diagnosis
So, I imagine that most clinicians couldn't tell you the Bayesian likelihoods of MDD (what I'm familiar with, I'm certain other diseases are also predictive for other psych disorders) on cancer versus "Metabolic Syndrome" (heart disease). If a doctor can't even estimate "what's the probability" based on the known diseases, of course you're a lot of false diagnoses.
(By the same token: do you diagnose someone with MDD if they have a "contributory illness"? If they have IBS, which is significantly more of a cause than a contributing factor? If you don't diagnose, are you still allowed to treat?)
This is quite good.
Two comments: 1. Curious in these clinical gold standard interviews, when there is lower reliability, how far off the mark so to speak the diagnosis is. It’s one thing if someone who has schizophrenia is being diagnosed as just ptsd or GAD, but it’s another if they are “off” by diagnosing schizoaffective, schizophreniform, schizotypal, substance induced psychosis etc. If people are diagnosing their diagnostic neighbors, then I don’t think it is necessarily as bad of a problem. I for example have almost no interest in appropriately diagnosing MDD versus PDD versus GAD, versus primary panic instead of a panic subtype - I think of it as general internalizing and neuroticism, and don’t want to falsely reify these diagnoses to patients. Typically the treatment each of these follows in the same way.
2. I worry about prototype matching as an explicit approach (since we do this anyways as clinicians) - this would work if our diagnoses were not so heterogeneous and frequently have poor construct validity, and as you mention largely the absence of diagnostic hierarchy. I’ve seen, for example, residents say a patient doesn’t have schizophrenia because the patient wasn’t hallucinating or disorganized; of course you only need 2 out of 5 core symptoms to meet criteria, and so you can have totally non-overlapping pictures for someone meeting criteria for schizophrenia but if you anchor to a certain prototype you will miss this. You would neglect lots of types of schizophrenia. Not to mention prototype matching biases towards an inertia in practice, and whatever you saw/conceptualized early isn’t necessarily right. Given that we have no blood test or image that can validate your early suppositions were right, you can easily veer off in psychiatry to continue to see what you want to see.
Thanks.
I don't know all the details re inter-rater reliability. It seems like, from the DSM-5 field trials paper ( https://psychiatryonline.org/doi/10.1176/appi.ajp.2012.12070999 ) that the raters were provided a menu of diagnoses at each site, so for instance one site had PTSD, MDD, AUD, mild TBI, and BPD, and other diagnosis. Others had schizophrenia, Bipolar 1, etc. So the diagnoses are in the same realm, but also the options were limited. Overall this seems like it would inflate the kappa values.
I don't think prototype matching is a "solution," but I do think it's an improvement. A benefit of the prototype system is that it's easy to define several subtypes of a disorder, so I would think that helps the heterogeneity problem. Also, it would be useful to have a way to distinguish "strong" matches vs "weak" matches. I don't think severity quite does this.
I agree that many of these diagnoses are very similar/overlapping and that traits capture a lot of it. Something like an unspecified neurosis would go a long way...
It would be interesting if patients knew that their depression diagnosis was likely wrong... Especially if they got it from a GP. I admit I come from a place of superiority - NGRI is a pretty good place to start from to assert the credibility of my bipolar diagnosis, I'm not sure what else 4 nights of no sleep, literally didn't lie down for 4 nights, bizarre behaviour repeatedly brought to ED by police, in someone who's never abused substances, could be called... Let alone severe psychosis into the bargain... And even I had a senior psychiatrist review me, floridly psychotic, and write in my notes that I didn't have a psychiatric illness (3 forensic psychiatrists disagreed with him)
I'm interested in the pathophysiology of psychopathology. Not hand-waving and BS about unbalanced neurotransmitters, which granted are fading from the conversation. I have epilepsy and OSA, both arising in childhood. Either of these diseases can cause psychological symptoms. The two together would presumably cause more intricate pattern of symptoms as each disease waxes and wanes along various physiological and developmental pathways.
How often do I hear psychiatrists mention epilepsy? Not often. How often do I hear about OSA in psych lit? Even less. Could you guys reconnect with internal medicine and stop navel-gazing over your ludicrously overspecified and byzantine diagnostic manual? You are doctors after all.
This is my primary interest as well! The missing piece of Psychiatry is a physiological model similar to what other medical specialties have. I think about this a lot. It seems that a big barrier to progress is that we "can't say no" to people. If someone has new shortness of breath, we can do an echocardiogram and say "good news, you don't have heart failure." But even if we had adequate brain imaging, it would seem wrong to tell a sad/down person, "good news, your dopamine circuits are functioning normally. You don't have depression!" For example.
There's a difference between "you need help" and "you have this issue." The first is a statement of "care" -- and most doctors are able to say "You have unspecified pain."
"You need treatment" is fine and dandy to say -- but the insurance company may ask for a diagnosis. And, particularly for mental health, they don't want to hear "Situation Unclear, ask me again in 6 more sessions."
People come to doctors because they want someone else's perspective. Unfortunately, 50% of doctors or so will prescribe placebos, rather than saying "you're fine!"
Yes. In other areas of medicine, it is understood that some symptoms correlate to "objective" findings or test results, and some do not, and there is usually a distinction between these. You can imagine something similar in psychiatry, where we ascribe causes to symptoms/syndromes in some cases, but in other cases are not able to do so. This could lead to progress in understanding pathophysiology. Maybe!
I want to suggest a different paradigm. It's weather forecasting, of all things. You generally have a "face" prediction (50% chance of rain today), and then a "technical discussion" where the NWS forecaster talks about all the models, where they differ, and when he's going with his gut.
You kind of want that "technical discussion" to be passed from clinician to clinician.
Ideally, of course, you want predictive "here's what I think will happen" so that you can actually be differentiating between diagnoses. "If he's got anxiety as primary, he will do XYZ in response to going to the anime convention." "If he's got MDD as primary, he will do ABC in response to going to the anime convention."
The way people cope with physical illnesses is more psychology than psychiatry... I'd say there's no evidence for any psychiatric aspects of OSA, it'll all be non evidence based and so small effects they're statistical noise... beyond the obvious consequences of sleepiness...
So what you are saying is that not only should psychiatry be divorced from internal medicine but also psychology? That is the kind of thinking that is the root problem of what I described.
Also OSA causes quite profound effects on the brain through oxygen desaturation and fragmented deep sleep. Whether you label the inevitable symptoms psychiatric or psychological is immaterial to me.
So I can ignore psychiatric diagnoses when I'm reviewing someone's chart? That is a reasonable corollary to your thesis.
Well, some of them.
If the person has irritable bowel syndrome, MDD is a side-effect more probably than a "natural" disorder. Resolve the IBS, and the MDD goes away.
But, by the same token, the MDD -is- diagnostically there, and you've got a much higher probability of it than you do, in say, a cancer patient.
Outstanding summary. Thank you for writing it.
Every so often, that story cycles around about how doctors were unable to apply Bayes' theorem to reason about false positive rates.
That's not really a problem of statistical literacy, it's just basic probability. I'm doubtful that adding adding more *statistics* to the medical curriculum would be very useful at this point in history.
But I think it probably would be good if MDs all knew basic probability in their bones the way a trader or poker player does. Not sure how to make this happen. Having trainees gamble on patient outcomes in the hospital would be very effective but unseemly.
I've been thinking about this. It would be easy enough to put a statistics calculator in an EMR so you could plug in the numbers. I think also changing how we write notes. If you imagine the one-liner/chief complaint as an introduction to the note, part of the expectation could be including at least your initial differential diagnosis, if not numeric probabilities. Some specialties, like Emergency Department, know these numbers pretty well already: 60 yo with this medical probability presenting with chest pain has this likelihood of STEMI, etc.
Having taught FM residents EBM decision making, the issue is actually the attendings. If I am the only attending showing them how to use likelihood ratio to train intuition, and they only work with me once a month they won’t learn it. I am not a big fan of just generating numbers from calculators since people tend to get false certainty from numbers. I used more of green, yellow, red tier system with green being diagnosis certain and treatment clear, yellow diagnosis is somewhat likely but other factors in play so gather information frequently while being cautious with treatment, red diagnosis not clear and need more information before treating or get a consult. Something with a good likelyhood ratio could take from red to green. Intermediate likely hood ratios would need multiple pieces of data pointing the same way. My big problem with the average therapist is lack of critical thinking about the certainty of diagnosis and then with shear amount of time they spend with the patient pigeon hole them into what is going on.
Good point about numbers and false certainty. That seems to be a general pitfall with Bayesian thinking. There also seems to be an issue with EMRs, the "publication bias" I alluded to, where a diagnosis that was intended to be provisional soon becomes "official" when the next provider looks at the problem list.