ARTICLES

Effects of Diagnostic Inclusion Criteria on Prevalence and Population Characteristics in Database Research Mark S. Bauer, M.D., Austin Lee, Ph.D., Christopher J. Miller, Ph.D., Laura Bajor, D.O., Mingfei Li, Ph.D., Robert B. Penfold, Ph.D.

Objectives: Studies of serious mental illnesses that use administrative databases have employed various criteria to establish diagnoses of interest. Several studies have assessed the validity of diagnostic inclusion criteria against research diagnoses. However, no studies have examined the effect of diagnostic inclusion criteria on prevalence and population characteristics across such groups. Methods: Administrative data for 2003–2010 from the Department of Veterans Affairs were used to calculate prevalence rates and assess effects of varying the diagnostic inclusion criteria on population composition for bipolar disorder, schizophrenia, and posttraumatic stress disorder (PTSD). Specifically, for each diagnosis, mutually exclusive subpopulations were compared on the basis of the following inclusion criteria for a given diagnosis: one treatment encounter, two outpatient encounters or one inpatient encounter, and any two encounters. For bipolar disorder and schizophrenia, effects of excluding individuals who had a competing diagnosis of, respectively, schizophrenia or bipolar

Analysis of administrative databases is critically important for disease surveillance, program planning, and health services research (1). Such analyses have the advantage of including larger numbers of participants than is typically feasible in clinical studies. Such studies make use of diagnoses rendered as part of clinical care, which are then secondarily rolled into large data sets for administrative and research purposes. However, diagnoses made during clinical care processes tend to be less accurate than diagnoses made in clinical research protocols (2–5). This may be attributable to differences, for example, in ascertainment method or clinician diagnostic tendencies. Thus analyses of administrative data sets may be sensitive to the rules used to convert these clinical diagnoses to administrative case definitions. There is no universally accepted standard for utilizing encounter- or claims-based diagnoses to establish mental health diagnoses in administrative database analyses, although evidence indicates that multiple service encounters or claims with a given diagnosis are required to provide acceptable predictive values for at least some mental health conditions Psychiatric Services 66:2, February 2015

disorder in the prior 12 months and since 2002 were also determined. Results: In 2010, moving from the broadest definitions of bipolar disorder (N=120,382), schizophrenia (N=91,977), and PTSD (N=554,028) to the most restrictive definitions reduced prevalence rates by, respectively, 28.7%, 34.9%, and 25.7%, with temporal trends for 2003–2010 paralleling results in 2010. Population composition changes with changing diagnostic inclusion criteria were variable, with predominantly small odds ratios. Conclusions: Population composition was relatively robust across common diagnostic inclusion criteria for each condition. Thus choice of criteria can focus on considerations of diagnostic validity and case-finding needs. Three mechanisms for the impact of diagnostic criteria on population composition in administrative data sets are discussed. Psychiatric Services 2015; 66:141–148; doi: 10.1176/appi.ps.201400115

(6–8). Criteria for assigning a particular diagnosis for database studies have varied on several dimensions, including the number of required clinical encounters, whether these encounters were tied to inpatient or outpatient visits, the time frame in which they occurred, and the occurrence of competing diagnoses. For example, some studies have required just one clinical encounter (inpatient or outpatient) to qualify for a diagnosis (9,10), and others have required two such encounters (11). Other studies have required one inpatient encounter (12), and others have required one inpatient or two outpatient visits (6,13,14). Several studies with various methodologies have investigated the validity of diagnostic schemata for some mental health conditions (6–8,15–18). However, we are aware of no studies that have characterized the impact of population definition on prevalence or population composition across several serious mental health conditions in administrative databases. Such data are necessary to assess the impact of decisions related to diagnoses on research and administrative analyses, which is of critical importance for large health care ps.psychiatryonline.org

141

EFFECTS OF DIAGNOSTIC INCLUSION CRITERIA IN DATABASE RESEARCH

systems such as the Department of Veterans Affairs (VA) and health maintenance organizations (19), as well as for commercial insurers, Medicare, and Medicaid. The impact of population definition on case finding and case mix is also of increasing importance to accountable care organizations (20), which are responsible for overall management of beneficiary populations. We therefore utilized a large administrative database from the VA to address two questions. What is the impact of varying inclusion and exclusion diagnostic inclusion criteria on the prevalence of bipolar disorder, schizophrenia, and posttraumatic stress disorder (PTSD)? What is the impact of varying diagnostic inclusion criteria on key demographic and clinical characteristics in these populations? METHODS Population Definitions The VA Central Institutional Review Board approved all study procedures following Declaration of Helsinki principles. We obtained administrative encounter records from the VA Corporate Data Warehouse for fiscal years (FY) 2003–2010. Three populations were compiled for separate analyses: all VA service users who had at least one clinical encounter for a diagnosis of bipolar disorder (ICD-9 codes 296.0, 296.1, and 296.4–296.89, including bipolar type I, type II, and not otherwise specified [NOS]); at least one clinical encounter for a diagnosis of schizophrenia spectrum disorder, including schizoaffective disorder (ICD-9 codes 290.0–295.9); or at least one encounter for a diagnosis of PTSD (ICD-9 code 309.81). To address our first study question, we utilized the above single-encounter inclusion criterion (9,10) as the base case (group A) against which to compare subpopulations by using two more restrictive inclusion criteria sets: at least one inpatient or two outpatient encounters with that diagnosis (6,13,14) in a given year (group B), or at least two inpatient or outpatient encounters with that diagnosis (11) in a given year (group C). In regard to use of competing diagnoses as exclusion criteria, there is well-documented uncertainty about potential diagnostic overlap between bipolar disorder and schizophrenia because of variability in patient presentation, provider diagnostic trends, and intermediate disease states (21,22). We therefore identified individuals who had a diagnosis of bipolar disorder or schizophrenia by using at least one inpatient or two outpatient encounters, respectively, in a given year (group B), without consideration of competing diagnoses (schizophrenia or bipolar disorder, respectively). This population (group B) served as the base group against which we compared subpopulations in which more restrictive criteria were applied—those with no competing diagnoses in the prior 12 months (group D) or those with no competing diagnoses since FY 2002 (group E). [A figure illustrating subpopulation construction is available in an online data supplement to this article.] 142

ps.psychiatryonline.org

Population Characteristics To address our second study question, determining the impact of diagnostic inclusion criteria on case mix, we identified a priori potential correlates by reviewing the literature to identify patient characteristics examined in prior administrative database studies—primarily common psychiatric and general medical comorbidities and specific general medical conditions that may affect treatment choice (for example, metabolic syndrome criteria in studies of antipsychotic use). Clinical diagnoses were considered present if identified on a single encounter within 12 months before the patient met criteria for the index diagnosis (ICD-9 diagnoses available on request). Demographic characteristics included age, gender, racial-ethnic minority status, marital status, and disability status ($50% service-connected VA disability). Psychiatric diagnoses included major depressive disorder with or without psychotic features, alcohol use disorders, drug use disorders, PTSD (for bipolar disorder and schizophrenia), and other anxiety disorders. Medical diagnoses included diabetes, obesity, hyperlipidemia, liver disorder, kidney disorder, thyroid disorder, cardiac dysrhythmia, tobacco use disorder, traumatic brain injury, and sleep disorder. Construction of Mutually Exclusive Subpopulations The subpopulations of interest are subsets of the larger reference population (groups B and C versus group A; groups D and E versus group B). To construct groups of independent observations, we constructed four disjoint sets based on inclusion criteria for each of the three mental health conditions on the basis of the above definitions: group B, members of group A but not group B, group C, and members of group A but not group C. Similarly, to investigate the impact of excluding competing diagnoses for bipolar disorder and schizophrenia, we again constructed four disjoint sets: group D, members of group B but not group D, group E, and members of group B but not group E. Statistical Analyses We first calculated prevalence rates for the three populations— bipolar disorder, schizophrenia, and PTSD—for FY 2003–2010 using the base case and more restrictive definitions. We calculated rates without regard to overlapping membership, because the focus of interest was the overall “yield” according to each definition. We then conducted our primary analyses of population composition for individuals who met diagnostic inclusion criteria for the given diagnosis in FY 2010, with identical analyses run on FY 2005 data to investigate stability of findings. There were no substantive differences, and only the FY 2010 data are presented here. Comparisons focused on determining the odds of having a particular characteristic, given membership in a specific subpopulation compared with the reference population. Analyses were conducted utilizing odds ratios (ORs), which, because of the large sample sizes, were almost all highly statistically significant (p,.001). However, the focus of interest in these analyses was magnitude of effect rather than Psychiatric Services 66:2, February 2015

BAUER ET AL.

statistical significance. We therefore characterized ORs as roughly equivalent to small (.20), medium (.50), and large (.80) effect sizes (23): respectively, $1.46 or #.68; $2.50 or #.40; and $4.14 or #.24 (24).

FIGURE 1. Individuals diagnosed as having bipolar disorder, schizophrenia, or PTSD, by fiscal year and diagnostic inclusion criteria groupa Bipolar disorder A=254,379 B=200,978 C=185,730 D=182,508 E=170,811

Effects of Inclusion Criteria on Prevalence Prevalence over time for the diagnostic inclusion criteria sets for bipolar disorder, schizophrenia, and PTSD in 2003–2010 is summarized in Figure 1. For bipolar disorder, the number of VA service users increased according to all definitions, with the least restrictive group (group A) increasing from 82,131 to 120,382 and the most restrictive group (group E) increasing from 53,587 to 85,856 (p,.001 for each). [A table in the online supplement presents individual time trend statistics.] The 2010 prevalence rates among all VA service users (N=5,536,465) for the least and most restrictive definitions were, respectively, 2.35% and 1.55%. In contrast, the number of service users treated for schizophrenia decreased from 100,853 to 91,977 (group A) and from 74,870 to 59,909 (group E) (p,.002 for each), with 2010 prevalence rates for the least and most restrictive definitions, respectively, of 1.66% and 1.08%. The PTSD population grew substantially, from 244,930 to 554,028 (group A) and from 180,261 to 411,526 (group E) (p,.001 for each), with 2010 prevalence rates for the least and most restrictive definitions, respectively, of 9.83% and 7.43%. In FY 2010, moving from the least (group A) to the most (group E) restrictive definition for bipolar disorder, schizophrenia, and PTSD reduced the number of individuals by 28.7%, 34.9%, and 25.7%, respectively.

120 110 100 90 80 70 60 50 40 2003

2004

Psychiatric Services 66:2, February 2015

2006

2007

2008

2009

2010

2008

2009

2010

2008

2009

2010

A=186,468 B=155,753 C=146,747 D=141,767 E=126,243

130 120 110 100 90 80 70 60 50 40 2003

2004

2005

2006

2007

PTSD

N of patients (thousands)

600

Effects of Inclusion Criteria on Population Characteristics Demographic and clinical characteristics of the bipolar, schizophrenia, and PTSD populations in FY 2010 (group A) are summarized in Table 1. Table 2 compares the effects on population composition of applying more restrictive definitions (group B or C) to the single-encounter base case (respectively, group B versus group A-not-B, and group C versus group A-not-C) for each disorder. Only 22 of 132 (17%) ORs reached even small effect sizes. Moreover, there were few differences in relative effects between the two more restrictive definitions (ORs for group B versus group A-not-B compared with ORs for group C versus group A-not-C). Only the percentage of individuals with PTSD who had a $50% service-connected disability reached a consistent medium effect size (ORs=2.55 and 2.58). Only the frequency of diagnosis of major depression with or without psychotic features was sensitive to change of diagnostic inclusion criteria across all three populations, with decreasing frequency in the bipolar disorder population (ORs=.55–.76) and schizophrenia population (ORs=.52–.65) and increasing frequency in the PTSD population (ORs=1.71–2.50).

2005

Schizophrenia

N of patients (thousands)

RESULTS

N of patients (thousands)

130

A=924,334 B=755,127 C=737,272 D=711,536 E=684,126

550 500 450 400 350 300 250 200 150 100 2003

2004

2005

2006

2007

Fiscal year a

Group A, one treatment encounter; group B, at least one inpatient or two outpatient encounters with the given diagnosis in a given year; group C, at least two inpatient or outpatient encounters with the given diagnosis in a given year; group D, no competing diagnoses in the prior 12 months; group E, no competing diagnoses since fiscal year (FY) 2002. The number listed for each group indicates total number of unique participants overall from FY 2003 to 2010.

Effects of Competing Diagnoses: Bipolar Disorder and Schizophrenia Table 3 summarizes the effects on population composition for bipolar disorder and schizophrenia of requiring no competing ps.psychiatryonline.org

143

EFFECTS OF DIAGNOSTIC INCLUSION CRITERIA IN DATABASE RESEARCH

TABLE 1. Characteristics of patients meeting the least restrictive inclusion criteria (group A) for three diagnoses in fiscal year 2010a Bipolar disorder (N=120,382) Characteristic

b

Age Male White Married $50% service-connected VA disability Mental health diagnosis PTSD Other anxiety disorder Alcohol use disorder Drug use disorder Major depression (without psychosis) Major depression (with psychosis) General medical diagnosis Hypertension Diabetes Obesity Hyperlipidemia Tobacco use disorder Cardiac dysrhythmia Liver disorder Kidney disorder Thyroid disorder Traumatic brain injury Sleep disorder a b

N

Schizophrenia (N=91,977)

PTSD (N=554,028)

%

N

%

N

%

53.0613.0 101,756 89,146 42,995 40,443

84.5 80.5 36.0 33.6

57.0611.5 85,714 50,149 21,374 45,346

93.2 58.0 23.5 49.5

54.8614.9 511,270 361,033 306,458 324,219

92.3 73.2 56.0 58.5

33,126 26,479 25,829 24,302 43,721 1,983

27.5 22.0 21.5 20.2 36.3 1.6

13,458 10,446 15,079 15,233 21,369 2,428

14.6 11.4 16.4 16.6 23.2 2.6

— 105,838 83,886 53,324 260,106 7,625

— 19.1 15.1 9.6 46.9 1.4

55,030 24,746 26,143 56,022 40,194 5,392 9,992 5,861 10,655 2,360 7,948

45.7 20.6 21.7 46.5 33.4 4.5 8.3 4.9 8.9 2.0 6.6

46,876 23,946 20,047 43,507 33,431 3,900 8,571 5,398 6,430 1,103 3,193

51.0 26.0 21.8 47.3 36.3 4.2 9.3 5.9 7.0 1.2 3.5

278,270 127,885 107,009 272,216 135,478 26,765 29,174 25,101 28,428 20,361 44,490

50.2 23.1 19.3 49.1 24.5 4.8 5.3 4.5 5.1 3.7 8.0

The least restrictive definition: at least one clinical encounter with the indicated diagnosis. For white, married, and disability variables, the percentage of missing values for bipolar were 8.0%, .8%, and .1%, respectively; for schizophrenia, they were 5.9%, 1.0%, and .4%, and for PTSD, they were 11%, 1.2%, and .4%, respectively.

diagnoses of the other disorder, either in the prior 12 months (group D versus group B-not-D) or since 2002 (group E versus group B-not-E). Analyses revealed ORs of small effect in 37 of 88 (42%) comparisons, with no ORs indicating a medium effect. There were few differences in relative effects between the two more restrictive definitions (ORs for group D versus group B-not-D, compared with ORs for group E versus group B-not-E). Small, same-direction ORs across the two conditions were seen for drug use disorders and major depression with psychosis. Reciprocal differences in ORs across the two populations appeared in the percentage of white individuals when the definition of bipolar disorder was restricted to those without a competing diagnosis of schizophrenia (ORs=2.00–2.03) and when the definition of schizophrenia was restricted to those without a competing diagnosis of bipolar disorder (ORs=.60–.65). Similar, but somewhat smaller, consistent reciprocal effects were seen in the percentage of males when more restrictive definitions of bipolar disorder were used (ORs=.66–.73) and schizophrenia (ORs=1.88–1.89). DISCUSSION Effects of Criteria on Population Prevalence To our knowledge, this is the only study of the effects of varying administrative data set diagnostic inclusion criteria across several serious mental health conditions on resulting population composition, an issue of clear relevance both to 144

ps.psychiatryonline.org

health services research and administrative analyses. Comparing single-encounter definitions for bipolar disorder, schizophrenia, and PTSD to more restrictive definitions, we found expected reductions in population size. The most restrictive definitions for these conditions reduced population size in 2010 by 25.7%234.9%. Though time trends were not the focus of this study, the prevalence differences over time were roughly consistent over the study years. The substantial increase in PTSD prevalence over time among VA service users is consistent with other reported data (25). However, we are not aware of data comparing the relative prevalence of bipolar disorder and schizophrenia, which were notably divergent over time for all definitions. The relative increase in diagnosis of bipolar disorder and decrease in schizophrenia among VA service users may be attributable to several factors. First, excess mortality among persons with schizophrenia compared with the general population has been well documented (26), although this rate may (27) or may not (28) exceed that for bipolar disorder. Second, the prevalence of the two disorders may be changing as a result of changes in the rate of incident cases. Comparison with U.S. community samples may be instructive, although prevalence rates in the population of VA service users are not necessarily expected to replicate those of the general population (29), and within-study longitudinal prevalence data are surprisingly scant. Although the National Comorbidity Survey (NCS) documented 12-month prevalence of 1.3% for mania and .5% for nonaffective psychoses (30), the NCS Psychiatric Services 66:2, February 2015

BAUER ET AL.

TABLE 2. Effects of using more restrictive diagnostic inclusion criteria on population characteristics for three diagnosesa Bipolar disorder group A (N=120,382)

Characteristic Ageb Male White Married $50% service-connected VA disability Mental health diagnosis PTSD Other anxiety disorder Alcohol use disorder Drug use disorder Major depression (without psychosis) Major depression (with psychosis) General medical diagnosis Hypertension Diabetes Obesity Hyperlipidemia Tobacco use disorder Cardiac dysrhythmia Liver disorder Kidney disorder Thyroid disorder Traumatic brain injury Sleep disorder

Schizophrenia group A (N=91,977)

PTSD group A (N=554,028)

Group B vs. A-not-B (104,168 vs. 16,214)

Group C vs. A-not-C (100,801 vs. 19,581)

Group B vs. A-not-B (83,689 vs. 8,288)

Group C vs. A-not-C (81,763 vs. 10,214)

Group B vs. A-not-B (474,869 vs. 79,159)

Group C vs. A-not-C (469,646 vs. 84,382)

.07 .88 1.31 .90 1.27

.07 .82 1.39 .99 1.21

.01 1.07 1.06 .74 1.75c

–.03 1.05 1.01 .81 1.81c

.11 1.03 1.04 1.11 2.55d

.10 1.02 1.05 1.17 2.58d

.99 1.04 1.36 1.36 .75 .76

.92 1.00 1.01 1.00 .68c .55c

.71 .78 1.11 1.11 .65c .63c

.68c .75 .89 .90 .60c .52c

— 1.31 1.47c 1.63c 1.78c 2.50d

— 1.29 1.26 1.32 1.71c 1.93c

1.19 1.22 1.32 1.32 1.35 1.30 1.27 1.42 1.48c 1.05 1.03

1.09 1.13 1.31 1.29 1.15 1.02 1.00 .99 1.41 .85 .96

1.26 1.33 1.37 1.42 1.68c 1.13 1.15 1.29 1.32 .79 .82

1.16 1.27 1.43 1.45 1.56c .77 .97 .87 1.16 .70 .75

1.33 1.33 1.33 1.42 1.28 1.22 1.41 1.28 1.20 1.83c 1.21

1.27 1.28 1.33 1.40 1.18 1.05 1.20 1.05 1.16 1.79c 1.20

a

Values are odds ratios (ORs), except for the values for age, which are Cohen’s d. Group A, one treatment encounter; group B, at least one inpatient or two outpatient encounters for the given diagnosis in a given year; group C, at least two inpatient or outpatient encounters for the given diagnosis in a given year b For age, negative effect size indicates age was greater in the reference group (group A-not-B or group A-not C); positive effect size indicates age was greater in more restrictive group (group B or C). c ORs corresponding to small effect sizes d ORs corresponding to medium effect sizes

replication documented a prevalence of 1.1% for bipolar I disorder but did not assess nonaffective psychoses (31). In contrast, a community study of bipolar disorder in Australia described an up-to-twofold increase in bipolar disorder between 1998 and 2008 (32). Third, trends in diagnostic practice, independent of true prevalence, may also be contributing factors. Some have bemoaned a trend to “overdiagnosis” of milder forms of bipolar disorder as a result of multiple factors (33). Others have pointed out that diagnosis of milder forms of bipolar disorder identifies a high-morbidity subpopulation that warrants attention (31). Our data indicate that the proportion of patients with milder bipolar disorder (type II or NOS) increased from 8.7% in 2003 to 24.4.6% in 2010 for group A; however, this growth accounted for only a minority of cases and cannot completely explain the rise in bipolar disorder prevalence. Effects of Criteria on Population Composition We found that increasing the restrictiveness of the definitions resulted in small to medium effects in only a minority of contrasts, and most of these were not consistent across populations. Thus population composition for all three mental Psychiatric Services 66:2, February 2015

health conditions was relatively robust across commonly used diagnostic inclusion criteria. This does not, however, imply that there are no meaningful effects of increasing diagnostic restrictiveness or that investigators and administrators should utilize the least restrictive, single-encounter definition to maximize case finding (6–8). Rather, these findings demonstrate predominantly small impacts on specific population characteristics of selecting diagnostic inclusion criteria, which may or may not be important in specific analyses. If maximizing case finding is a priority, the least restrictive definition might be most appropriate, and investigators can expect few shifts in population composition when more restrictive definitions are applied in subsequent database analyses. On the other hand, if more restrictive definitions are used to maximize predictive value, our analyses indicate it is unlikely that the investigators’ choice would substantively impact the population composition compared with less restrictive definitions. Three Mechanisms of Effect When differences in population characteristics are seen across diagnostic inclusion criteria sets, they may be attributable to ps.psychiatryonline.org

145

EFFECTS OF DIAGNOSTIC INCLUSION CRITERIA IN DATABASE RESEARCH

TABLE 3. Effects of excluding competing diagnoses on population characteristics for bipolar disorder and schizophreniaa Bipolar disorder group B (N=104,168)

Characteristic Ageb Male White Married $50% service-connected VA disability Mental health diagnosis PTSD Other anxiety disorder Alcohol use disorder Drug use disorder Major depression (without psychosis) Major depression (with psychosis) General medical diagnosis Hypertension Diabetes Obesity Hyperlipidemia Tobacco use disorder Cardiac dysrhythmia Liver disorder Kidney disorder Thyroid disorder Traumatic brain injury Sleep disorder

Schizophrenia group B (N=83,689)

Group D vs. B-not-D (95,883 vs. 8,285)

Group E vs. B-not-E (85,856 vs. 18,312)

Group D vs. B-not-D (75,555 vs. 8,134)

Group E vs. B-not-E (59,909 vs. 23,780)

–.07 .66c 2.03c 1.84c .56c

–.15 .73 2.00c 1.77c .59c

.31c 1.89c .57c .96 1.30

.25c 1.88c .64c 1.12 1.10

1.45 1.33 .76 .65c 1.31 .41c

1.22 1.34 .78 .65c 1.36 .42c

.55c .54c .51c .49c .67c .66c

.63c .66c .56c .51c .81 .75

.76 .67c .71 .86 .63c .85 .74 .61c .80 1.05 1.12

.75 .69 .74 .86 .68 .86 .68c .60c .78 1.19 1.24

1.01 .99 .74 .94 .77 .79 .84 .75 .60c .66c .50c

.97 .94 .74 .92 .72 .90 .79 .84 .60c .69 .63c

a

Values are odds ratios (ORs), except for the values for age, which are Cohen’s d. Group B, at least one inpatient or two outpatient encounters for the given in a given year; group D, no competing diagnoses in the prior 12 months; group E, no competing diagnoses since fiscal year 2002. b For age, negative effect size indicates age was greater in the reference group (group B-not-D or group B-not-E); positive effect size indicates age was greater in more restrictive group (group D or E). c ORs corresponding to small effect sizes

one or more of at least three mechanisms: real differences in population composition, differences in clinician diagnostic tendencies, or differences in ascertainment. For example, the reciprocal changes in the percentage of males found when bipolar disorder and schizophrenia were defined more restrictively (Table 3) correspond to welldocumented gender differences in clinical studies of the two disorders—that is, the proportion of women is larger among persons with bipolar disorder than among those with schizophrenia (34). That is, when more restrictive definitions remove borderline cases, the gender differences move in the direction predicted by clinical samples: increasing the proportion of women in the bipolar disorder population and the proportion of men in the schizophrenia population. The reciprocal differences in the proportion of whites as the diagnosis becomes more restrictive may also be attributable to underlying population differences. However, the differences may also result from the well-documented tendency of clinicians to diagnose schizophrenia more readily than bipolar disorder among individuals from racial-ethnic minority groups (35,36). Finally, ascertainment effects are also relevant whenever comorbidity rates are considered: increased diagnostic rates may be linked to increasing surveillance of those assessed more frequently or comprehensively. Despite these likely effects, 146

ps.psychiatryonline.org

there were remarkably small, inconsistent differences in the odds of comorbidities across inclusion criteria for any of the three populations (Table 2). However, it is instructive to consider the effect on major depressive disorder of changing the inclusion criteria for each of these disorders, which illustrates the potential complexity of using administrative data to extrapolate to true clinical characteristics of the underlying population. Specifically, the increased odds of major depression with increasing restrictiveness of diagnosis of PTSD (ORs=1.71–2.50) may reflect a true difference among those with more certain diagnoses of PTSD or may reflect more extensive care; however, it may also be that those who are treated more frequently undergo greater surveillance for other disorders, with major depression correspondingly diagnosed more frequently because of increased surveillance rather than because of true increased prevalence. In contrast, the decreasing odds of major depression with increasingly restrictive criteria for bipolar disorder (ORs=.55–.76) and schizophrenia (ORs=.52–.65), although similar in magnitude, may be due to different mechanisms. An ICD-9 diagnosis of major depression is incompatible with bipolar disorder, and thus it is likely that clinicians, appropriately, attribute depressive symptoms to bipolar disorder among persons with more a certain bipolar diagnosis. For schizophrenia, Psychiatric Services 66:2, February 2015

BAUER ET AL.

however, the diagnoses are not incompatible, and the lower rates of major depression diagnoses may reflect a “negative” ascertainment effect, where symptoms of depression are ascribed to schizophrenia in clearer cases of the latter. Limitations These analyses were affected by the limitations of any study of administrative data sets, which rely on diagnoses made by clinicians rather than on research diagnoses (1). Thus differences in prevalence and sample composition cannot be ascribed to underlying differences in the population rather than to differences in diagnostic tendencies of clinicians. In addition, lack of differences does not necessarily increase confidence that the populations are similar—again because of the reliance on clinician diagnosis. Estimates of the validity and reliability of administrative diagnoses are not possible without research diagnoses. However, the purpose of these analyses was not to establish validity, reliability, or relative scientific merit of the diagnostic inclusion criteria but rather, working within administrative data sets, to determine the impact of typical diagnostic rules on prevalence and population composition. These analyses were also limited to VA service users, and it is likely that the impacts of choosing diagnostic inclusion criteria may differ in other care settings (37). However, the VA may also serve as a “microcosm” (29), which can inform methods used in other health care systems. Finally, findings may differ for other mental health conditions such as major depression or other anxiety disorders. CONCLUSIONS These findings can guide analyses of data from VA, health maintenance organizations, commercial insurers, Medicare, Medicaid, and accountable care organizations. Investigators can make decisions about diagnostic inclusion and exclusion criteria with the knowledge that population composition in all three mental health conditions is relatively stable across commonly used diagnostic criteria. An awareness of three potential mechanisms that produce differences in population composition across inclusion criteria can help guide investigators in their choices. AUTHOR AND ARTICLE INFORMATION Dr. Bauer is with the Center for Healthcare Organization and Implementation Research, Department of Veterans Affairs (VA), Jamaica Plain, Massachusetts, and with the Department of Psychiatry, Harvard Medical School, Boston (e-mail: [email protected]). Dr. Lee is with the Department of Surgery, Massachusetts General Hospital, Boston. Dr. Miller is with Center for Healthcare Organization and Implementation Research and Dr. Bajor is with the Center for Organization, Leadership, and Management Research, both at VA Boston Healthcare System, Boston. Dr. Bajor is also with the Department of Psychiatry, Harvard Medical School, Boston. Dr. Li is with the Department of Mathematical Sciences, Bentley University, Waltham, Massachusetts. Dr. Penfold is with the Department of Health Services Research, Group Health Research Institute, Seattle, Washington. This material is based on work supported by Health Services Research and Development grant IIR-10-314 (to Dr. Bauer) from the Office of Psychiatric Services 66:2, February 2015

Research and Development, Veterans Health Administration, U.S. Department of Veterans Affairs. The funding organization provided competitive grant support for these analyses but was not otherwise involved in the design or conduct of the study or review or approval of the manuscript. The authors report no financial relationships with commercial interests.

REFERENCES 1. Alaghehbandan R, MacDonald D: Use of administrative health databases and case definitions in surveillance of depressive disorders: a review. OA Epidemiology 1:3, 2013 2. Zimmerman M, Mattia JI: Psychiatric diagnosis in clinical practice: is comorbidity being missed? Comprehensive Psychiatry 40:182–191, 1999 3. Miller PR: Inpatient diagnostic assessments: 2. interrater reliability and outcomes of structured vs unstructured interviews. Psychiatry Research 105:265–271, 2001 4. Miller PR, Dasher R, Collins R, et al: Inpatient diagnostic assessments: 1. accuracy of structured vs unstructured interviews. Psychiatry Research 105:255–264, 2001 5. Moilanen K, Veijola J, Läksy K, et al: Reasons for the diagnostic discordance between clinicians and researchers in schizophrenia in the Northern Finland 1966 Birth Cohort. Social Psychiatry and Psychiatric Epidemiology 38:305–310, 2003 6. Solberg LI, Engebretson KI, Sperl-Hillen JM, et al: Are claims data accurate enough to identify patients for performance measures or quality improvement? The case of diabetes, heart disease, and depression. American Journal of Medical Quality 21:238–245, 2006 7. Frayne SM, Miller DR, Sharkansky EJ, et al: Using administrative data to identify mental illness: what approach is best? American Journal of Medical Quality 25:42–50, 2010 8. Gravely AA, Cutting A, Nugent S, et al: Validity of PTSD diagnoses in VA administrative data: comparison of VA administrative PTSD diagnoses to self-reported PTSD Checklist scores. Journal of Rehabilitation Research and Development 48:21–30, 2011 9. Sajatovic M, Blow FC, Kales HC, et al: Age comparison of treatment adherence with antipsychotic medications among individuals with bipolar disorder. International Journal of Geriatric Psychiatry 22:992–998, 2007 10. Rosenheck R, Fontana A: Use of mental health services by veterans with PTSD after the terrorist attacks of September 11. American Journal of Psychiatry 160:1684–1690, 2003 11. Lambert BL, Cunningham FE, Miller DR, et al: Diabetes risk associated with use of olanzapine, quetiapine, and risperidone in Veterans Health Administration patients with schizophrenia. American Journal of Epidemiology 164:672–681, 2006 12. Kales HC, Blow FC, Bingham CR, et al: Race and inpatient psychiatric diagnoses among elderly veterans. Psychiatric Services 51: 795–800, 2000 13. Soumerai SB, McLaughlin TJ, Ross-Degnan D, et al: Effects of a limit on Medicaid drug-reimbursement benefits on the use of psychotropic agents and acute mental health services by patients with schizophrenia. New England Journal of Medicine 331:650–655, 1994 14. Bauer MS, Lee A, Li M, et al: Off-label use of second generation antipsychotics for post-traumatic stress disorder in the Department of Veterans Affairs: time trends and sociodemographic, comorbidity, and regional correlates. Pharmacoepidemiology and Drug Safety 23: 77–86, 2014 15. Rawson NS, Malcolm E, D’Arcy C: Reliability of the recording of schizophrenia and depressive disorder in the Saskatchewan health care datafiles. Social Psychiatry and Psychiatric Epidemiology 32: 191–199, 1997 16. West SL, Richter A, Melfi CA, et al: Assessing the Saskatchewan database for outcomes research studies of depression and its treatment. Journal of Clinical Epidemiology 53:823–831, 2000 ps.psychiatryonline.org

147

EFFECTS OF DIAGNOSTIC INCLUSION CRITERIA IN DATABASE RESEARCH

17. Spettell CM, Wall TC, Allison J, et al: Identifying physicianrecognized depression from administrative data: consequences for quality measurement. Health Services Research 38:1081–1102, 2003 18. Kisely S, Lin E, Lesage A, et al: Use of administrative data for the surveillance of mental disorders in 5 provinces. Canadian Journal of Psychiatry 54:571–575, 2009 19. Copeland LA, Zeber JE: Advancing research in the era of healthcare reform: the 19th annual HMO Research Network Conference, April 16–18, 2013, San Francisco, California. Clinical Medicine and Research 11:120–122, 2013 20. Fisher ES, McClellan MB, Bertko J, et al: Fostering accountable health care: moving forward in Medicare. Health Affairs 28:w219–w231, 2009 21. Ketter TA, Wang PW, Becker OV, et al: Psychotic bipolar disorders: dimensionally similar to or categorically different from schizophrenia? Journal of Psychiatric Research 38:47–61, 2004 22. Maier W, Zobel A, Wagner M: Schizophrenia and bipolar disorder: differences and overlaps. Current Opinion in Psychiatry 19:165–170, 2006 23. Cohen J: Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ, Erlbaum, 1988 24. Chen H, Cohen P, Chen S: How big is a big odds ratio? Interpreting the magnitudes of odds ratios in epidemiological studies. Communications in Statistics Simulation and Computation 39:860–864, 2010 25. Bernardy NC, Lund BC, Alexander B, et al: Prescribing trends in veterans with posttraumatic stress disorder. Journal of Clinical Psychiatry 73:297–303, 2012 26. Saha S, Chant D, McGrath J: A systematic review of mortality in schizophrenia: is the differential mortality gap worsening over time? Archives of General Psychiatry 64:1123–1131, 2007 27. Laursen TM: Life expectancy among persons with schizophrenia or bipolar affective disorder. Schizophrenia Research 131:101–104, 2011

148

ps.psychiatryonline.org

28. Castagnini A, Foldager L, Bertelsen A: Excess mortality of acute and transient psychotic disorders: comparison with bipolar affective disorder and schizophrenia. Acta Psychiatrica Scandinavica 128: 370–375, 2013 29. Morgan RO, Teal CR, Reddy SG, et al: Measurement in Veterans Affairs Health Services Research: veterans as a special population. Health Services Research 40:1573–1583, 2005 30. Kessler RC, McGonagle KA, Zhao S, et al: Lifetime and 12-month prevalence of DSM-III-R psychiatric disorders in the United States: results from the National Comorbidity Survey. Archives of General Psychiatry 51:8–19, 1994 31. Merikangas KR, Akiskal HS, Angst J, et al: Lifetime and 12-month prevalence of bipolar spectrum disorder in the National Comorbidity Survey Replication. Archives of General Psychiatry 64: 543–552, 2007 32. Zutshi A, Eckert KA, Hawthorne G, et al: Changes in the prevalence of bipolar disorders between 1998 and 2008 in an Australian population. Bipolar Disorders 13:182–188, 2011 33. Mitchell PB: Bipolar disorder: the shift to overdiagnosis. Canadian Journal of Psychiatry 57:659–665, 2012 34. Diagnostic and Statistical Manual of Mental Disorders, 5th ed. Arlington, Va, American Psychiatric Association, 2013 35. Mukherjee S, Shukla S, Woodle J, et al: Misdiagnosis of schizophrenia in bipolar patients: a multiethnic comparison. American Journal of Psychiatry 140:1571–1574, 1983 36. Strakowski SM, Shelton RC, Kolbrener ML: The effects of race and comorbidity on clinical diagnosis in patients with psychosis. Journal of Clinical Psychiatry 54:96–102, 1993 37. Folsom DP, Lindamer L, Montross LP, et al: Diagnostic variability for schizophrenia and major depression in a large public mental health care system dataset. Psychiatry Research 144:167–175, 2006

Psychiatric Services 66:2, February 2015

Effects of diagnostic inclusion criteria on prevalence and population characteristics in database research.

Studies of serious mental illnesses that use administrative databases have employed various criteria to establish diagnoses of interest. Several studi...
636KB Sizes 1 Downloads 5 Views