AJPH PERSPECTIVES
likely to die in the next five years than were their sampled male peers who did not participate. Again, applied to these selective data, the approach of Masters et al. was not able to recover the “lower-at-older-ages” hazard ratio pattern present in the full data (Figure C and Appendix 4). The greater the selectivity I introduced, the more the corrected age-specific male-tofemale hazard ratios overshot the target at older ages and undershot it at lower ages. The Appendices 1, 2, and 4 contains several implicit pleas to authors. I end with two explicit ones: that new ways to directly correct heretofore uncorrectable biases first be tested on simulated data generated by
known parameter values and that reported model-based corrections not be so drastic that (as with the life insurance premiums) they seemingly correct the problem at one end of the age scale by creating one at the other end. I also make a plea to editors: instead of asking authors to report what software—and what version—they used to prepare the data and derive the reported results, might they ensure instead that the computer code used is publicly available? REFERENCES 1. Masters RK, Reither EN, Powers DA, Yang YC, Burger AE, Link BG. The impact of obesity on US mortality levels: the importance of age and cohort factors
in population estimates. Am J Public Health. 2013;103(10): 1895–1901. 2. Masters RK, Powers DA, Link BG. Obesity and US mortality risk over the adult life course. Am J Epidemiol. 2013;177(5):431–442. 3. Chapter 11: selection bias. In: Kleinbaum DG, Kupper LL, Morgenstern H. Epidemiologic Research: Principles and Quantitative Methods, Chapter 11. Belmont, CA: Lifetime Learning Publications; 1982:194–219. 4. Masters RK, Powers DA, Link BG. The authors reply. [letter] Am J Epidemiol. 2014;179(4):530–532. 5. Wang Z, Liu M. Obesity-mortality association with age: wrong conclusion based on calculation error. [letter] Am J Public Health. 2014;104(7):e3–e4. 6. Galea S, Tracy M. Participation rates in epidemiologic studies. Ann Epidemiol. 2007;17(9):643–653.
Masters et al. Respond Ryan K. Masters, PhD, Daniel A. Powers, PhD, Eric N. Reither, PhD, Y. Claire Yang, PhD, and Bruce G. Link, PhD
This article was jointly published in the American Journal of Epidemiology (Am J Epidemiol. 2017;185(6):412–413) and the AJPH (Am J Pub Health. 2017;107(4):505–506).
A
valuable role that journal editors can play is to create a level playing field where debates can be aired and the scientific merits of the issues can be judged by the full scientific community. It is rare for editors to bypass this step and render their own judgments, which is what happened in this case. Editors of the AJPH and the American Journal of Epidemiology have decided our approach is seriously flawed.1,2 This decision was based largely on Dr. Hanley’s assessment,3,4 which they solicited and which was neither externally nor anonymously peer reviewed. The Editors have allowed us only 600 words to defend our peer-reviewed articles5,6 in response to their editorial, Dr. Hanley’s solicited commentary, and his extensive Appendices. We do not have space here to fully describe our position in response to these statements, and we therefore invite interested readers to examine our responses in our Appendix (available as a supplement to the online version of this article at http://www.ajph.org). Our articles represented earnest efforts to address selection biases in survey-based estimates of the obesity–mortality association. We proposed an alternative to the standard method because that method completely ignores these biases. Dr. Hanley’s simulations convinced him that our approach creates more problems than it
April 2017, Vol 107, No. 4
AJPH
solves. We respect Dr. Hanley’s effort and acknowledge that it is an important part of the scientific enterprise—a strong and wellreasoned challenge. In response to Hanley’s challenge, we wrote an evidence-based rebuttal that the Editors of the AJPH (then the American Journal of Public Health) declined to publish. The essence of our response was that Hanley’s simulation assumptions inaccurately reflected the full scale of the selection biases that affect the obesity– mortality association in data from the National Health Interview Survey (NHIS). Nonetheless, we refitted our survival models, taking into account Dr. Hanley’s concerns. Results from these new analyses were consistent with those from our original articles—namely, that apparent age-related declines in the obesity–mortality association strongly reflect selection bias.
Furthermore, we showed that the approach used in our articles corrected this bias in NHIS data and provided accurate estimates of true male–female mortality hazard ratios in official US mortality data. We did this to counter Dr. Hanley’s test of our approach, in which he used known male–female hazard ratios but simulated a selection pattern that was not observed in the NHIS or the National Health and Nutrition Examination Surveys. Taken together, our analyses show that (1) Hanley’s simulation bears little resemblance to real survey data and (2) our approach provides accurate estimates of known hazard ratios using data from the NHIS and National Health and Nutrition Examination Surveys. For our complete response, please see the Appendix. For us, the most critical issue remains the strong likelihood that uncorrected survey
ABOUT THE AUTHORS Ryan K. Masters is with the Department of Sociology, University of Colorado Boulder, Boulder. Daniel A. Powers is with the Department of Sociology, College of Liberal Arts, University of Texas at Austin, Austin. Eric N. Reither is with the Department of Sociology, Social Work, and Anthropology, Utah State University, Logan. Y. Claire Yang is with the Department of Sociology and Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill. Bruce G. Link is with the Department of Sociology and Public Policy, University of California-Riverside. Correspondence should be sent to Ryan K. Masters, Department of Sociology, UCB 327, Ketchum Hall 264, University of Colorado Boulder, Boulder, CO 80309 (e-mail: [email protected]). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This editorial was accepted January 17, 2017. doi: 10.2105/AJPH.2017.303715
Masters et al.
Editorial
505
AJPH PERSPECTIVES
estimates of the obesity–mortality relationship are biased. Hanley’s own simulations show that hazard ratios estimated from a conventional approach “are quite a bit lower than those in the unselected (‘population’) ones, particularly at older ages” (Appendix 4 of Hanley3,4). In other words, according to Hanley, any approach that fails to address sample selection will produce biased estimates. Furthermore, when sample selection is not addressed in NHIS data, the results indicate that overweight and obesity reduced mortality in the US adult population by nearly 10% between 1986 and 2006 (see Appendix). This patently absurd finding shows that the statistical cure is not worse than the bias, as Hanley alleges. It also underscores our central point that conventional approaches to estimating the obesity–mortality association are seriously flawed, with potentially devastating consequences for public health policies that to date have not addressed the obesity epidemic with sufficient urgency. The editorial1,2 is, in our opinion, remiss in its failure to situate the issue in this broader framework and to recognize the challenge that lies before us all as we seek to understand the mortality effects of obesity.
the AJPH and the two methodologists at the AJE who reviewed it did so very carefully and had no conflicts of interest. Their impeccable reputations convinced us of the accuracy of Hanley’s concerns. In addition, as part of an overall assessment of the case, the commentary by Hanley was reviewed by an external and independent Advisory Ethical Committee convened by the AJPH that comprised three public health scientists whose recommendations were followed.
REFERENCES 1. Morabia A, Szklo M, Vaughan R. Editorial: note about inaccurate results published in the American Journal of Epidemiology and the American Journal of Public Health. Am J Public Health. 2017;107(4):502. 2. Morabia A, Szklo M, Vaughan R, et al. Editorial: note about inaccurate results published in the American Journal of Epidemiology and the American Journal of Public Health. Am J Epidemiol. 2017;185(6):407–408. 3. Hanley J. Correction of selection bias in survey data: is the statistical cure worse than the bias? Am J Epidemiol. 2017;185(6):409–411. 4. Hanley J. Correction of selection bias in survey data: is the statistical cure worse than the bias? Am J Public Health. 2017;107(4):503–505. 5. Masters RK, Powers DA, Link BG. Obesity and US mortality risk over the adult life course. Am J Epidemiol. 2013;177(5):431–442. 6. Masters RK, Reither EN, Powers DA, Yang YC, Burger AE, Link BG. The impact of obesity on US mortality levels: the importance of age and cohort factors in population estimates. Am J Public Health. 2013;103(10):1895–1901.
EDITORS’ NOTE Although Masters et al. are correct in saying that the commentary by Hanley was not anonymously reviewed, the methodologist at
506
Editorial
Masters et al.
AJPH
April 2017, Vol 107, No. 4
likely to die in the next five years than were their sampled male peers who did not participate. Again, applied to these selective data, the approach of Masters et al. was not able to recover the “lower-at-older-ages” hazard ratio pattern present in the full data (Figure C and Appendix 4). The greater the selectivity I introduced, the more the corrected age-specific male-tofemale hazard ratios overshot the target at older ages and undershot it at lower ages. The Appendices 1, 2, and 4 contains several implicit pleas to authors. I end with two explicit ones: that new ways to directly correct heretofore uncorrectable biases first be tested on simulated data generated by
known parameter values and that reported model-based corrections not be so drastic that (as with the life insurance premiums) they seemingly correct the problem at one end of the age scale by creating one at the other end. I also make a plea to editors: instead of asking authors to report what software—and what version—they used to prepare the data and derive the reported results, might they ensure instead that the computer code used is publicly available? REFERENCES 1. Masters RK, Reither EN, Powers DA, Yang YC, Burger AE, Link BG. The impact of obesity on US mortality levels: the importance of age and cohort factors
in population estimates. Am J Public Health. 2013;103(10): 1895–1901. 2. Masters RK, Powers DA, Link BG. Obesity and US mortality risk over the adult life course. Am J Epidemiol. 2013;177(5):431–442. 3. Chapter 11: selection bias. In: Kleinbaum DG, Kupper LL, Morgenstern H. Epidemiologic Research: Principles and Quantitative Methods, Chapter 11. Belmont, CA: Lifetime Learning Publications; 1982:194–219. 4. Masters RK, Powers DA, Link BG. The authors reply. [letter] Am J Epidemiol. 2014;179(4):530–532. 5. Wang Z, Liu M. Obesity-mortality association with age: wrong conclusion based on calculation error. [letter] Am J Public Health. 2014;104(7):e3–e4. 6. Galea S, Tracy M. Participation rates in epidemiologic studies. Ann Epidemiol. 2007;17(9):643–653.
Masters et al. Respond Ryan K. Masters, PhD, Daniel A. Powers, PhD, Eric N. Reither, PhD, Y. Claire Yang, PhD, and Bruce G. Link, PhD
This article was jointly published in the American Journal of Epidemiology (Am J Epidemiol. 2017;185(6):412–413) and the AJPH (Am J Pub Health. 2017;107(4):505–506).
A
valuable role that journal editors can play is to create a level playing field where debates can be aired and the scientific merits of the issues can be judged by the full scientific community. It is rare for editors to bypass this step and render their own judgments, which is what happened in this case. Editors of the AJPH and the American Journal of Epidemiology have decided our approach is seriously flawed.1,2 This decision was based largely on Dr. Hanley’s assessment,3,4 which they solicited and which was neither externally nor anonymously peer reviewed. The Editors have allowed us only 600 words to defend our peer-reviewed articles5,6 in response to their editorial, Dr. Hanley’s solicited commentary, and his extensive Appendices. We do not have space here to fully describe our position in response to these statements, and we therefore invite interested readers to examine our responses in our Appendix (available as a supplement to the online version of this article at http://www.ajph.org). Our articles represented earnest efforts to address selection biases in survey-based estimates of the obesity–mortality association. We proposed an alternative to the standard method because that method completely ignores these biases. Dr. Hanley’s simulations convinced him that our approach creates more problems than it
April 2017, Vol 107, No. 4
AJPH
solves. We respect Dr. Hanley’s effort and acknowledge that it is an important part of the scientific enterprise—a strong and wellreasoned challenge. In response to Hanley’s challenge, we wrote an evidence-based rebuttal that the Editors of the AJPH (then the American Journal of Public Health) declined to publish. The essence of our response was that Hanley’s simulation assumptions inaccurately reflected the full scale of the selection biases that affect the obesity– mortality association in data from the National Health Interview Survey (NHIS). Nonetheless, we refitted our survival models, taking into account Dr. Hanley’s concerns. Results from these new analyses were consistent with those from our original articles—namely, that apparent age-related declines in the obesity–mortality association strongly reflect selection bias.
Furthermore, we showed that the approach used in our articles corrected this bias in NHIS data and provided accurate estimates of true male–female mortality hazard ratios in official US mortality data. We did this to counter Dr. Hanley’s test of our approach, in which he used known male–female hazard ratios but simulated a selection pattern that was not observed in the NHIS or the National Health and Nutrition Examination Surveys. Taken together, our analyses show that (1) Hanley’s simulation bears little resemblance to real survey data and (2) our approach provides accurate estimates of known hazard ratios using data from the NHIS and National Health and Nutrition Examination Surveys. For our complete response, please see the Appendix. For us, the most critical issue remains the strong likelihood that uncorrected survey
ABOUT THE AUTHORS Ryan K. Masters is with the Department of Sociology, University of Colorado Boulder, Boulder. Daniel A. Powers is with the Department of Sociology, College of Liberal Arts, University of Texas at Austin, Austin. Eric N. Reither is with the Department of Sociology, Social Work, and Anthropology, Utah State University, Logan. Y. Claire Yang is with the Department of Sociology and Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill. Bruce G. Link is with the Department of Sociology and Public Policy, University of California-Riverside. Correspondence should be sent to Ryan K. Masters, Department of Sociology, UCB 327, Ketchum Hall 264, University of Colorado Boulder, Boulder, CO 80309 (e-mail: [email protected]). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This editorial was accepted January 17, 2017. doi: 10.2105/AJPH.2017.303715
Masters et al.
Editorial
505
AJPH PERSPECTIVES
estimates of the obesity–mortality relationship are biased. Hanley’s own simulations show that hazard ratios estimated from a conventional approach “are quite a bit lower than those in the unselected (‘population’) ones, particularly at older ages” (Appendix 4 of Hanley3,4). In other words, according to Hanley, any approach that fails to address sample selection will produce biased estimates. Furthermore, when sample selection is not addressed in NHIS data, the results indicate that overweight and obesity reduced mortality in the US adult population by nearly 10% between 1986 and 2006 (see Appendix). This patently absurd finding shows that the statistical cure is not worse than the bias, as Hanley alleges. It also underscores our central point that conventional approaches to estimating the obesity–mortality association are seriously flawed, with potentially devastating consequences for public health policies that to date have not addressed the obesity epidemic with sufficient urgency. The editorial1,2 is, in our opinion, remiss in its failure to situate the issue in this broader framework and to recognize the challenge that lies before us all as we seek to understand the mortality effects of obesity.
the AJPH and the two methodologists at the AJE who reviewed it did so very carefully and had no conflicts of interest. Their impeccable reputations convinced us of the accuracy of Hanley’s concerns. In addition, as part of an overall assessment of the case, the commentary by Hanley was reviewed by an external and independent Advisory Ethical Committee convened by the AJPH that comprised three public health scientists whose recommendations were followed.
REFERENCES 1. Morabia A, Szklo M, Vaughan R. Editorial: note about inaccurate results published in the American Journal of Epidemiology and the American Journal of Public Health. Am J Public Health. 2017;107(4):502. 2. Morabia A, Szklo M, Vaughan R, et al. Editorial: note about inaccurate results published in the American Journal of Epidemiology and the American Journal of Public Health. Am J Epidemiol. 2017;185(6):407–408. 3. Hanley J. Correction of selection bias in survey data: is the statistical cure worse than the bias? Am J Epidemiol. 2017;185(6):409–411. 4. Hanley J. Correction of selection bias in survey data: is the statistical cure worse than the bias? Am J Public Health. 2017;107(4):503–505. 5. Masters RK, Powers DA, Link BG. Obesity and US mortality risk over the adult life course. Am J Epidemiol. 2013;177(5):431–442. 6. Masters RK, Reither EN, Powers DA, Yang YC, Burger AE, Link BG. The impact of obesity on US mortality levels: the importance of age and cohort factors in population estimates. Am J Public Health. 2013;103(10):1895–1901.
EDITORS’ NOTE Although Masters et al. are correct in saying that the commentary by Hanley was not anonymously reviewed, the methodologist at
506
Editorial
Masters et al.
AJPH
April 2017, Vol 107, No. 4
