E DI TO R IA L
BJD
British Journal of Dermatology
Guidelines for statistical reporting in the British Journal of Dermatology
DOI: 10.1111/bjd.13882
Guidelines
• •
Use the SAMPL guidelines for general statistical reporting. Use study-specific guidelines (e.g. CONSORT, STROBE, TRIPOD) for details.
General aspects
• • • • •
Start with a clear research question in the introduction. The statistical analysis should answer the research question. Match the order of the results to the statistical analyses. Present clinically relevant outcome measures. Present estimates with 95% confidence intervals, not just P-values.
The use of reporting guidelines In this issue of the BJD, McClean and Silverberg report that that the quality of statistical reporting of randomized controlled trials (RCTs) in the dermatological literature is very low.1 This is a sad observation, because it concerns research involving humans. In article 36 of the Declaration of Helsinki it is stated that ‘Researchers, authors, sponsors, editors and publishers all have ethical obligations with regard to the publication and dissemination of the results of research. Researchers have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports.’2 Participating patients are promised that their participation will be beneficial for others. Correct (statistical) reporting of the study is required in order to fulfil this promise. The statistical analysis should be thoroughly thought about before the start of the study. After the study has been performed, it should be described in such a way that researchers with access to the original data should be able to reproduce the study results with the information provided in the article.3 In order to improve statistical reporting, we recommend that all authors follow the Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines when submitting to the BJD.4 Existing study-specific guidelines (e.g. CONSORT, STROBE, PRISMA) from the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network should be used to
© 2015 British Association of Dermatologists
improve reporting of the study-specific items, such as interim analyses and stopping guidelines in an RCT (http:// www.equator-network.org/).5 Although these are reporting guidelines, which implies that they should be used after the study has been performed, these guidelines should also be used in the starting phase of the study in order avoid missing items. For clinical trials it is often compulsory to submit the research protocol to a clinical trials register, in which key issues such as the study population, the primary end point and sample size are predefined. This ensures that the original study aims are accessible to peers prior to publication, to reviewers in the manuscript submission phase and by readers after publication. Also, this reduces the chances of embarking on similar studies simultaneously and reduces reporting bias. In theory, this should also apply to other fields of research.
Match the research question, statistical analyses and results The study protocol and final article both start with a clear research question, which is presented at the end of the introduction. A PICO (patients, intervention, controls, outcome) approach is helpful to structure the research question and provides a starting point to think about which types of data and analysis can be used to test the hypothesis. The PICO already contains information about the number of groups to be compared and the primary outcome measure, which are both relevant to determine the correct statistical test. Authors should make a clear distinction between the primary research question, secondary research questions and exploratory or subgroup analyses throughout the article. The result of the primary analysis should be presented in the abstract. Present the statistical analyses and the results in the same order of importance: primary analysis, secondary analyses and any ancillary or exploratory analyses.3 The results section should match the methods section: for every method (what has been done), there should be a corresponding result (what has been found) and vice versa.5 Sample-size calculations are based on the primary end point.6–8 To prevent type I errors (no difference between the groups, but a P-value < 005 by chance), results from secondary analyses should be presented with caution and at the end of the methods and results. If many secondary outcomes are planned, a correction for multiple testing may be appropriate.9 Make clear which findings result from post hoc analyses, and explain that these findings are intended to generate new hypotheses.5,10
British Journal of Dermatology (2015) 173, pp3–5
3
4 Editorial
Description of the methods
Missing data
The SAMPL guidelines contain general principles for reporting statistical methods. We recommend that all principles are applied. Here we highlight a few aspects.
Missing data is an aspect that is almost unavoidable in every type of research, but the handling and reporting of missing data is poor in many studies.14–16 Inadequate handling of missing data can lead to substantial bias.17 We encourage authors to report missing data and to describe how missing data were handled.17–19
Purpose of the analysis The main analysis should answer the primary research question. It should be clearly described whether the analysis answers the primary or secondary research questions, and whether it is an ancillary analysis (e.g. sensitivity analysis, missing data imputation model) or a post hoc analysis. Do not assume that all readers are familiar with the statistical technique that you used.3 Try to summarize the essence of the statistical technique in one or two sentences, followed by a methodological reference. Clinical importance If possible, the minimal clinically important difference should be identified in advance. The calculation of P-values depends on the sample size and the effect size; even the smallest difference becomes statistically significant if the sample size is large enough, and vice versa. Specifying in advance what difference you define as clinically relevant (delta of the power calculation) not only assures sufficient power, but also prevents small statistically significant differences being interpreted as clinically relevant. Outcome measures The choice of outcome definition is crucially important, and is often a trade-off between maximizing data information and data interpretation.11 Categorization of continuous scores eases the interpretation but also introduces subjectivity and results in loss of data information. For example, if the change in the Psoriasis Area and Severity Index (PASI) score is the outcome of interest, the change in the PASI score can be used as a continuous outcome measure (maximum data information) or it can be dichotomized to obtain odds ratios of a decrease of the PASI by 75% (PASI 75) compared with no PASI 75 response (easing data interpretation). In addition, an odds ratio is a relative risk measure, but it is commonly misinterpreted as a risk ratio when events are common.12 Primary outcome measures may also be used to calculate clinically relevant outcome measures for clinicians and decision makers, such as the number needed to treat and number needed to harm for experimental studies, or relative attributable fractions (e.g. population attributable fraction) and absolute attributable population measures (e.g. population incidence difference) in observational research.13 The reason for the choice of outcome measure may include statistical arguments, clinical importance or feasibility and should be described in the methods.
British Journal of Dermatology (2015) 173, pp3–5
Presentation of the results The reporting of numbers and descriptive statistics according to the SAMPL guidelines can be applied to all types of research. Specific guidelines for the most common types of statistical analyses are also included in the SAMPL guidelines and should be followed. For some types of research, more detailed guidelines for the statistical analysis can be found in study-specific guidelines, such as the TRIPOD statement for multivariable prediction modelling. Again, here we highlight just some of the principles, but we recommend that all principles are followed. Never report results with P-values only The interpretation of results based on P-values only can be misleading and is therefore discouraged.5,20,21 We recommend presenting effect sizes, such as odds ratios hazard ratios or mean differences, together with their 95% confidence intervals. In contrast to the P-value, the 95% confidence interval also shows the direction of the treatment effect (whether towards harmful or beneficial effects) and the magnitude and precision of the effect estimate.5,21 It should also be kept in mind that besides the statistical significance, there are other factors which are important in interpreting the results, such as the magnitude of the risk estimates (e.g. absolute excess risk, risk ratio, odds ratio, survival benefit); a correct causal model; a correct statistical model; no systematic error (bias and confounding); and findings of prior studies.21,22 Report numbers with an appropriate degree of precision Make consistent use of meaningful decimals in tables and figures. For very large numbers, decimals may not be necessary (e.g. incidence rate of 134 per 100 000 person-years instead of 13401 per 100 000 person-years). Report absolute and relative measures Relative measures should always be presented with the absolute number of cases [e.g. the percentage was 27% (68 of 250) in the treatment group compared with 17% (43 of 250) in the placebo group]. This makes interpretation of the study results easier and helps readers to distinguish between statistical significance and clinical relevance.
© 2015 British Association of Dermatologists
Editorial
Final recommendation We recommend that all authors and reviewers of the BJD use the SAMPL guidelines for basic statistical reporting. Study-specific guidelines should be used as complementary guidelines, because they also contain items for statistical reporting. Hereby we aim to increase transparency and improve statistical reporting in the BJD in order to contribute to the BJD’s core mission to improve the quality of reporting of dermatological research.
Conflicts of interest None declared. Department of Dermatology, Erasmus MC University Medical Center, Rotterdam, the Netherlands E-mail: [email protected]
L.M. HOLLESTEIN T. NIJSTEN
References 1 McClean M, Silverberg JI. Statistical reporting in randomized controlled trials from the dermatology literature: a review of 44 dermatology journals. Br J Dermatol 2015; 173:172–83. 2 World Medical Association. Declaration of Helsinki – Ethical Principles for Medical Research Involving Human Subjects. Available at: http://www.wma.net/en/30publications/10policies/b3/ (last accessed 22 April 2015). 3 Kotz D, Cals JW. Effective writing and publishing scientific papers, part IV: methods. J Clin Epidemiol 2013; 66:817. 4 Lang T, Altman D. Basic statistical reporting for articles published in biomedical journals. The ‘Statistical Analyses and Methods in the Published Literature’ or the ‘SAMPL Guidelines’. Available at: http://www.equator-network.org/reporting-guidelines/sampl/ (la st accessed 22 April 2015). 5 Kotz D, Cals JW. Effective writing and publishing scientific papers, part V: results. J Clin Epidemiol 2013; 66:945.
© 2015 British Association of Dermatologists
5
6 Chow S, Shao J, Wang H. Sample Size Calculation in Clinical Research. New York: Marcel Dekker, 2003. 7 Piantadosi S. Clinical Trials: a Methodological Perspective. New Jersey: John Wiley & Sons, 2005. 8 Zhang E, Wu VQ, Chow SC, Zhang HG. Package ‘TrialSize’ in R. Available at: http://cran.r-project.org/web/packages/TrialSize/ (last accessed 22 April 2015). 9 Bender R, Lange S. Adjusting for multiple testing – when and how? J Clin Epidemiol 2001; 54:343–9. 10 Sun X, Ioannidis JP, Agoritsas T et al. How to use a subgroup analysis: users’ guide to the medical literature. JAMA 2014; 311:405–11. 11 Steyerberg E. Clinical Prediction Models. New York: Springer, 2009. 12 Sainani KL. Understanding odds ratios. P M R 2011; 3:263–7. 13 Citrome L, Ketter TA. When does a difference make a difference? Interpretation of number needed to treat, number needed to harm, and likelihood to be helped or harmed. Int J Clin Pract 2013; 67:407–11. 14 Wood AM, White IR, Thompson SG. Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials 2004; 1:368–76. 15 Wiebe N, Vandermeer B, Platt RW et al. A systematic review identifies a lack of standardization in methods for handling missing variance data. J Clin Epidemiol 2006; 59:342–53. 16 Eekhout I, de Boer RM, Twisk JW et al. Missing data: a systematic review of how they are reported and handled. Epidemiology 2012; 23:729–32. 17 Donders AR, van der Heijden GJ, Stijnen T, Moons KG. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol 2006; 59:1087–91. 18 Little RJ, D’Agostino R, Cohen ML et al. The prevention and treatment of missing data in clinical trials. N Engl J Med 2012; 367:1355–60. 19 Kistin CJ. Transparent reporting of missing outcome data in clinical trials: applying the general principles of CONSORT 2010. Evid Based Med 2014; 19:161–2. 20 Chia KS. ‘Significant-itis’ – an obsession with the P-value. Scand J Work Environ Health 1997; 23:152–4. 21 Stang A, Poole C, Kuss O. The ongoing tyranny of statistical significance testing in biomedical research. Eur J Epidemiol 2010; 25:225–30. 22 Hill AB. The environment and disease: association or causation? Proc R Soc Med 1965; 58:295–300.
British Journal of Dermatology (2015) 173, pp3–5
BJD
British Journal of Dermatology
Guidelines for statistical reporting in the British Journal of Dermatology
DOI: 10.1111/bjd.13882
Guidelines
• •
Use the SAMPL guidelines for general statistical reporting. Use study-specific guidelines (e.g. CONSORT, STROBE, TRIPOD) for details.
General aspects
• • • • •
Start with a clear research question in the introduction. The statistical analysis should answer the research question. Match the order of the results to the statistical analyses. Present clinically relevant outcome measures. Present estimates with 95% confidence intervals, not just P-values.
The use of reporting guidelines In this issue of the BJD, McClean and Silverberg report that that the quality of statistical reporting of randomized controlled trials (RCTs) in the dermatological literature is very low.1 This is a sad observation, because it concerns research involving humans. In article 36 of the Declaration of Helsinki it is stated that ‘Researchers, authors, sponsors, editors and publishers all have ethical obligations with regard to the publication and dissemination of the results of research. Researchers have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports.’2 Participating patients are promised that their participation will be beneficial for others. Correct (statistical) reporting of the study is required in order to fulfil this promise. The statistical analysis should be thoroughly thought about before the start of the study. After the study has been performed, it should be described in such a way that researchers with access to the original data should be able to reproduce the study results with the information provided in the article.3 In order to improve statistical reporting, we recommend that all authors follow the Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines when submitting to the BJD.4 Existing study-specific guidelines (e.g. CONSORT, STROBE, PRISMA) from the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network should be used to
© 2015 British Association of Dermatologists
improve reporting of the study-specific items, such as interim analyses and stopping guidelines in an RCT (http:// www.equator-network.org/).5 Although these are reporting guidelines, which implies that they should be used after the study has been performed, these guidelines should also be used in the starting phase of the study in order avoid missing items. For clinical trials it is often compulsory to submit the research protocol to a clinical trials register, in which key issues such as the study population, the primary end point and sample size are predefined. This ensures that the original study aims are accessible to peers prior to publication, to reviewers in the manuscript submission phase and by readers after publication. Also, this reduces the chances of embarking on similar studies simultaneously and reduces reporting bias. In theory, this should also apply to other fields of research.
Match the research question, statistical analyses and results The study protocol and final article both start with a clear research question, which is presented at the end of the introduction. A PICO (patients, intervention, controls, outcome) approach is helpful to structure the research question and provides a starting point to think about which types of data and analysis can be used to test the hypothesis. The PICO already contains information about the number of groups to be compared and the primary outcome measure, which are both relevant to determine the correct statistical test. Authors should make a clear distinction between the primary research question, secondary research questions and exploratory or subgroup analyses throughout the article. The result of the primary analysis should be presented in the abstract. Present the statistical analyses and the results in the same order of importance: primary analysis, secondary analyses and any ancillary or exploratory analyses.3 The results section should match the methods section: for every method (what has been done), there should be a corresponding result (what has been found) and vice versa.5 Sample-size calculations are based on the primary end point.6–8 To prevent type I errors (no difference between the groups, but a P-value < 005 by chance), results from secondary analyses should be presented with caution and at the end of the methods and results. If many secondary outcomes are planned, a correction for multiple testing may be appropriate.9 Make clear which findings result from post hoc analyses, and explain that these findings are intended to generate new hypotheses.5,10
British Journal of Dermatology (2015) 173, pp3–5
3
4 Editorial
Description of the methods
Missing data
The SAMPL guidelines contain general principles for reporting statistical methods. We recommend that all principles are applied. Here we highlight a few aspects.
Missing data is an aspect that is almost unavoidable in every type of research, but the handling and reporting of missing data is poor in many studies.14–16 Inadequate handling of missing data can lead to substantial bias.17 We encourage authors to report missing data and to describe how missing data were handled.17–19
Purpose of the analysis The main analysis should answer the primary research question. It should be clearly described whether the analysis answers the primary or secondary research questions, and whether it is an ancillary analysis (e.g. sensitivity analysis, missing data imputation model) or a post hoc analysis. Do not assume that all readers are familiar with the statistical technique that you used.3 Try to summarize the essence of the statistical technique in one or two sentences, followed by a methodological reference. Clinical importance If possible, the minimal clinically important difference should be identified in advance. The calculation of P-values depends on the sample size and the effect size; even the smallest difference becomes statistically significant if the sample size is large enough, and vice versa. Specifying in advance what difference you define as clinically relevant (delta of the power calculation) not only assures sufficient power, but also prevents small statistically significant differences being interpreted as clinically relevant. Outcome measures The choice of outcome definition is crucially important, and is often a trade-off between maximizing data information and data interpretation.11 Categorization of continuous scores eases the interpretation but also introduces subjectivity and results in loss of data information. For example, if the change in the Psoriasis Area and Severity Index (PASI) score is the outcome of interest, the change in the PASI score can be used as a continuous outcome measure (maximum data information) or it can be dichotomized to obtain odds ratios of a decrease of the PASI by 75% (PASI 75) compared with no PASI 75 response (easing data interpretation). In addition, an odds ratio is a relative risk measure, but it is commonly misinterpreted as a risk ratio when events are common.12 Primary outcome measures may also be used to calculate clinically relevant outcome measures for clinicians and decision makers, such as the number needed to treat and number needed to harm for experimental studies, or relative attributable fractions (e.g. population attributable fraction) and absolute attributable population measures (e.g. population incidence difference) in observational research.13 The reason for the choice of outcome measure may include statistical arguments, clinical importance or feasibility and should be described in the methods.
British Journal of Dermatology (2015) 173, pp3–5
Presentation of the results The reporting of numbers and descriptive statistics according to the SAMPL guidelines can be applied to all types of research. Specific guidelines for the most common types of statistical analyses are also included in the SAMPL guidelines and should be followed. For some types of research, more detailed guidelines for the statistical analysis can be found in study-specific guidelines, such as the TRIPOD statement for multivariable prediction modelling. Again, here we highlight just some of the principles, but we recommend that all principles are followed. Never report results with P-values only The interpretation of results based on P-values only can be misleading and is therefore discouraged.5,20,21 We recommend presenting effect sizes, such as odds ratios hazard ratios or mean differences, together with their 95% confidence intervals. In contrast to the P-value, the 95% confidence interval also shows the direction of the treatment effect (whether towards harmful or beneficial effects) and the magnitude and precision of the effect estimate.5,21 It should also be kept in mind that besides the statistical significance, there are other factors which are important in interpreting the results, such as the magnitude of the risk estimates (e.g. absolute excess risk, risk ratio, odds ratio, survival benefit); a correct causal model; a correct statistical model; no systematic error (bias and confounding); and findings of prior studies.21,22 Report numbers with an appropriate degree of precision Make consistent use of meaningful decimals in tables and figures. For very large numbers, decimals may not be necessary (e.g. incidence rate of 134 per 100 000 person-years instead of 13401 per 100 000 person-years). Report absolute and relative measures Relative measures should always be presented with the absolute number of cases [e.g. the percentage was 27% (68 of 250) in the treatment group compared with 17% (43 of 250) in the placebo group]. This makes interpretation of the study results easier and helps readers to distinguish between statistical significance and clinical relevance.
© 2015 British Association of Dermatologists
Editorial
Final recommendation We recommend that all authors and reviewers of the BJD use the SAMPL guidelines for basic statistical reporting. Study-specific guidelines should be used as complementary guidelines, because they also contain items for statistical reporting. Hereby we aim to increase transparency and improve statistical reporting in the BJD in order to contribute to the BJD’s core mission to improve the quality of reporting of dermatological research.
Conflicts of interest None declared. Department of Dermatology, Erasmus MC University Medical Center, Rotterdam, the Netherlands E-mail: [email protected]
L.M. HOLLESTEIN T. NIJSTEN
References 1 McClean M, Silverberg JI. Statistical reporting in randomized controlled trials from the dermatology literature: a review of 44 dermatology journals. Br J Dermatol 2015; 173:172–83. 2 World Medical Association. Declaration of Helsinki – Ethical Principles for Medical Research Involving Human Subjects. Available at: http://www.wma.net/en/30publications/10policies/b3/ (last accessed 22 April 2015). 3 Kotz D, Cals JW. Effective writing and publishing scientific papers, part IV: methods. J Clin Epidemiol 2013; 66:817. 4 Lang T, Altman D. Basic statistical reporting for articles published in biomedical journals. The ‘Statistical Analyses and Methods in the Published Literature’ or the ‘SAMPL Guidelines’. Available at: http://www.equator-network.org/reporting-guidelines/sampl/ (la st accessed 22 April 2015). 5 Kotz D, Cals JW. Effective writing and publishing scientific papers, part V: results. J Clin Epidemiol 2013; 66:945.
© 2015 British Association of Dermatologists
5
6 Chow S, Shao J, Wang H. Sample Size Calculation in Clinical Research. New York: Marcel Dekker, 2003. 7 Piantadosi S. Clinical Trials: a Methodological Perspective. New Jersey: John Wiley & Sons, 2005. 8 Zhang E, Wu VQ, Chow SC, Zhang HG. Package ‘TrialSize’ in R. Available at: http://cran.r-project.org/web/packages/TrialSize/ (last accessed 22 April 2015). 9 Bender R, Lange S. Adjusting for multiple testing – when and how? J Clin Epidemiol 2001; 54:343–9. 10 Sun X, Ioannidis JP, Agoritsas T et al. How to use a subgroup analysis: users’ guide to the medical literature. JAMA 2014; 311:405–11. 11 Steyerberg E. Clinical Prediction Models. New York: Springer, 2009. 12 Sainani KL. Understanding odds ratios. P M R 2011; 3:263–7. 13 Citrome L, Ketter TA. When does a difference make a difference? Interpretation of number needed to treat, number needed to harm, and likelihood to be helped or harmed. Int J Clin Pract 2013; 67:407–11. 14 Wood AM, White IR, Thompson SG. Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials 2004; 1:368–76. 15 Wiebe N, Vandermeer B, Platt RW et al. A systematic review identifies a lack of standardization in methods for handling missing variance data. J Clin Epidemiol 2006; 59:342–53. 16 Eekhout I, de Boer RM, Twisk JW et al. Missing data: a systematic review of how they are reported and handled. Epidemiology 2012; 23:729–32. 17 Donders AR, van der Heijden GJ, Stijnen T, Moons KG. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol 2006; 59:1087–91. 18 Little RJ, D’Agostino R, Cohen ML et al. The prevention and treatment of missing data in clinical trials. N Engl J Med 2012; 367:1355–60. 19 Kistin CJ. Transparent reporting of missing outcome data in clinical trials: applying the general principles of CONSORT 2010. Evid Based Med 2014; 19:161–2. 20 Chia KS. ‘Significant-itis’ – an obsession with the P-value. Scand J Work Environ Health 1997; 23:152–4. 21 Stang A, Poole C, Kuss O. The ongoing tyranny of statistical significance testing in biomedical research. Eur J Epidemiol 2010; 25:225–30. 22 Hill AB. The environment and disease: association or causation? Proc R Soc Med 1965; 58:295–300.
British Journal of Dermatology (2015) 173, pp3–5
