Disability and Rehabilitation

ISSN: 0963-8288 (Print) 1464-5165 (Online) Journal homepage: http://www.tandfonline.com/loi/idre20

What pilot studies tell us! Sabrina Figueiredo & Nancy E. Mayo To cite this article: Sabrina Figueiredo & Nancy E. Mayo (2015) What pilot studies tell us!, Disability and Rehabilitation, 37:18, 1694-1695, DOI: 10.3109/09638288.2015.1043471 To link to this article: http://dx.doi.org/10.3109/09638288.2015.1043471

Published online: 07 Jul 2015.

Submit your article to this journal

Article views: 84

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=idre20 Download by: [Central Michigan University]

Date: 07 November 2015, At: 03:45

http://informahealthcare.com/dre ISSN 0963-8288 print/ISSN 1464-5165 online Disabil Rehabil, 2015; 37(18): 1694–1695 ! 2015 Informa UK Ltd. DOI: 10.3109/09638288.2015.1043471

LETTER TO THE EDITOR

What pilot studies tell us! Sabrina Figueiredo1,2 and Nancy E. Mayo1,2,3 Faculty of Medicine, School of Physical and Occupational Therapy, McGill University, Montreal, Quebec, Canada, 2Division of Clinical Epidemiology, McGill University, Montreal, Quebec, Canada, and 3Department of Geriatrics, Faculty of Medicine, McGill University, Montreal, Quebec, Canada

Downloaded by [Central Michigan University] at 03:45 07 November 2015

1

Thank you very much for your interest in our 2013 paper reporting on a pilot randomized trial of Nordic Walking as an adjunct to Geriatric Rehabilitation [1]. The most important point about our article is that it was designed, ethically approved and registered as a pilot study, as clearly stated in the title and ‘‘Methods’’ section. Pilot studies are defined [2–4] as ‘‘small scale test of the methods and procedures to be used on a larger scale if the pilot study demonstrates that these methods and procedures can work’’ [3]. Part of demonstrating capacity to ‘‘work’’ is deriving an estimate of the extent to which the intervention group changes (potential for efficacy) as, if no change is observed, perhaps pursuing this avenue of intervention is not warranted. A pilot study is not a hypothesis testing study [5]; however, it is always beneficial to have a control group (even if there is no intent to do a between group comparison) as ‘‘nothing improves the appearance of an intervention so much as the absence of a control group’’ [6]. This is particular true in the context of Geriatric Rehabilitation as the expectation is for improvement, and we desired to estimate whether adding a modality to usual care would result in any benefit without inducing harm. The objectives of our pilot were as follows: (i) to provide evidence for the efficacy potential for Nordic Walking to guide future studies in this topic and (ii) to address issues of feasibility and safety as shoulder pain, hand pain and falls were possible when using poles. In addition, as stated at the end of the introduction, we never hypothesized that one treatment would be more (statistically) significant than the other. In the context of our study, statistical testing is not appropriate as the focus is on parameter estimation and the control group is only present to address expectation for change under usual circumstances and to aid in the interpretation of observed change, if any, in the intervention group. The study was never designed as a parallel group comparison as we could not justify the resources in the absence of any data on safety and efficacy of this novel approach. Hence, our sample size does not provide enough power for tests of between-group difference and testing of this contrast should not be attempted in this situation, as, if it exists it is likely to be spuriously high, through a mechanism identified by Cronbach in 1980 as ‘‘superrealization bias’’ [7].

Address for correspondence: Sabrina Figueiredo, Division of Clinical Epidemiology, McGill University, 687, Pine Av. W, R4.34, Montreal, H3A 1A1 Quebec, Canada. Tel: +514 934 1934 ext. 36926. Fax: +514 843 1493. E-mail: [email protected]

The parameter estimated in our study was effect size and we used the estimator recommended by many authors [8–10] (difference baseline to post/SD baseline). As we were interested in the relative efficacy, we formed the ratio of effect sizes between Nordic Walking and Overground Walking, an estimator akin to relative efficiency [8]. The formula presented in our article shows how to make a ratio of two effect sizes in order to facilitate comprehension by clinical readers of the results shown in Table IV. As we were concerned about safety, we also reported on adverse events (i.e. falls, see p. 971). Owing to the opportunity afforded by Allison et al. to relook at our data, we identified how many participants made a clinical meaningful change in gait speed in either direction – improvement or deterioration – and we show this in Table 1. With data from the Overground Walking group to provide the expected values for benefit or harm, we calculated the probability of observing the events in the Nordic Walking group, using the normal approximation to the binomial distribution. Of note is that 4 of the 13 people in the Usual Care group deteriorated in gait speed (event rate 31%) while only 1 of 13 did so in the Nordic walking group (event rate 8%), an event rate that had a small probability of occurring by chance (p ¼ 0.055). This provides additional evidence that Nordic Walking may be potentially effective as an adjunct to Geriatric Rehabilitation and this article would provide support for mounting such a trial. We reported the limitations of our study in the discussion, however, in the same section, we emphasized that findings from our study were sufficient to motivate a larger randomized clinical trial (RCT). The limitations faced by our study are common to all the small studies [11] and, indeed, Mayo et al. have further elaborated on the use of pilot studies in a recent publication which may be of interest to readers and researchers alike [12]. In light of the above-mentioned, we beg to differ with Allison et al. Our conclusion is not inaccurate, misleading, nor based on inappropriate statistical methods. We did the analysis and reporting of our study following recommendations for pilot trials [5,13–15]. As stated in our study (see p. 973), we firmly believe that patients receiving Nordic Walking as an adjunct to therapy in Geriatric Rehabilitation Program are more likely to improve and less likely to deteriorate on gait speed. In addition, we highlighted that Nordic Walking and Overground Walking are as effective in improving walking endurance. We continue to emphasize that this study provides useful evidence for others who would like to undertake the appropriately powered study on this topic in order to contribute more robust evidence.

What pilot studies tell us!

DOI: 10.3109/09638288.2015.1043471

Table 1. Number of participants reaching clinical meaningful change in gait speed.

Downloaded by [Central Michigan University] at 03:45 07 November 2015

Nordic waking Usual (overground) walking

Improvement

No change

Deterioration

9 6

3 3

1 4

In summary, pilot studies are small-scale test of the methods and procedures to be used on a larger trial if the pilot study demonstrates that these methods and procedures can work. These types of studies are often employed in order to estimate the potential for efficacy and safety of an intervention, prior to embarking on an expensive and large-scale Randomized Controlled Trial (RCT). Hence, pilot studies have their own guidelines for reporting and analyzing the data, which differs from the guidelines of a Consort Statement – especially developed for RCTs. Following the same idea, as a pilot study is not a hypothesis test study, statistical approaches other than significant tests can and should be used, since tests of between-group difference are not powered for establishing significance. Pilot studies can contribute immensely toward evidence, especially in the rehabilitation field. Readers just need to be aware of the differences between Pilot Trials and RCTs in order to translate the study results into their practice.

Declaration of interest The authors declare no conflict of interest.

References 1. Figueiredo S, Finch L, Mai J, et al. Nordic walking for geriatric rehabilitation: a randomized pilot trial. Disabil Rehab J 2013;35: 968–75.

1695

2. Everitt B. Medical statistics from A to Z: a guide for clinicians and medical students. Cambridge: Cambridge University Press; 2006. 3. Porta M. (ed.). A dictionary of epidemiology. Oxford: Oxford University Press; 2014. 4. Everitt B. The Cambridge dictionary of statistics. Cambridge: Cambridge University Press; 2006. 5. Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res 2011;45:626–9. 6. Bearman JE, Loewenson RB, Gullen WH. Muench’s postulates, laws, and corollaries, or biometricians’ views of clinical studies. Biometrics Note No. 4 (April). Bethesda (MD): Office of Biometry and Epidemiology, National Eye Institute, National Institutes of Health; 1974. 7. Cronbach LJ, Ambron SR, Dornbusch SM, et al. Toward reform of program evaluation: aims, methods, andinstitutional arrangements. San Francisco: Jossey-Bass; 1980. 8. Liang MH, Larson MG, Cullen KE, Schwartz JA. Comparative measurement efficiency and sensitivity of five health status instruments for arthritis research. Arthritis Rheum 1985;28:542–7. 9. Fitzgerald JF, Dittus RS. Institutionalized patients with hip fractures: characteristics associated with returning to community dwelling. J Gen Intern Med 1990;5:298–303. 10. Deyo RA, Diehr P, Patrick DL. Reproducibility and responsiveness of helath status measures; statistics and strategies for evaluation. Control Clin Trials 1991;12:142S–58S. 11. Sackett DL, Cook DJ. Can we learn anything from small trials? Ann N Y Acad Sci 1993;703:25–31. 12. Mayo NE, Moriello C, Scott SC, et al. Pedometer-facilitated walking intervention shows promising effectiveness for reducing cancer fatigue: a pilot randomized trial. Clin Rehabil 2014;28:1198–209. 13. Thabane L, Ma J, Chu R, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol 2010;10:1. 14. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract 2004; 10:307–12. 15. Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol 2010;10:67.

What pilot studies tell us!

What pilot studies tell us! - PDF Download Free
417KB Sizes 0 Downloads 7 Views