Journal of Pediatric Surgery 49 (2014) 811–814
Contents lists available at ScienceDirect
Journal of Pediatric Surgery journal homepage: www.elsevier.com/locate/jpedsurg
A pilot investigation of a pediatric surgery journal club William Shihao Lao a, Pramod Puligandla b, Robert Baird b,⁎ a b
McGill University, Montreal Quebec, Canada Department of Pediatric General and Thoracic Surgery, The Montreal Children’s Hospital, McGill University, Montreal Quebec Canada
a r t i c l e
i n f o
Article history: Received 11 February 2014 Accepted 13 February 2014 Key words: Evidence based medicine Surgical journal club Journal club
a b s t r a c t Background: The CanMEDS competency “scholar” encompasses the creation, dissemination, application, and translation of medical knowledge. We hypothesize that a structured journal club (JC) for pediatric surgical trainees would meet these objectives in an enjoyable and long-lasting manner. Methods: A JC involving two pediatric surgery training programs was created with each session focusing on a specific study design. Pre-tests/post-tests were administered before/after each session with durability of learning assessed during the following session. Metrics analyzed included participant satisfaction and an appraisal of evidence-based medicine (EBM) principals. Test results were analyzed using the paired T-test with p b 0.05 considered significant. Results: On average, 14 participants attended each session, with all trainees present (4). While participants believed they understood EBM principles, 40% were unfamiliar with question formulation, 48% were unfamiliar with critical appraisal tools, and 60% had not appraised an article within the previous year. Pre-test to post-test comparison yielded an improvement in mean score (20 = perfect score): 10.8 to 16.9, p b 0.01. Measures of participant satisfaction were uniformly positive. Conclusion: A structured Pediatric Surgery Journal Club addresses scholarly training objectives in a highly satisfactory manner and yields durable learning. A web-based curriculum based on this model could serve as an important educational tool for trainees and attending staff alike. © 2014 Elsevier Inc. All rights reserved.
The principle at the heart of evidence-based medicine (EBM) is the application of the most current and valid scientific evidence to patient care [1]. Although the concept of EBM has been widely accepted as essential in the practice of modern medicine [2], there exists a general lack of emphasis and proper teaching of EBM throughout medical training. The scope of this problem is exemplified by the fact that combing through the thousands of new monthly publications requires hundreds of hours even for a well trained epidemiologist [3]. As such, the ability to pinpoint strong evidence in the literature is an ever more crucial skill for the modern clinician. Within the narrow field of pediatric surgery, the main obstacle to evidence-based clinical practice is not a lack of publications per se, but rather a general paucity of scientifically rigorous data. In 1999, only 0.3% of pediatric surgery publications were identified as prospective randomized trials [4]. In several pediatric surgical studies of lesser methodology, multiple barriers to the conduct of rigorous trials and creation of high-level evidence were identified [5,6]. Despite the weaknesses in the available evidence, these publications have unanimously stressed the importance of adopting and implementing evidence-based decision-making in clinical practice to ensure optimal patient care.
⁎ Corresponding author at: Montreal Children’s Hospital, 2300 Tupper Street, room C-812, Montreal, Quebec H3H1P3, Canada. Tel.: +1 514 412 4438; fax: +1 514 412 4289. E-mail address: [email protected] (R. Baird). http://dx.doi.org/10.1016/j.jpedsurg.2014.02.068 0022-3468/© 2014 Elsevier Inc. All rights reserved.
The CanMEDS Physician Competency Framework specifies 7 roles, which, when integrated, define the profession of the physician [7]. In one of these roles, the “scholar”, a physician must demonstrate a lifelong commitment to reflective learning, as well as the creation, dissemination, application and translation of medical knowledge [8]. Several studies have demonstrated that the incorporation of an evidence-based journal club into the training curriculum of surgical residents offers a high-yield method for teaching EBM [9–11]. This study seeks to determine the inherent value and efficacy of a pediatric surgery, domain-specific journal club as an educational method for conveying the principles of EBM to a select audience. 1. Methods The Quebec Pediatric Surgery Journal Club (QPSJC) is a 2-year-long curriculum consisting of 6 journal club sessions. It was instituted in 2012 and involves two accredited pediatric surgery training programs in Montreal, Quebec, Canada. These sessions cover various core concepts of evidence-based practice, including question formulation, research methodology, critical appraisal, and rudimentary biostatistics, with each session focusing on a specific study design. The curriculum consists of 1 session each dedicated to the following topics: question formulation, randomized controlled trials (RCTs), systematic reviews, case–control studies, database studies, and cohort design. Materials disseminated in advance included session-specific
812
W.S. Lao et al. / Journal of Pediatric Surgery 49 (2014) 811–814
learning objectives as well as the article to be appraised. The targeted audience included the 4 pediatric surgery fellows from both institutions, with additional participants consisting of 4–8 junior surgical residents, 2–4 medical students, and 4–8 attending pediatric surgeons. Each 2-h session was held in the evening outside of regular working hours. A pediatric surgery fellow was tasked with leading each session and was responsible for both the presentation and appraisal of the chosen article. Several weeks prior to each session, the journal article and session-specific targeted learning objectives were distributed electronically to all potential participants. Participation in the journal club activity was considered mandatory for pediatric surgical fellows only; trainees were invited but not obliged to complete questionnaires and testing through an informed consent process. At the start of each journal club cycle, a selfassessment questionnaire consisting of 8 elements was distributed to all attendees. The purpose of this questionnaire was to allow each participant to anonymously describe their views of EBM and to gauge their comfort with its core concepts. Grading was performed using a Likert scale from 1 to 5, with 1 being “strongly disagree”, and 5 being “strongly agree”. In addition, all participants attempted a pre-session test containing 10 questions that assessed their knowledge of core EBM principles and session specific concepts (e.g. power calculation for Randomized Controlled Trials). Each answer was evaluated individually and graded as 0, 1, or 2, for a maximum of 20 points. At the end of each session, all participants repeated the same test. Presession and post-session scores on the test were compared and analyzed in order to gauge the impact of the journal club session on the participants’ level of understanding. Answers to each test were only provided the following month as a means of reinforcing the acquired knowledge. At the following session’s pre-test, each participant was again assessed for retention and durability of learning. Finally, all attendees were given the opportunity to rate the journal club session through a session evaluation form. Variables analyzed included participant satisfaction, as well as session relevancy and usefulness. All scores and values are reported as a mean ± standard deviation (SD). Test results were analyzed using the Welch student T-test for uneven-sized samples with p b 0.05 considered significant. 2. Results On average, 14 participants were able to attend the sessions, with all 4 pediatric surgery fellows (target learners) present. In total, 11 participants completed the self-assessment questionnaire (Fig. 1).
Participants agreed that EBM was a valuable tool worth learning (mean 4.7 ± 0.47), with 80% of responders stating that they used evidence-based principles in practice (mean 4.0 ± 0.63). In addition, 75% of participants also believed they were familiar with the principles of EBM (mean 3.9 ± 0.70). However, in later statements querying each participant’s familiarity with said principles, up to 40% stated they were unfamiliar with standard question formulation (mean 3.4 ± 1.6), 50% were unfamiliar with critical appraisal tools (mean 3.5 ± 0.85), and 60% were unfamiliar with methods of acquiring the best evidence (mean 3.3 ± 1.1). Lastly, 60% of participants had not critically appraised an article within the previous year. In total, 9 of the attendees returned the pre-session test and 5 attendees returned the initial post-session test (Fig. 2). A comparison of pre-session and (early) post-session test scores revealed a significant improvement with the mean score increasing from 11.1 ± 5.7 to 16.2 ± 4.9 (max = 20; p = 0.05). Five participants returned the second (late) post-session test at the subsequent session. The mean score of this assessment of learning durability was 16.8 ± 3.6 (p = 0.03) compared to the original scores. Overall, all respondents (n = 9) were satisfied with the journal club (mean 4.4 ± 0.52), and found the articles to be clinically relevant (mean 4.0 ± 0.5) in addition to being well summarized and useful (mean 4.2 ± 0.44). Moreover, the methodological review was perceived by all to be both relevant (mean 4.4 ± 0.53) and helpful (4.6 ± 0.53). These results are summarized in Fig. 3.
3. Discussion It has been well demonstrated that one of the main barriers to evidence-based practice in pediatric surgery is the lack of high level studies to inform clinical decision-making [4,12,13]. Even when a well-conducted trial is performed, recognized hurdles inhibit practice change by impeding the dissemination, acceptance and application of that evidence. The results of this study suggest an additional fundamental barrier to evidence-based practice: the lack of knowledge of key EBM concepts. Addressing this problem is of utmost importance as a proper understanding of EBM not only aides in the critical appraisal of existing literature, but also allows for more thoughtful study design, in turn, ensuring the creation of more robust evidence. Ultimately, it is the generation of new knowledge that will drive the profession forward; graduating trainees must be equipped with the necessary tools to accomplish this.
Fig. 1. Summary of Principles of Evidence Based Medicine Self-Assessment Exercise (n = 11 respondents).
W.S. Lao et al. / Journal of Pediatric Surgery 49 (2014) 811–814
Fig. 2. Results of Pre-session (n = 9 respondents) and Post-session (n = 5 respondents) Evaluation.
This study indicates that a structured JC was effective in disseminating EBM principles as it employed many of the critical characteristics of successful journal clubs [9]. These include regular and anticipated meetings, mandatory attendance for trainees, clear long- and short-term objectives, appropriate meeting times and incentives, a trained journal club leader for choosing papers and leading discussions, using established critical appraisal processes, and summarizing journal club findings [14]. In addition, using the internet to circulate the papers and resources prior to the meeting allowed participants to easily access resources outside of sessions. In fact, one of the possible future milestones for the journal club is the creation of an online resource/database to supplement the current format [15,16]. For the time being, the ‘in person’ nature of the session ensured an engaged learning environment for all attendees. Several aspects of the JC were extremely positive. Fig. 1 demonstrated wide variability in learner self-assessment, which was also true specifically for the 4 members of the target audience. All respondents either agreed or strongly agreed with the statement that EB principles have value and are worth learning. While many participants believed they understood the methods of EBM, both the initial self-assessment and pre-session test demonstrated that this was not the case. In fact, a large portion of the participants did not comprehend the methodologies used for formulating questions, acquiring evidence, or appraising literature. A mean score below 60% on the
813
pre-session test further confirmed the finding. This discordance between the self-assigned familiarity and actual knowledge suggests the false perception that the core concepts of EBM are self-evident and do not require formal teaching. However, a significant improvement in the post-session scores illustrated that this unperceived learning need is well addressed by the JC as presented here. Several important limitations exist in the current journal club format. Despite all efforts to appeal to potential participants, incomplete attendance remains problematic. For example, not all staff surgeons from both institutions were able to attend the sessions, reducing their overall educational value and undermining the implied importance of the QPSJC to trainees in attendance. Furthermore, the small number of attendees at each session undermined the power of the present study, especially given that some of the participants did not complete the questionnaires or the tests. As the community of pediatric surgery in Montreal is fairly small in size, there is only a certain limit to which the power of this study can be increased. Moreover, the small sample size may also limit the external validity of our findings. Nonetheless, considerable variability likely exists in the background understanding of EBM concepts among practicing pediatric surgeons; a standardized EBM teaching curriculum addresses this variability by ensuring that core concepts are emphasized using examples derived from the field of pediatric surgery. In general, this pilot study is further limited by its small dataset. The fact that only 9 out of 14 (about 65%) participants completed the test raises the possibility that the results may not be completely representative of the group. As an even smaller number of participants (5) completed the post-session test, the validity of the increase in score is significantly undermined. Further compounding the problem is the anonymous way under which the tests were administered. With the research methodology as constructed, there was no way of determining whether the 5 participants who completed the post-session tests had done a pre-session test at all. One of the ways to address this problem in future sessions is to make all questionnaires mandatory and to ensure that all have been properly filled prior to starting a session, and after its end. In addition, as the journal club is held every three–four months, the elapsed time between each session may allow for significant “knowledge decay” over time. The full extent of such decay may only be confirmed or rejected once knowledge retention is assessed at future sessions. In fact, most successful journal clubs suggest meeting more than 3–4 times per year [14]. Another problem facing this study
Fig. 3. Summary of Session Evaluation (n = 9 respondents).
814
W.S. Lao et al. / Journal of Pediatric Surgery 49 (2014) 811–814
is the fact that the tools used for participant self-assessment and knowledge retention has not been validated. While examples of evaluative tools exist for evidence-based principles in surgery, these tend to address knowledge of EBM as a whole and do not sufficiently address the focus of specific sessions or this activity as a whole [17,18]. Our preliminary results indicate that a well structured, evidencebased Pediatric Surgery Journal Club is an appropriate educational method for addressing scholarly training objectives. Despite the small sample size, this study identified a significant “gap” between the perceived understanding of EBM principles by participants and objective findings based on a session pre-test. This was remedied through the current activity, as participants were able to achieve highyield, durable learning in an appealing and organized fashion. As a future development, a web-based curriculum modeled after the current methods may serve as an easily-accessible educational tool for trainees and attending staff throughout North America.
References [1] Sackett DL, Rosenberg WM, Gray JA, et al. Evidence based medicine: what it is and what it isn't. BMJ 1996;312:71–2. [2] Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet 2003;362:1225–30. [3] Alper BS, Hand JA, Elliott SG, et al. How much effort is needed to keep up with the literature relevant for primary care? J Med Libr Assoc 2004;92:429–37. [4] Hardin Jr WD, Stylianos S, Lally KP. Evidence-based practice in pediatric surgery. J Pediatr Surg 1999;34:908–12.
[5] Rangel SJ, Kelsey J, Henry MC, et al. Critical analysis of clinical research reporting in pediatric surgery: justifying the need for a new standard. J Pediatr Surg 2003;38:1739–43. [6] Curry JI, Reeves B, Stringer MD. Randomized controlled trials in pediatric surgery: could we do better? J Pediatr Surg 2003;38:556–9. [7] Cruess SR, Johnston S, Cruess RL. "Profession": a working definition for medical educators. Teach Learn Med 2004;16:74–6. [8] Frank JR. The CanMEDS 2005 physician competency framework. Better standards. Better physicians. Better care. Ottawa, Canada: The Royal College of Physicians and Surgeons of Canada; 2005. [9] Lee AG, Boldt HC, Golnik KC, et al. Structured journal club as a tool to teach and assess resident competence in practice-based learning and improvement. Ophthalmology 2006;113:497–500. [10] Spillane AJ, Crowe PJ. The role of the journal club in surgical training. Aust N Z J Surg 1998;68:288–91. [11] Shifflette V, Mitchell C, Mangram A, et al. Current approaches to journal club by general surgery programs within the southwestern surgical congress. J Surg Educ 2012;69:162–6. [12] Ostlie DJ, St Peter SD. The current state of evidence-based pediatric surgery. J Pediatr Surg 2010;45:1940–6. [13] Blakely ML, Lally KP. Using evidence-based medicine principles to improve quality of patient care in pediatric surgery. Semin Pediatr Surg 2002;11:42–5. [14] Deenadayalan Y, Grimmer-Somers K, Prior M, et al. How to run an effective journal club: a systematic review. Clin Pract 2008;14:898–911. [15] Kuppersmith RB, Stewart MG, Ohlms LA, et al. Use of an Internet-based journal club. Otolaryngol Head Neck Surg 1997;116:497–8. [16] Macrae HM, Regehr G, McKenzie M, et al. Teaching practicing surgeons critical appraisal skills with an Internet-based journal club: a randomized, controlled trial. Surgery 2004;136:641–6. [17] Toedter LJ, Thompson LL, Rohatgi C. Training surgeons to do evidence-based surgery: a collaborative approach. J Am Coll Surg 2004;199:293–9. [18] Ahmadi N, McKenzie ME, Maclean A, et al. Teaching evidence based medicine to surgery residents-is journal club the best format? A systematic review of the literature. J Surg Educ 2012;69:91–100.
Contents lists available at ScienceDirect
Journal of Pediatric Surgery journal homepage: www.elsevier.com/locate/jpedsurg
A pilot investigation of a pediatric surgery journal club William Shihao Lao a, Pramod Puligandla b, Robert Baird b,⁎ a b
McGill University, Montreal Quebec, Canada Department of Pediatric General and Thoracic Surgery, The Montreal Children’s Hospital, McGill University, Montreal Quebec Canada
a r t i c l e
i n f o
Article history: Received 11 February 2014 Accepted 13 February 2014 Key words: Evidence based medicine Surgical journal club Journal club
a b s t r a c t Background: The CanMEDS competency “scholar” encompasses the creation, dissemination, application, and translation of medical knowledge. We hypothesize that a structured journal club (JC) for pediatric surgical trainees would meet these objectives in an enjoyable and long-lasting manner. Methods: A JC involving two pediatric surgery training programs was created with each session focusing on a specific study design. Pre-tests/post-tests were administered before/after each session with durability of learning assessed during the following session. Metrics analyzed included participant satisfaction and an appraisal of evidence-based medicine (EBM) principals. Test results were analyzed using the paired T-test with p b 0.05 considered significant. Results: On average, 14 participants attended each session, with all trainees present (4). While participants believed they understood EBM principles, 40% were unfamiliar with question formulation, 48% were unfamiliar with critical appraisal tools, and 60% had not appraised an article within the previous year. Pre-test to post-test comparison yielded an improvement in mean score (20 = perfect score): 10.8 to 16.9, p b 0.01. Measures of participant satisfaction were uniformly positive. Conclusion: A structured Pediatric Surgery Journal Club addresses scholarly training objectives in a highly satisfactory manner and yields durable learning. A web-based curriculum based on this model could serve as an important educational tool for trainees and attending staff alike. © 2014 Elsevier Inc. All rights reserved.
The principle at the heart of evidence-based medicine (EBM) is the application of the most current and valid scientific evidence to patient care [1]. Although the concept of EBM has been widely accepted as essential in the practice of modern medicine [2], there exists a general lack of emphasis and proper teaching of EBM throughout medical training. The scope of this problem is exemplified by the fact that combing through the thousands of new monthly publications requires hundreds of hours even for a well trained epidemiologist [3]. As such, the ability to pinpoint strong evidence in the literature is an ever more crucial skill for the modern clinician. Within the narrow field of pediatric surgery, the main obstacle to evidence-based clinical practice is not a lack of publications per se, but rather a general paucity of scientifically rigorous data. In 1999, only 0.3% of pediatric surgery publications were identified as prospective randomized trials [4]. In several pediatric surgical studies of lesser methodology, multiple barriers to the conduct of rigorous trials and creation of high-level evidence were identified [5,6]. Despite the weaknesses in the available evidence, these publications have unanimously stressed the importance of adopting and implementing evidence-based decision-making in clinical practice to ensure optimal patient care.
⁎ Corresponding author at: Montreal Children’s Hospital, 2300 Tupper Street, room C-812, Montreal, Quebec H3H1P3, Canada. Tel.: +1 514 412 4438; fax: +1 514 412 4289. E-mail address: [email protected] (R. Baird). http://dx.doi.org/10.1016/j.jpedsurg.2014.02.068 0022-3468/© 2014 Elsevier Inc. All rights reserved.
The CanMEDS Physician Competency Framework specifies 7 roles, which, when integrated, define the profession of the physician [7]. In one of these roles, the “scholar”, a physician must demonstrate a lifelong commitment to reflective learning, as well as the creation, dissemination, application and translation of medical knowledge [8]. Several studies have demonstrated that the incorporation of an evidence-based journal club into the training curriculum of surgical residents offers a high-yield method for teaching EBM [9–11]. This study seeks to determine the inherent value and efficacy of a pediatric surgery, domain-specific journal club as an educational method for conveying the principles of EBM to a select audience. 1. Methods The Quebec Pediatric Surgery Journal Club (QPSJC) is a 2-year-long curriculum consisting of 6 journal club sessions. It was instituted in 2012 and involves two accredited pediatric surgery training programs in Montreal, Quebec, Canada. These sessions cover various core concepts of evidence-based practice, including question formulation, research methodology, critical appraisal, and rudimentary biostatistics, with each session focusing on a specific study design. The curriculum consists of 1 session each dedicated to the following topics: question formulation, randomized controlled trials (RCTs), systematic reviews, case–control studies, database studies, and cohort design. Materials disseminated in advance included session-specific
812
W.S. Lao et al. / Journal of Pediatric Surgery 49 (2014) 811–814
learning objectives as well as the article to be appraised. The targeted audience included the 4 pediatric surgery fellows from both institutions, with additional participants consisting of 4–8 junior surgical residents, 2–4 medical students, and 4–8 attending pediatric surgeons. Each 2-h session was held in the evening outside of regular working hours. A pediatric surgery fellow was tasked with leading each session and was responsible for both the presentation and appraisal of the chosen article. Several weeks prior to each session, the journal article and session-specific targeted learning objectives were distributed electronically to all potential participants. Participation in the journal club activity was considered mandatory for pediatric surgical fellows only; trainees were invited but not obliged to complete questionnaires and testing through an informed consent process. At the start of each journal club cycle, a selfassessment questionnaire consisting of 8 elements was distributed to all attendees. The purpose of this questionnaire was to allow each participant to anonymously describe their views of EBM and to gauge their comfort with its core concepts. Grading was performed using a Likert scale from 1 to 5, with 1 being “strongly disagree”, and 5 being “strongly agree”. In addition, all participants attempted a pre-session test containing 10 questions that assessed their knowledge of core EBM principles and session specific concepts (e.g. power calculation for Randomized Controlled Trials). Each answer was evaluated individually and graded as 0, 1, or 2, for a maximum of 20 points. At the end of each session, all participants repeated the same test. Presession and post-session scores on the test were compared and analyzed in order to gauge the impact of the journal club session on the participants’ level of understanding. Answers to each test were only provided the following month as a means of reinforcing the acquired knowledge. At the following session’s pre-test, each participant was again assessed for retention and durability of learning. Finally, all attendees were given the opportunity to rate the journal club session through a session evaluation form. Variables analyzed included participant satisfaction, as well as session relevancy and usefulness. All scores and values are reported as a mean ± standard deviation (SD). Test results were analyzed using the Welch student T-test for uneven-sized samples with p b 0.05 considered significant. 2. Results On average, 14 participants were able to attend the sessions, with all 4 pediatric surgery fellows (target learners) present. In total, 11 participants completed the self-assessment questionnaire (Fig. 1).
Participants agreed that EBM was a valuable tool worth learning (mean 4.7 ± 0.47), with 80% of responders stating that they used evidence-based principles in practice (mean 4.0 ± 0.63). In addition, 75% of participants also believed they were familiar with the principles of EBM (mean 3.9 ± 0.70). However, in later statements querying each participant’s familiarity with said principles, up to 40% stated they were unfamiliar with standard question formulation (mean 3.4 ± 1.6), 50% were unfamiliar with critical appraisal tools (mean 3.5 ± 0.85), and 60% were unfamiliar with methods of acquiring the best evidence (mean 3.3 ± 1.1). Lastly, 60% of participants had not critically appraised an article within the previous year. In total, 9 of the attendees returned the pre-session test and 5 attendees returned the initial post-session test (Fig. 2). A comparison of pre-session and (early) post-session test scores revealed a significant improvement with the mean score increasing from 11.1 ± 5.7 to 16.2 ± 4.9 (max = 20; p = 0.05). Five participants returned the second (late) post-session test at the subsequent session. The mean score of this assessment of learning durability was 16.8 ± 3.6 (p = 0.03) compared to the original scores. Overall, all respondents (n = 9) were satisfied with the journal club (mean 4.4 ± 0.52), and found the articles to be clinically relevant (mean 4.0 ± 0.5) in addition to being well summarized and useful (mean 4.2 ± 0.44). Moreover, the methodological review was perceived by all to be both relevant (mean 4.4 ± 0.53) and helpful (4.6 ± 0.53). These results are summarized in Fig. 3.
3. Discussion It has been well demonstrated that one of the main barriers to evidence-based practice in pediatric surgery is the lack of high level studies to inform clinical decision-making [4,12,13]. Even when a well-conducted trial is performed, recognized hurdles inhibit practice change by impeding the dissemination, acceptance and application of that evidence. The results of this study suggest an additional fundamental barrier to evidence-based practice: the lack of knowledge of key EBM concepts. Addressing this problem is of utmost importance as a proper understanding of EBM not only aides in the critical appraisal of existing literature, but also allows for more thoughtful study design, in turn, ensuring the creation of more robust evidence. Ultimately, it is the generation of new knowledge that will drive the profession forward; graduating trainees must be equipped with the necessary tools to accomplish this.
Fig. 1. Summary of Principles of Evidence Based Medicine Self-Assessment Exercise (n = 11 respondents).
W.S. Lao et al. / Journal of Pediatric Surgery 49 (2014) 811–814
Fig. 2. Results of Pre-session (n = 9 respondents) and Post-session (n = 5 respondents) Evaluation.
This study indicates that a structured JC was effective in disseminating EBM principles as it employed many of the critical characteristics of successful journal clubs [9]. These include regular and anticipated meetings, mandatory attendance for trainees, clear long- and short-term objectives, appropriate meeting times and incentives, a trained journal club leader for choosing papers and leading discussions, using established critical appraisal processes, and summarizing journal club findings [14]. In addition, using the internet to circulate the papers and resources prior to the meeting allowed participants to easily access resources outside of sessions. In fact, one of the possible future milestones for the journal club is the creation of an online resource/database to supplement the current format [15,16]. For the time being, the ‘in person’ nature of the session ensured an engaged learning environment for all attendees. Several aspects of the JC were extremely positive. Fig. 1 demonstrated wide variability in learner self-assessment, which was also true specifically for the 4 members of the target audience. All respondents either agreed or strongly agreed with the statement that EB principles have value and are worth learning. While many participants believed they understood the methods of EBM, both the initial self-assessment and pre-session test demonstrated that this was not the case. In fact, a large portion of the participants did not comprehend the methodologies used for formulating questions, acquiring evidence, or appraising literature. A mean score below 60% on the
813
pre-session test further confirmed the finding. This discordance between the self-assigned familiarity and actual knowledge suggests the false perception that the core concepts of EBM are self-evident and do not require formal teaching. However, a significant improvement in the post-session scores illustrated that this unperceived learning need is well addressed by the JC as presented here. Several important limitations exist in the current journal club format. Despite all efforts to appeal to potential participants, incomplete attendance remains problematic. For example, not all staff surgeons from both institutions were able to attend the sessions, reducing their overall educational value and undermining the implied importance of the QPSJC to trainees in attendance. Furthermore, the small number of attendees at each session undermined the power of the present study, especially given that some of the participants did not complete the questionnaires or the tests. As the community of pediatric surgery in Montreal is fairly small in size, there is only a certain limit to which the power of this study can be increased. Moreover, the small sample size may also limit the external validity of our findings. Nonetheless, considerable variability likely exists in the background understanding of EBM concepts among practicing pediatric surgeons; a standardized EBM teaching curriculum addresses this variability by ensuring that core concepts are emphasized using examples derived from the field of pediatric surgery. In general, this pilot study is further limited by its small dataset. The fact that only 9 out of 14 (about 65%) participants completed the test raises the possibility that the results may not be completely representative of the group. As an even smaller number of participants (5) completed the post-session test, the validity of the increase in score is significantly undermined. Further compounding the problem is the anonymous way under which the tests were administered. With the research methodology as constructed, there was no way of determining whether the 5 participants who completed the post-session tests had done a pre-session test at all. One of the ways to address this problem in future sessions is to make all questionnaires mandatory and to ensure that all have been properly filled prior to starting a session, and after its end. In addition, as the journal club is held every three–four months, the elapsed time between each session may allow for significant “knowledge decay” over time. The full extent of such decay may only be confirmed or rejected once knowledge retention is assessed at future sessions. In fact, most successful journal clubs suggest meeting more than 3–4 times per year [14]. Another problem facing this study
Fig. 3. Summary of Session Evaluation (n = 9 respondents).
814
W.S. Lao et al. / Journal of Pediatric Surgery 49 (2014) 811–814
is the fact that the tools used for participant self-assessment and knowledge retention has not been validated. While examples of evaluative tools exist for evidence-based principles in surgery, these tend to address knowledge of EBM as a whole and do not sufficiently address the focus of specific sessions or this activity as a whole [17,18]. Our preliminary results indicate that a well structured, evidencebased Pediatric Surgery Journal Club is an appropriate educational method for addressing scholarly training objectives. Despite the small sample size, this study identified a significant “gap” between the perceived understanding of EBM principles by participants and objective findings based on a session pre-test. This was remedied through the current activity, as participants were able to achieve highyield, durable learning in an appealing and organized fashion. As a future development, a web-based curriculum modeled after the current methods may serve as an easily-accessible educational tool for trainees and attending staff throughout North America.
References [1] Sackett DL, Rosenberg WM, Gray JA, et al. Evidence based medicine: what it is and what it isn't. BMJ 1996;312:71–2. [2] Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet 2003;362:1225–30. [3] Alper BS, Hand JA, Elliott SG, et al. How much effort is needed to keep up with the literature relevant for primary care? J Med Libr Assoc 2004;92:429–37. [4] Hardin Jr WD, Stylianos S, Lally KP. Evidence-based practice in pediatric surgery. J Pediatr Surg 1999;34:908–12.
[5] Rangel SJ, Kelsey J, Henry MC, et al. Critical analysis of clinical research reporting in pediatric surgery: justifying the need for a new standard. J Pediatr Surg 2003;38:1739–43. [6] Curry JI, Reeves B, Stringer MD. Randomized controlled trials in pediatric surgery: could we do better? J Pediatr Surg 2003;38:556–9. [7] Cruess SR, Johnston S, Cruess RL. "Profession": a working definition for medical educators. Teach Learn Med 2004;16:74–6. [8] Frank JR. The CanMEDS 2005 physician competency framework. Better standards. Better physicians. Better care. Ottawa, Canada: The Royal College of Physicians and Surgeons of Canada; 2005. [9] Lee AG, Boldt HC, Golnik KC, et al. Structured journal club as a tool to teach and assess resident competence in practice-based learning and improvement. Ophthalmology 2006;113:497–500. [10] Spillane AJ, Crowe PJ. The role of the journal club in surgical training. Aust N Z J Surg 1998;68:288–91. [11] Shifflette V, Mitchell C, Mangram A, et al. Current approaches to journal club by general surgery programs within the southwestern surgical congress. J Surg Educ 2012;69:162–6. [12] Ostlie DJ, St Peter SD. The current state of evidence-based pediatric surgery. J Pediatr Surg 2010;45:1940–6. [13] Blakely ML, Lally KP. Using evidence-based medicine principles to improve quality of patient care in pediatric surgery. Semin Pediatr Surg 2002;11:42–5. [14] Deenadayalan Y, Grimmer-Somers K, Prior M, et al. How to run an effective journal club: a systematic review. Clin Pract 2008;14:898–911. [15] Kuppersmith RB, Stewart MG, Ohlms LA, et al. Use of an Internet-based journal club. Otolaryngol Head Neck Surg 1997;116:497–8. [16] Macrae HM, Regehr G, McKenzie M, et al. Teaching practicing surgeons critical appraisal skills with an Internet-based journal club: a randomized, controlled trial. Surgery 2004;136:641–6. [17] Toedter LJ, Thompson LL, Rohatgi C. Training surgeons to do evidence-based surgery: a collaborative approach. J Am Coll Surg 2004;199:293–9. [18] Ahmadi N, McKenzie ME, Maclean A, et al. Teaching evidence based medicine to surgery residents-is journal club the best format? A systematic review of the literature. J Surg Educ 2012;69:91–100.
