2014, 1–10, Early Online
Development and validation of the AMEET inventory: An instrument measuring medical faculty members’ perceptions of their educational environment SYED ILYAS SHEHNAZ, GAMINI PREMADASA, MOHAMED ARIFULLA, JAYADEVAN SREEDHARAN & KADAYAM GURUSWAMI GOMATHI Gulf Medical University, United Arab Emirates
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Abstract We developed, through a three-round modified Delphi technique, an inventory [Assessment of Medical Education Environment by Teachers (AMEET)] to assess viewpoints of medical faculty on educational environment experienced by teachers. In Round 1, 16 members from international settings graded (0–3) and modified an initial list of statements, or suggested new statements for inclusion. In Round 2, the new statements were graded and previous responses reconsidered. Subsequently, statements with median scores 1 were excluded. The questionnaire was compiled following reconsideration of responses to new statements in Round 3. A pilot test enabled assessment of the validity and reliability of the instrument. The panel members modified and graded the initial 50 statements of the preliminary inventory and suggested five new statements. The new statements were graded, responses to previous 50 statements were reconsidered, and three statements with median scores 51 were removed following Round 2. On review of the 52 statements by panel, two statements (each with agreement of575%) were deleted. The final 50-item inventory comprising six domains was piloted with 62 faculty. Cronbach’s alpha of 0.94 was obtained. The AMEET inventory is valid, highly reliable and practically useful to assess medical faculty members’ perceptions of educational environment.
Introduction
Practice points
Understanding educational environment (EE) is important for effective learning in health professions and has been emphasized by Genn (2001) and Harden (2001). A conducive EE will result in positive student-learning outcomes and satisfaction. The environment of any medical school influences the behavior of both students and the faculty. The EE experienced by teachers involves all the conditions affecting teaching and learning activities. Faculty members’ EE is influenced by the organizational structure of the institution and may include aspects such as their ‘‘involvement in the process of curriculum development and delivery; value and respect rendered to the staff and their contribution to curriculum development; enhancement of staff well being; concrete measures to reduce staff stress’’, etc. (Genn 2001). Faculty perceptions of the EE are important not only for teachers but also for students. The perceptions of the participants/inhabitants regarding the environment influence their final behavior and hence, studying their perceptions can provide valuable input for improving the quality of the EE in any institution (Genn 2001). A great deal of attention has been given to the health professions students’ perspective about the EE and instruments assessing the students’ perceptions have been developed and
Educational environment (EE) experienced by teachers involves all conditions which affect teaching and learning activities. The faculty EE is closely linked to the students’ EE and has a strong impact on students’ achievements and levels of satisfaction. The AMEET questionnaire is a valid, highly reliable, and practically useful instrument to assess medical faculty members’ perceptions of educational environment in their institution.
validated. A systematic review identified 31 instruments for measuring the students’ perceptions of EE in health professions (Soemantri et al. 2010). However, to the best of our knowledge, only two studies have looked into perspectives of the other stakeholders such as the teaching staff (Miles & Leinster 2009; Shehnaz et al. 2012). Moreover, in these latter studies, the faculty stated their opinion about the students’ EE (as experienced by the students) and not of the faculty’s own educational environment. In view of the relevance of the faculty’s role in the EE of any institution and in the absence of a
Correspondence: Dr. Syed Ilyas Shehnaz, Department of Pharmacology, Gulf Medical University, P.O. Box 4184, Ajman, United Arab Emirates. Tel: 97 16 7431333x316; Fax: +97 16 7431222; E-mail: [email protected] ISSN 0142-159X print/ISSN 1466-187X online/14/000001–10 ß 2014 Informa UK Ltd. DOI: 10.3109/0142159X.2014.947935
1
S. I. Shehnaz et al.
specific and appropriate instrument to evaluate their viewpoints, we aimed to develop and validate an inventory to measure the medical faculty members’ perceptions of the EE at their institutions. The Assessment of Medical Education Environment by Teachers (AMEET) is the product of our endeavor.
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Methods A three-stage methodology was used in developing the instrument: (1) literature review and group discussions, (2) modified Delphi technique to identify consensus, and (3) pilot study. In the first stage, we identified the theoretical framework(s) underpinning the pertinent concepts that should be measured in establishing the EE of the teaching faculty and subsequently the characteristics to be included in the AMEET inventory. We anticipated that a sound theoretical framework would circumvent shortcomings in our questionnaire. For this, in September 2011, we followed the Cochrane Library and the Best Evidence Medical Education (BEME) guidelines (Best Evidence Medical Education Collaboration 2003; Moher et al. 2009) and MEDLINE/PUBMED, Academic Search Premier, CINAHL, EMBASE, ERIC, Ingentaconnect, Topics in Medical Education (TIMELIT), Google Scholar, PsycARTICLES, and PsycINFO were searched for publications related to EE of the faculty. Primary search terms curriculum; educational environment/ climate; learning environment; medical faculty; perception; undergraduate medical education; survey, test, scale, measure, instrument, inventory or questionnaire were used in various combinations. In Medline (MESH database), the search terms education, medical, and undergraduate were used in MAJR (main subject heading) combined (AND) with medical faculty and perceptions. All search terms were ‘‘exploded’’. Articles with both abstracts and free full texts published in English and English abstracts of articles published in other languages involving the EE in health professions education (undergraduate or postgraduate) were scrutinized. A manual search of leading medical education journals: Medical Education, Academic Medicine, Advances in Health Sciences Education, and Medical Teacher was also performed. Hand search of reference lists, potentially relevant citations of articles identified through the primary search and review articles dealing with EE in health professions education, was also conducted. As correlates of staff well-being are presumed to be part of the faculty EE, we also searched articles related to the faculty development efforts. Both qualitative and quantitative studies were included. Non-English language articles and any publications before January 1990 were excluded from the study. The preliminary inventory of statements, supported by comprehensive theoretical frameworks and developed from the literature review, were then reviewed and modified on the basis of relevance to teachers’ EE by an expert group (research team) of one medical education expert (GP), one faculty medical doctor (SS), and three professors in the medical faculty (M. A., J. S., and K. G. G.) through four rounds of discussions. 2
In the second stage, a three-round modified Delphi approach was used to derive agreement on generation and classification of the statements of the preliminary inventory. The Delphi procedure achieved consensus among a panel of experts through a sequence of structured rounds involving opinion generation (agreement/disagreement) on statements or concepts. Our Delphi panel was derived by a two-stage process. The members identified could be categorized into at least one of the following: medical education experts with minimum two years’ experience; basic sciences faculty, clinical faculty, and clinical psychologists with at least five years’ experience. Subsequently, we included a ‘‘nomination’’ process in which the identified experts were asked to recommend other eligible experts (Fish & Busby 1996). We invited members with the eligibility criteria listed above to become part of the panel, included other proposed eligible experts and short listed all those who were willing to take part. All participants were assured that anonymity would be maintained and an e-mail consent was obtained. In Round 1, the panel members were asked to accept, reject, or question the inclusion of each statement of the preliminary inventory. They were invited to modify the statements, add new statements, or submit other comments, suggestions or remarks explaining their ratings. The panel was asked, using a Likert-type scale where 0 ¼ not necessary, 1 ¼ desirable, 2 ¼ important and 3 ¼ absolutely essential, to rate the inclusion of the statements. The survey was administered online with the panel members’ opinions collected independently and anonymously. Only statements with median scores 1 point or more were incorporated into the next-stage questionnaire, as a score below 1 indicated ‘‘not necessary’’. Free text comments were noted, and suggestions regarding format and clarification incorporated in the subsequent round. When several suggestions were given for a single item, the most commonly suggested modification was used. The results of the completed questionnaires were assessed by the expert group and a second inventory was developed for the Round 2 of the modified Delphi process. The Delphi panel accepted/rejected or modified/queried the inclusion of newly suggested statements by using the previous grading scheme and also reconsidered their response for the previous statements taking into consideration the Delphi group members’ percentage responses for each one of the statements. Following this, the statements with median scores 1 were removed from analysis. In the final round (Round 3), the responses to the newly suggested statements were reconsidered and the final inclusion of each statement was indicated through agreement/ disagreement/uncertainty. Panel agreement 4 75% was deemed necessary for inclusion of the statement into the final AMEET inventory. Subscales of the AMEET questionnaire were subsequently identified through group discussions among the expert group and confirmed using inputs from the Delphi panel (obtaining validity evidence based on test content) (American Education Research Association and American Psychological Association 1999). The Delphi panel was provided with a preliminary list of domains, the statements constituting it and requested to
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Development and validation of AMEET inventory
indicate agreement to the domain allocation or to reallocate the statement to their preferred domain. If at least 11 participants assigned a statement to the same domain, we considered the implication of the statement was clear and allocated the statement to that domain. The third stage (pilot study) was intended to refine the AMEET inventory involving the statements approved by the Delphi panel and evaluate its reliability. The inventory was piloted with a small group of faculty for face validity, and ambiguities in the statements were cleared. Finally, the selfadministered AMEET questionnaire was anonymously answered by all the teaching faculty of the institution. Each statement was scored on a Likert-type scale: 0 ¼ strongly disagree, 1 ¼ disagree, 2 ¼ uncertain, 3 ¼ agree and 4 ¼ strongly agree. The percentage responses, median and range for statements of Rounds 1 and 2 were calculated. The coefficient of range was also calculated to compare the reduction in the variation of scores from Round 1 to Round 2 (for preliminary statements) and from Round 2 to Round 3 (for newly suggested statements). The internal consistency of the inventory and its subscales were checked using Cronbach’s alpha. All the statistical calculations were performed using SPSS-20 (SPSS Inc., Chicago, IL). The study was approved by the institutional Ethics Committee and all participating faculty gave a written informed consent.
Results Stage I: Literature review and group discussions The literature search identified more than 100 articles related to EE and specifically seven recently developed inventories related to medical students’ EE. The Dundee Ready Education Environment Measure (DREEM) (Roff et al. 1997), the Postgraduate Hospital Educational Environment Measure (PHEEM) (Roff et al. 2005), the Dutch Residency Educational Climate Test (D-RECT) (Boor et al. 2011), DR-CLE questionnaire – diagnostic radiology clinical learning environment questionnaire (Bloomfield & Subramaniam 2008), STEEM – surgical theatre educational environment measure (Cassar 2004), ATEEM – the Anaesthetic Theatre Educational Environment Measure (Holt & Roff 2004) and the Clinical Learning Environment (CLE) (Saarikoski & Leino-Kilpi 1999), the Supervision and Dental Student Learning Environment Survey (DSLES) (Henzi et al. 2005) are some of the reported valid and reliable instruments for undergraduate medical, postgraduate medical, nursing, and dental students, respectively. Three review articles also gave insights into the various instruments for measuring the EE among different health professions student groups (McAleer et al. 2009; Soemantri et al. 2010; Scho¨nrock-Adema et al. 2012) and identified teaching, supervision, training, atmosphere, emotional climate, social support, learning opportunities, orientation to learning, workload, goal, direction, and emphasis on scholarship as common themes in the instruments. Scho¨nrock-Adema et al. (2012) gave insights into three domains (based on the framework formulated by Moos 1973,
1974) related to goal orientation (the content and aims of education, which involves concepts such as clarity about learning objectives, relevant learning content, and constructive criticism); relationships (an open and friendly atmosphere and affiliation, certain characteristics of which include student involvement, support freedom of expression, and involvement in educational settings); and organization/regulation (organization, role clarity, teacher control, innovation, work pressure, and physical comfort) as theoretical frameworks for almost all student EE inventories. Only two previous studies identified what medical faculty perceived about the students’ experiences of their EE (Miles & Leinster 2009; Shehnaz et al. 2012). In contrast, much research had been done in assessing school teachers’ climate (Rentoul & Fraser 1983; Burden & Fraser 1994; Johnson & Stevens 2001; Johnson et al. 2007), which was based on Moos’s three dimensions (Moos 1973, 1974). As almost all the EE inventories for health professions students and school teachers had the same three domains (goal orientation; relationships, and organization/regulation) as the key elements, we postulated that this framework was also applicable to the EE of medical faculty and we have used it as theoretical underpinning for the development of our preliminary inventory. The preliminary inventory of statements, based on the three domains of theoretical frameworks, following extensive literature search and the four expert group discussions review, consisted of 50 statements.
Stage II: Modified Delphi process In September 2012, we invited 10 medical educationalists, 20 specialists (clinical and non-clinical specialty tutors), and 10 psychologists to be members in our Delphi panel. The 40 panelists were chosen considering their involvement in undergraduate education, possession of a certificate/diploma level of medical educational training (for clinical/non-clinical faculty and psychologists), and worldwide geographical representation (USA, UK, Australia, India, and Sri Lanka). No monetary incentives were offered for involvement in the complete Delphi procedure and voluntary participation was emphasized. A brief introduction to the study, research objectives and details of all three rounds of the modified Delphi were given at the outset, and consent solicited assuring confidentiality for all participants. Twenty-five faculty members consented to participate in the Delphi panel (nine medical educationalists, seven clinical faculty, six non-clinical faculty, and three psychologists). After one month, the preliminary inventory was sent to the consenting panel members, who modified and graded each statement for its relevance with respect to faculty EE. Completed inventories were returned by 20 of the 25 participants in Round 1 to SS who summarized the remarks. Five of the participants (one medical educationalist, two clinicians, one non-clinical faculty and one psychologist) did not reply, despite telephone/e-mail reminders and the offer of another month for completion of the inventory. Based on the experts’ suggestions, five new statements were added to Round 2 inventory and certain statements were simplified to improve clarity.
3
S. I. Shehnaz et al.
The second inventory with 55 statements was sent to the 20 experts who had answered in Round 1. The five new statements added were modified, graded and the responses to the previous 50 statements were reconsidered in Round 2. Four panel members did not complete Round 2 in spite of repeated reminders. Table 1 displays the outcomes of Rounds 1 and 2. Following this round, three statements (Nos. 4, 20, and 26) were removed as their median scores were less than 1. In the last round (Round 3), the responses for the five new statements were reconsidered (Table 2) and agreement/disagreement/uncertainty solicited from the 16 panel members for the final inclusion of the 52 statements. There were no significant differences in the median panel responses between Rounds 1 and 2 for the 50 statements and
between Rounds 2 and 3 for the five new statements. The coefficient of range showed that the deviation was reduced for 15 preliminary statements (from Round 1 to Round 2) and three newly suggested statements (from Round 2 to Round 3). No change was observed for the rest of the 35 preliminary statements and two newly suggested statements (Tables 1 and 2). The minimum reduction observed for the 18 statements was 30% and the maximum reduction observed was 80%. Analysis after Round 3 resulted in deletion of two more statements as panel agreement was 575% (Table 3). Consensus was reached in this round: 39 statements garnered more than 90% agreement. The final AMEET inventory comprised 50 statements. Since non-participation influences changes in the median, the final analysis was restricted to only those 16 panel members who participated in all three rounds.
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Table 1. Percentage ratings and median (range) for Rounds 1 and 2 (n ¼ 16).
Round 1
Round 2
Percentage responses
S. no. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
23 24 25 26 27 28 29
Percentage responses
Statements
NN 0
D1
I2
AE 3
Median (range)
NN 0
D1
I2
AE 3
Median (range)
The teacher finds his/her role as a teacher interesting There is too much of emphasis on learning factual information The teachers are domineering and authoritative The teachers make the students look foolish The comments given by the teachers help students to improve themselves The preparation that the teachers undertake for doing their classes is adequate The emphasis in classes is on what the teacher does rather than on what the student learns The content and the teaching methods match the learning objectives of the course The teacher respects the students’ viewpoints The learning is ‘‘student centered’’ Students have sufficient opportunities to develop competence Students have sufficient opportunities to develop confidence The students have a clear idea of what they are expected to gain from the course The class environment makes the students want to get actively involved The students are encouraged to be life-long learners Students are encouraged to understand and apply what they learn rather than remember isolated facts The students are well mannered and respectful The students complete their assignments on time The students are motivated to learn The students’ attitude and activities annoy the teachers The students come well prepared for their learning activities The students are not sufficiently prepared for the formative assessments (the assessments conducted for giving feedback/practice) The students feel comfortable in learning with other students coming from varied background(s) The atmosphere in the hospital setting is conducive to learning The time allotted to different learning sessions is appropriate Cheating at examinations is widespread at the institution The atmosphere during most of the teaching/learning activities is conducive to learning The students have sufficient opportunities to improve their communication skills The students find the learning experience disappointing
18.8 6.3 31.3 56.3 –
18.8 43.8 18.8 12.5 6.3
18.8 31.3 37.5 18.8 43.8
43.8 18.8 12.5 12.5 50.0
2.0(3) 1.5(3) 1.5(3) 0.0(3) 2.5(2)
6.3 – 18.8 62.5 –
18.8 50.0 25.0 18.8 –
25.0 31.3 50.0 12.5 25.0
50.0 18.8 6.3 6.3 75.0
2.5(3) 1.5(2)a 2.0(3) 0.0(3) 3.0(1)a
–
18.8
31.3
50.0
2.5(2)
–
6.3
37.5
56.3
3.0(2)
6.3
18.8
25.0
50.0
2.5(3)
–
12.5
43.8
43.8
2.0(2)a
6.3
6.3
25.0
62.5
3.0(3)
–
–
31.3
68.8
3.0(1)a
– – – – –
12.5 6.3 12.5 31.3 6.3
31.3 31.3 31.3 18.8 31.3
56.3 62.5 56.3 50.0 62.5
3.0(2) 3.0(2) 3.0(2) 2.5(2) 3.0(2)
– – – – –
6.3 6.3 – 25.0 –
43.8 25.0 31.3 31.3 25
50.0 68.8 68.8 43.8 75
2.5(2) 3.0(2) 3.0(1)a 2.0(2) 3.0(1)a
–
12.5
18.8
68.8
3.0(2)
–
6.3
25.0
68.8
3.0(2)
– –
6.3 –
25.0 25.0
68.8 75.0
3.0(2) 3.0(1)
– –
6.3 –
18.8 18.8
75.0 81.3
3.0(2) 3.0(1)
18.8 6.3 – 50 – 18.8
25.0 25.0 6.3 18.8 31.3 25.0
37.5 62.5 37.5 18.8 43.8 37.5
18.8 6.3 56.3 12.5 25.0 18.8
2.0(3) 2.0(3) 3.0(2) 0.5(3) 2.0(2) 2.0(3)
18.8 – – 56.3 – 18.8
18.8 25.0 6.3 12.5 25.0 18.8
43.8 68.8 31.3 18.8 50.0 50.0
18.8 6.3 62.5 12.5 25.0 12.5
2.0(3) 2.0(2)a 3.0(2) 0.0(3) 2.0(2) 2.0(3)
–
12.5
50.0
37.5
2.0(2)
–
6.3
56.3
37.5
2.0(2)
– – 50 –
25.0 25.0 18.8 12.5
25.0 37.5 18.8 56.3
50.0 37.5 12.5 31.3
2.5(2) 2.0(2) 0.5(3) 2.0(2)
– – 56.3 –
6.3 6.3 12.5 12.5
37.5 56.3 18.8 56.3
56.3 37.5 12.5 31.3
3.0(2) 2.0(2) 0.0(3) 2.0(2)
–
18.8
31.3
50.0
2.5(2)
–
–
43.8
56.3
3.0(1)a
25.0
43.8
18.8
12.5
1.0 (3)
25.0
43.8
18.8
12.5
1.0(3)
(continued )
4
Development and validation of AMEET inventory
Table 1. Continued.
Round 1
Round 2
Percentage responses
Statements
NN 0
D1
I2
AE 3
Median (range)
NN 0
D1
I2
AE 3
Median (range)
The teaching learning experience is very stressful for the students The students find studying medicine enjoyable The students’ feel comfortable about asking any questions they want The students’ feedback about the curriculum is taken into consideration Teachers are encouraged to provide suggestions regarding curriculum Teachers are involved in collaborative efforts related to educational activities Teachers are encouraged to teach a trans-disciplinary approach to health problems Teachers have good interpersonal communication skills for dealing with peers and students Teachers are flexible and open to change/suggestions The attitudes of the teachers are conducive to effective team work Teachers dominate over subordinates, peers, or students I am satisfied with the opportunities that I get for working with other faculty There is a formal support system for faculty who get stressed Teachers have sufficient time to plan their teaching activities I am knowledgeable in educational concepts for my role as a teacher I possess the necessary teaching skills for undertaking my duties I have effective communication skills for dealing with hospital patients Opportunities are available for trying out alternative approaches in teaching I have opportunities for learning research methodology and research communication skills I have opportunities for developing the skills of educational administration The Faculty Development Programs are adequate in preparing me for the tasks expected of me
18.8
25.0
43.8
12.5
2.0(3)
31.3
18.8
43.8
6.3
1.5(3)
6.3 6.3
37.5 6.3
18.8 43.8
37.5 43.8
2.0(3) 2.0(3)
6.3 6.3
43.8 6.3
18.8 37.5
31.3 50.0
1.5(3) 2.5(3)
–
12.5
43.8
43.8
2.0(2)
–
6.3
50.
43.8
2.0(2)
–
6.3
31.3
62.5
3.0(2)
–
–
31.3
68.8
3.0(1)a
–
6.3
37.5
56.3
3.0(2)
–
–
43.8
56.3
3.0(1)a
–
12.5
31.3
56.3
3.0(2)
–
6.3
37.5
56.3
3(2)
–
12.5
31.3
56.3
3.0(2)
–
6.3
25.0
68.8
3.0(2)
– –
6.3 –
50.0 43.8
43.8 56.3
2.0(2) 3.0(1)
– –
– –
56.3 31.3
43.8 68.8
2.0(1)a 3.0(1)
18.8 –
25.0 18.8
37.5 31.3
18.8 50.0
2.0(3) 2.5(2)
25.0 –
31.3 18.8
31.3 37.5
12.5 43.8
1.0(3) 2.0(2)
– – –
25.0 18.8 12.5
43.8 50.0 43.8
31.3 31.3 43.8
2.0(2) 2.0(2) 2.0(2)
– – –
18.8 18.8 6.3
56.3 50.0 50.0
25.0 31.3 43.8
2.0(2) 2(2) 2(2)
– 6.3
6.3 12.5
43.8 43.8
50.0 37.5
2.5(3) 2.0(3)
– –
6.3 6.3
31.3 50.
62.5 43.8
3(3) 2.0(2)a
6.3
25.0
31.3
37.5
2.0(3)
–
6.3
50.0
43.8
2.0(2)a
6.3
31.3
12.5
50.0
2.5(3)
–
31.3
18.8
50.0
2.5(2)a
6.3
25.0
25.0
43.8
2.0(3)
–
31.3
18.8
50.0
2.5(2)a
–
–
62.5
37.5
2.0(1)
–
–
56.3
43.8
2.0(1)
S. no. 30 31 32 33 34 35
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
Percentage responses
NN, not necessary; D, desirable; I, important; AE, absolutely essential. Statements in bold: median score 5 1. a Statements with reduced coefficients of range.
The expert group initially suggested six subscales and some statements were reallocated to different domains on the advice of the Delphi panel. The statements were allocated to six domains according to the content of the statements: Teachers’ Perceptions of Teaching (nine statements); Teachers’ Perceptions of Learning Activities (nine statements); Teachers’ Perceptions of Students (six statements); Teachers’ Perceptions of Learning Atmosphere (10 statements); Teachers’ Perceptions of Collaborative Atmosphere (eight statements); Teachers’ Professional Self-Perceptions (eight statements).
Stage III: Pilot study The 50-item AMEET inventory was piloted with a representative small group of faculty (n ¼ 10) to confirm the face validity. As a result, three statements were re-written for better clarity. The inventory was then self-administered to the teaching faculty of the institution (n ¼ 62; response rate ¼ 89%). The
resulting internal consistency coefficient Alpha computed from the data is shown in Table 4. Figure 1 summarizes the methodological process and findings of the three stages of the development of the AMEET inventory.
Discussion The development of the AMEET inventory, a questionnaire to evaluate the teacher’s educational climate, was directed through a diverse methodology involving theoretical frameworks based on earlier findings (Scho¨nrock-Adema et al. 2012), a modified Delphi procedure and a pilot test. The key features of the Delphi are the multiple consultation rounds, structured flow of information, analysis of the collective panel responses in each round which is then updated to the Delphi members to facilitate the development of the questionnaire of the next round, the opportunity for experts to give qualitative inputs and reconsider their own ratings in the
5
S. I. Shehnaz et al.
Table 2. Percentage ratings and median (range) for five new statements introduced in Round 2 (n ¼ 16).
Round 2
Round 3
Percentage responses
S. no. 51 52 53
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
54 55
Statements The teachers adopt a variety of teaching methods to cater to diverse styles of learning Adequate teaching and learning resources (including physical and material) are available The students have sufficient opportunities to discuss their learning problems with the faculty My peers appreciate when I make professional progress I am encouraged to go to conferences that improve my knowledge and skills
NN 0
D1
I2
AE 3
6.3
31.3
31.3
31.3
–
18.8
43.8
–
12.5
12.5 6.3
43.8 37.5
Percentage responses Median (range)
D1
2.0(3)
–
18.8
50
31.3
2.0(2)a
37.5
2.0(2)
–
12.5
56.3
31.3
2.0(2)
56.3
31.3
2.0(2)
–
6.3
68.8
25
2.0(2)
43.8 43.8
– 12.5
1.0(2) 2.0(3)
6.3 –
50 50
–
2.0(1)a 2.0(2)a
43.8 43.8
I2
AE 3
Median (range)
NN 0
6.3
NN, not necessary; D, desirable; I, important; AE, absolutely essential. a Statements with reduced coefficients of range.
light of the regular summarized feedback ratings of the complete panel, the anonymity of the panel providing unbiased opinions in a non-threatening environment, and participation of experts across geographical barriers precluding their physical presence (Jones & Hunter 1995; Clayton 1997; Hasson et al. 2000). Many modifications to the classical Delphi technique have been proposed and one such modification involves the generation of initial statements from literature review rather than proposed by the expert panel (de Villiers et al. 2005). In our study too, the initial statements were not generated by the expert panel but from previous research and the concept of explicating the most important issues by deleting the less important ones through several rounds was maintained (Keeney et al. 2005; Thangaratinam & Redman 2005; Wallengren 2011). The selection of the Delphi expert panel is paramount for the validity of the process. To select our Delphi panel experts, we used a purposive sampling technique which utilizes the researcher’s expert knowledge about the general population to identify the cases to be included in the sample (Polit et al. 2001). Our final Delphi panel had 16 members who were medical educationalists, clinical, and non-clinical faculty or psychologists. It has been suggested that adequate panel size ranged from 15 to 30 members since groups larger than 30 is unmanageable and rarely improves the outcome, and smaller samples produce invalid numbers in the final rounds due to dropouts (Clayton 1997). We selected the Delphi technique to content validate the inventory through the opinions of a heterogenic group of experts in spite of geographical barriers. The iterative nature of the Delphi process also augmented the concurrent validity and the good response rates in the three rounds have not affected the validity adversely (Hasson et al. 2000). For analysis of the responses in Rounds 1 and 2, the percentage responses along with the medians and ranges were used (rather than mean and standard deviation) because a Likert-type scale (an ordinal scale) was used to grade the statements. The collected opinion had been subsequently 6
communicated back to the panel at the end of Rounds 1 and 2 (Hasson et al. 2000). This ensured that the panel members were aware of their own responses in relation to those of the group and also verified the level of agreement among respondents (Jones & Hunter 1995). The reduction in coefficient of range for 18 statements (30–80%) indicated that the agreement between the participants increased as the rounds progressed. Moreover, the agreement between the participants was not reduced in the rest of the 35 preliminary statements and the two newly suggested statements as no increase in the coefficient of range was detected. In order to maintain a manageable list of statements, some studies have also restricted statements from progressing to the next round if they did not meet a predetermined level of consensus (Wallengren 2011). In our study, the level of consensus was set at a median score of 1. The results of the final round indicated that the stable consensus for almost all statements had been attained as around 80% of the statements were unanimously agreed on by the panel (90–100% agreement). The achievement of consensus is the quintessence of the Delphi technique and offers scope for maximum contention for this research technique. Hence, it is imperative that consensus be predetermined and non-consensus clearly interpreted. The convergence of opinions with minimal divergence typically identifies that consensus is attained and this is characteristically reached after three rounds with maximum variations occurring between Rounds 1 and 2 (Bellamy et al. 1991). However, there is no unanimously agreed level, with different studies reporting consensus attained anywhere between 51% and 80% agreement among Delphi panel members (Green et al. 1999; de Villiers et al. 2005) or indicated by stability of the response through the iterative process (Crisp et al. 1997). We had pre-specified a panel agreement of more than 75% as attainment of consensus. It is generally agreed that the minimum acceptable value of Cronbach’s alpha is 0.70. The high alpha coefficients of the AMEET inventory and most of its domains in our pilot study indicated that this instrument has adequate reliability for
Development and validation of AMEET inventory
Table 3. Percentage agreement/disagreement and finalized domains in Round 3.
Statements 1 2 3 4 5 6 7
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
8 9 10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
Teachers’ perceptions of teaching I find my role as a teacher interesting There is too much of emphasis on learning factual information The teachers are domineering and authoritative The teachers make the students look foolish The comments given by the teachers help students to improve themselves The preparation that the teachers undertake for doing their classes is adequate The emphasis in classes is on what the teacher does rather than on what the student learns The content and the teaching methods match the learning objectives of the course The teacher respects the students’ viewpoints Teachers’ perceptions of learning activities The learning is ‘‘student centered’’ Students have sufficient opportunities to develop competence Students have sufficient opportunities to develop confidence The students have a clear idea of what they are expected to gain from the course The class environment makes the students want to get actively involved The students are encouraged to be life-long learners Students are encouraged to understand and apply what they learn rather than remember isolated facts Teachers’ perceptions of students The students are well mannered and respectful The students complete their assignments on time The students are motivated to learn The students’ attitude and activities annoy the teachers The students come well prepared for their learning activities The students are not sufficiently prepared for the formative assessments (the assessments conducted for giving feedback/practice) The students feel comfortable in learning with other students coming from varied background(s) Teachers’ perceptions of learning atmosphere The atmosphere in the hospital setting is conducive to learning The time allotted to different learning sessions is appropriate Cheating at examinations is widespread at the institution The atmosphere during most of the teaching/learning activities is conducive to learning The students have sufficient opportunities to improve their communication skills The students find the learning experience disappointing The teaching learning experience is very stressful for the students The students find studying medicine enjoyable The students’ feel comfortable about asking any questions they want The students’ feedback about the curriculum is taken into consideration Teachers are encouraged to provide suggestions regarding curriculum Teachers’ perceptions of collaborative atmosphere Teachers are involved in collaborative efforts related to educational activities Teachers are encouraged to teach a trans-disciplinary approach to health problems Teachers have good interpersonal communication skills for dealing with peers and students Teachers are flexible and open to change/suggestions The attitudes of the teachers are conducive to effective team work Teachers dominate over subordinates, peers or students I am satisfied with the opportunities that I get for working with other faculty There is a formal support system for faculty who get stressed Teachers have sufficient time to plan their teaching activities Teachers’ professional self-perceptions I am knowledgeable in educational concepts for my role as a teacher I possess the necessary teaching skills for undertaking my duties I have effective communication skills for dealing with hospital patients Opportunities are available for trying out alternative approaches in teaching I have opportunities for learning research methodology and research communication skills I have opportunities for developing the skills of educational administration The Faculty Development Programs are adequate in preparing me for the tasks expected of me The teachers adopt a variety of teaching methods to cater to diverse styles of learning Adequate teaching and learning resources (including physical and material) are available The students have sufficient opportunities to discuss their learning problems with the faculty My peers appreciate when I make professional progress I am encouraged to go to conferences that improve my knowledge and skills
Percentage agreement
Percentage disagreement
Percentage uncertainty
93.8 100 68.8 Deleted 100 100 87.5
– – 6.3
6.3 – 25
– – 6.3
– – 6.3
100 100
– –
– –
100 100 100 100 100 100 100
– – – – – – –
– – – – – – –
12.5 – –
6.3 – –
– 6.3
6.3 12.5
–
–
81.3 100 100 Deleted 93.8 81.3 100
93.8 93.8 Deleted 100 100 81.3 81.3 87.5 100 93.8 100
– –
6.3 6.3
– – 12.5 18.8 6.3 – – –
– – 6.3 – 6.3 – 6.3 –
100 93.8 100
– – –
– 6.3 –
100 100 75 87.5 93.8 100
– – 18.8 6.3 – –
– – 6.3 6.3 6.3 –
100 93.8 93.8 100 100
– – – – –
– 6.3 6.3 – –
93.8 100
6.3 –
– –
100 100 100
– – –
– – –
25.0 6.3
6.3 12.5
68.8 81.3
Statements in bold: agreement5 75%.
7
S. I. Shehnaz et al.
measurement at our institution, although the coefficient for perceptions of teaching was slightly low (Table 4). The AMEET inventory reveals the medical teachers’ perceptions of EE and also sheds light on other dependent factors such as professional self-development, inter-personal relationships, and student achievement. This instrument may, therefore, be useful in the following: The diagnosis of the strengths and weaknesses of curricula in a health professions setting. This may help
Table 4. Cronbach’s alpha for domains and full meet inventory.
Number Cronbach’s of items alpha
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Domains Teachers’ perceptions of teaching Teachers’ perceptions of learning activities Teachers’ perceptions of students Teachers’ perceptions of learning atmosphere Teachers’ perceptions of collaborative atmosphere Teachers’ professional self-perceptions Full AMEET inventory
9 9 6 10 8 8 50
0.60 0.79 0.75 0.71 0.75 0.68 0.93
to evaluate the curriculum and will be useful in improving the curriculum delivered. The comparative analyses of faculty’ perceptions of education environments within an institution and between institutions. Providing insights into the extent of alignment between expected outcomes of faculty development programs and actual benefits perceived by the faculty so as to make suggestions for future improvements. Providing information to the faculty themselves in identifying areas of EE they desire to improve upon. We also anticipate that studying teachers’ perceptions of the EE will provide useful inputs for creating a productive environment for the teachers which will, in turn, contribute to the continuous quality improvement and innovation at the institution. The strength of this study was the modified Delphi process wherein three different strategies were used in developing the inventory and a diverse group of experts from different geographical regions validated the inventory. Thus, we have endeavored to develop an instrument which can be presumed to be used in medical institutions in different countries, cultures, and contexts. However, further studies confirming
STAGE I A. Literature review and search for theoretical frameworks B. Four group discussions (n=5)
STAGE II: DELPHI TECHNIQUE (n=16) ROUND 1 A. Panel accepts / rejects, queries the inclusion of each statement using a grading scheme (0-3) B. Panel modifies the statements or sends comments or suggestions C. Panel suggests five new statements addressing missed out areas.
55 Statements
ROUND 2 A. Panel accepts / rejects, modifies /queries inclusion of five newly suggested statements using previous grading scheme. B. Panel reconsiders their response for previous 50 statements taking into consideration the Delphi members’ percentage responses C. Three statements removed from analysis as Median was < 1.
52 Statements
ROUND 3 A. Panel reconsiders their response for five newly suggested statements taking into consideration Delphi members’ responses. B. Panel indicates through agreement/disagreement/uncertainty about final inclusion of all 52 statements. C. Two statements removed from analysis as panel agreement was less than 75%. D. Domains identified through inputs from Delphi members.
50 Statements Six domains
STAGE III: PILOT STUDY A. Among small group of faculty for confirming the face validity B. Among all teaching faculty (n=62) for testing the reliability
Figure 1. 8
Preliminary 50 statements
Flow chart of methodology process.
Final AMEET
Development and validation of AMEET inventory
the consistency of reliability are justified. A limitation of our study was the in-house pilot study with a small sample size (62 faculty). Moreover, although validity evidence on the test content was obtained, validity evidence on internal structure (exploratory factor analysis) was not possible as at least five faculty per item (250 faculty) are necessary for a sound analysis (Streiner 1994) and the pilot study had a limited number of participants. Further research evaluating the psychometrics of this inventory and perceptions of faculty in different countries are warranted for validation in international settings, exploratory and confirmatory analyses, reliability including generalisability theory (Crossley et al. 2002), and subsequent widespread use.
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Conclusion The educational environment experienced by teachers involves all conditions affecting teaching and learning activities. The faculty EE is closely linked to that of the students and makes a strong impact on the students’ achievements and level of satisfaction. Our study resulted in the development and validation of the AMEET questionnaire, a valid, highly reliable and practically useful instrument to assess medical faculty members’ perceptions of the educational environment. However, the construct validity and international applicability of this inventory are areas of further research.
Notes on contributors SYED ILYAS SHEHNAZ, MBBS, MD (Pharmacology), Grad Dip HPE, is an Assistant Professor of Pharmacology at Gulf Medical University, Ajman, United Arab Emirates. She is also the coordinator of the Medical Education Unit of the university. GAMINI PREMADASA, MD (Medical Education), is a medical educationist at Gulf Medical University, Ajman, United Arab Emirates. He had served previously as Director, RTTC of WHO (SEARO) at the University of Peradeniya, Sri Lanka; Assistant Director, Medical Education Unit, Faculty of Medicine, Kuwait University and Director, CME Center, Kuwait Institute for Medical Specialization. MOHAMED ARIFULLAHAS received PhD in Pharmacology and a postdoctoral fellowship for two years at Texas Medical University, Lubbock, USA. He is currently the Dean for Admissions and Registers and Head of Department of Pharmacology in the Gulf Medical University, Ajman, United Arab Emirates. JAYADEVAN SREEDHARAN has a PhD in Biostatistics (India), Diploma in Cancer Prevention (USA) and Graduate Diploma in Health Professions Education (UAE). He is also a fellow of UICC. At present he is working as an Assistant Director, Statistical Support Facility and Professor of Biostatistics at Gulf Medical University, Ajman, UAE. KADAYAM GURUSWAMI GOMATHI has a PhD in Biochemistry and a Graduate Diploma in Health Professions Education. She is currently a Professor of Biochemistry and an Associate Dean for Admissions and Registers in the Gulf Medical University, Ajman. She is also a member of the Medical Education Unit of the university.
Acknowledgements The Delphi panel members are gratefully acknowledged. The authors also thank all the faculty who participated in the study.
Declaration of interest: The authors report no declarations of interest.
References American Education Research Association and American Psychological Association. 1999. Standards for educational and psychological testing. Washington, DC: American Education Research Association. Bellamy N, Anastassiades TP, Buchanan WW, Davis P, Lee P, McCain GA, Wells GA, Campbell J. 1991. Rheumatoid arthritis anti-rheumatic drug trials, III: Setting the Delta for clinical trials of anti-rheumatic drugs— results of a consensus development (Delphi) exercise. J Rheumatol 118(12):1908–1915. Best Evidence Medical Education Collaboration. 2003. Guide for topic review groups on carrying out BEME systematic reviews. Retrieved February 3, 2014. Available from http://www.bemecollaboration.org. Bloomfield L, Subramaniam R. 2008. Development of an instrument to measure the clinical learning environment in diagnostic radiology. J Med Imaging Radiat Oncol 52:262–268. Boor K, Van Der Vleuten C, Teunissen P, Scherpbier A, Scheele F. 2011. Development and analysis of D-RECT, an instrument measuring residents’ learning climate. Med Teach 33(10):820–827. Burden R, Fraser B. 1994. Examining teachers’ perceptions of their working environments: Introducing the School Level Environment Questionnaire. Educ Psychol Pract 10:67–71. Cassar K. 2004. Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees. Med Teach 26(3):260–264. Clayton MJ. 1997. Delphi: A technique to harness expert opinion for critical decision-making tasks in education. Educ Psychol 17(4):373–384. Crisp J, Pelletier D, Duffield C, Adams A, Nagy S. 1997. The Delphi method? Nurs Res 46:116–118. Crossley J, Davies H, Humphris G, Jolly B. 2002. Generalisability: A key to unlock professional assessment. Med Educ 36(10):972–978. de Villiers MR, de Villiers PJ, Kent AP. 2005. The Delphi technique in health sciences education research. Med Teach 27(7):639–643. Fish LS, Busby DM. 1996. The Delphi method. In: Prenkl DH, Moon SM, editors. Research methods in family therapy. New York: The Guilford Press. pp 469–482. Genn JM. 2001. AMEE medical education guide no. 23 (part 2): Curriculum, environment, climate, quality and change in medical education – A unifying perspective. Med Teach 23(5):445–454. Green B, Jones M, Hughes D, Williams A. 1999. Applying the Delphi technique in a study of GPs’ information requirements. Health Soc Care Commun 7(3):198–205. Harden RM. 2001. The learning environment and the curriculum. Med Teach 23:335–336. Hasson F, Keeney S, McKenna H. 2000. Research guidelines for Delphi survey technique. J Adv Nurs 32(4):1008–1015. Henzi D, Davis E, Jasinevicius R, Hendricson W, Cintron L, Isaacs M. 2005. Appraisal of the dental school learning environment: The students’ view. J Dent Educ 69(10):1137–1143. Holt MC, Roff S. 2004. Development and validation of the Anaesthetic Theatre Educational Environment Measure (ATEEM). Med Teach 26(6):553–558. Johnson B, Stevens JJ, Zvoch K. 2007. Teachers’ perceptions of school climate: A validity study of scores from the revised school level environment questionnaire. Educ Psychol Measur 67: 833–844. Johnson B, Stevens JJ. 2001. Exploratory and confirmatory factor analysis of the School Level Environment Questionnaire (SLEQ). Learn Environ Res 4:325–344. Jones J, Hunter D. 1995. Consensus methods for medical and health services research. Br Med J 311(7001):376–380. Keeney S, Hasson F, McKenna HP. 2005. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 38: 195–200.
9
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
S. I. Shehnaz et al.
McAleer S, Soemantri D, Roff S. 2009. Educational environment. In: Dent JA, Harden RM, editors. A practical guide for medical teachers. Edinburgh: Churchill Livingstone. pp 64–70. Miles S, Leinster SJ. 2009. Comparing staff and student perceptions of the student experience at a new medical school. Med Teach 31:539–546. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. 2009. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Open Med 3:123–130. Moos RH. 1973. Conceptualizations of human environments. Am Psychol 28:652–655. Moos RH. 1974. The social climate scales: An overview. Palo Alto, CA: Consulting Psychologists Press. Polit DF, Beck CT, Hungler BP. 2001. Essentials of nursing research: Methods, appraisal, and utilisation. Philadelphia: Lippincott. Rentoul AJ, Fraser BJ. 1983. Development of a school-level environment questionnaire. J Educ Admin 21(1):21–39. Roff S, McAleer S, Harden RM, Al-Qahtani M, Uddin AA, Deza H, Groenen G, Primparyon P. 1997. Development and validation of the Dundee Ready Education Environment Measure (DREEM). Med Teach 19(4):295–299. Roff S, McAleer S, Skinner A. 2005. Development and validation of an instrument to measure the postgraduate clinical learning and teaching
10
educational environment for hospital-based junior doctors in the UK. Med Teach 27(4):326–331. Saarikoski M, Leino-Kilpi H. 1999. Association between quality of ward nursing care and students’ assessment of the ward as a learning environment. J Res Nurs 4(6):467–474. Scho¨nrock-Adema J, Bouwkamp-Timmer T, van Hell EA, Cohen-Schotanus J. 2012. Key elements in assessing the educational environment: Where is the theory? Adv Health Sci Educ Theory Pract 17(5):727–742. Shehnaz SI, Sreedharan J, Gomathi KG. 2012. Faculty and students’ perceptions of student experiences in a medical school undergoing curricular transition in the United Arab Emirates. Sultan Qaboos Univ Med J 12(1):628–636. Soemantri D, Herrera C, Riquelme A. 2010. Measuring the educational environment in health professions studies: A systematic review. Med Teach 32:947–952. Streiner DL. 1994. Figuring out factors: The use and misuse of factor analysis. Can J Psychiat 39(3):135–140. Thangaratinam S, Redman CWE. 2005. The Delphi technique. Obstet Gynaecol 7:120–125. Wallengren J. 2011. Identification of core competencies for primary care of allergy patients using a modified Delphi technique. BMC Med Educ 11:12.
Development and validation of the AMEET inventory: An instrument measuring medical faculty members’ perceptions of their educational environment SYED ILYAS SHEHNAZ, GAMINI PREMADASA, MOHAMED ARIFULLA, JAYADEVAN SREEDHARAN & KADAYAM GURUSWAMI GOMATHI Gulf Medical University, United Arab Emirates
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Abstract We developed, through a three-round modified Delphi technique, an inventory [Assessment of Medical Education Environment by Teachers (AMEET)] to assess viewpoints of medical faculty on educational environment experienced by teachers. In Round 1, 16 members from international settings graded (0–3) and modified an initial list of statements, or suggested new statements for inclusion. In Round 2, the new statements were graded and previous responses reconsidered. Subsequently, statements with median scores 1 were excluded. The questionnaire was compiled following reconsideration of responses to new statements in Round 3. A pilot test enabled assessment of the validity and reliability of the instrument. The panel members modified and graded the initial 50 statements of the preliminary inventory and suggested five new statements. The new statements were graded, responses to previous 50 statements were reconsidered, and three statements with median scores 51 were removed following Round 2. On review of the 52 statements by panel, two statements (each with agreement of575%) were deleted. The final 50-item inventory comprising six domains was piloted with 62 faculty. Cronbach’s alpha of 0.94 was obtained. The AMEET inventory is valid, highly reliable and practically useful to assess medical faculty members’ perceptions of educational environment.
Introduction
Practice points
Understanding educational environment (EE) is important for effective learning in health professions and has been emphasized by Genn (2001) and Harden (2001). A conducive EE will result in positive student-learning outcomes and satisfaction. The environment of any medical school influences the behavior of both students and the faculty. The EE experienced by teachers involves all the conditions affecting teaching and learning activities. Faculty members’ EE is influenced by the organizational structure of the institution and may include aspects such as their ‘‘involvement in the process of curriculum development and delivery; value and respect rendered to the staff and their contribution to curriculum development; enhancement of staff well being; concrete measures to reduce staff stress’’, etc. (Genn 2001). Faculty perceptions of the EE are important not only for teachers but also for students. The perceptions of the participants/inhabitants regarding the environment influence their final behavior and hence, studying their perceptions can provide valuable input for improving the quality of the EE in any institution (Genn 2001). A great deal of attention has been given to the health professions students’ perspective about the EE and instruments assessing the students’ perceptions have been developed and
Educational environment (EE) experienced by teachers involves all conditions which affect teaching and learning activities. The faculty EE is closely linked to the students’ EE and has a strong impact on students’ achievements and levels of satisfaction. The AMEET questionnaire is a valid, highly reliable, and practically useful instrument to assess medical faculty members’ perceptions of educational environment in their institution.
validated. A systematic review identified 31 instruments for measuring the students’ perceptions of EE in health professions (Soemantri et al. 2010). However, to the best of our knowledge, only two studies have looked into perspectives of the other stakeholders such as the teaching staff (Miles & Leinster 2009; Shehnaz et al. 2012). Moreover, in these latter studies, the faculty stated their opinion about the students’ EE (as experienced by the students) and not of the faculty’s own educational environment. In view of the relevance of the faculty’s role in the EE of any institution and in the absence of a
Correspondence: Dr. Syed Ilyas Shehnaz, Department of Pharmacology, Gulf Medical University, P.O. Box 4184, Ajman, United Arab Emirates. Tel: 97 16 7431333x316; Fax: +97 16 7431222; E-mail: [email protected] ISSN 0142-159X print/ISSN 1466-187X online/14/000001–10 ß 2014 Informa UK Ltd. DOI: 10.3109/0142159X.2014.947935
1
S. I. Shehnaz et al.
specific and appropriate instrument to evaluate their viewpoints, we aimed to develop and validate an inventory to measure the medical faculty members’ perceptions of the EE at their institutions. The Assessment of Medical Education Environment by Teachers (AMEET) is the product of our endeavor.
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Methods A three-stage methodology was used in developing the instrument: (1) literature review and group discussions, (2) modified Delphi technique to identify consensus, and (3) pilot study. In the first stage, we identified the theoretical framework(s) underpinning the pertinent concepts that should be measured in establishing the EE of the teaching faculty and subsequently the characteristics to be included in the AMEET inventory. We anticipated that a sound theoretical framework would circumvent shortcomings in our questionnaire. For this, in September 2011, we followed the Cochrane Library and the Best Evidence Medical Education (BEME) guidelines (Best Evidence Medical Education Collaboration 2003; Moher et al. 2009) and MEDLINE/PUBMED, Academic Search Premier, CINAHL, EMBASE, ERIC, Ingentaconnect, Topics in Medical Education (TIMELIT), Google Scholar, PsycARTICLES, and PsycINFO were searched for publications related to EE of the faculty. Primary search terms curriculum; educational environment/ climate; learning environment; medical faculty; perception; undergraduate medical education; survey, test, scale, measure, instrument, inventory or questionnaire were used in various combinations. In Medline (MESH database), the search terms education, medical, and undergraduate were used in MAJR (main subject heading) combined (AND) with medical faculty and perceptions. All search terms were ‘‘exploded’’. Articles with both abstracts and free full texts published in English and English abstracts of articles published in other languages involving the EE in health professions education (undergraduate or postgraduate) were scrutinized. A manual search of leading medical education journals: Medical Education, Academic Medicine, Advances in Health Sciences Education, and Medical Teacher was also performed. Hand search of reference lists, potentially relevant citations of articles identified through the primary search and review articles dealing with EE in health professions education, was also conducted. As correlates of staff well-being are presumed to be part of the faculty EE, we also searched articles related to the faculty development efforts. Both qualitative and quantitative studies were included. Non-English language articles and any publications before January 1990 were excluded from the study. The preliminary inventory of statements, supported by comprehensive theoretical frameworks and developed from the literature review, were then reviewed and modified on the basis of relevance to teachers’ EE by an expert group (research team) of one medical education expert (GP), one faculty medical doctor (SS), and three professors in the medical faculty (M. A., J. S., and K. G. G.) through four rounds of discussions. 2
In the second stage, a three-round modified Delphi approach was used to derive agreement on generation and classification of the statements of the preliminary inventory. The Delphi procedure achieved consensus among a panel of experts through a sequence of structured rounds involving opinion generation (agreement/disagreement) on statements or concepts. Our Delphi panel was derived by a two-stage process. The members identified could be categorized into at least one of the following: medical education experts with minimum two years’ experience; basic sciences faculty, clinical faculty, and clinical psychologists with at least five years’ experience. Subsequently, we included a ‘‘nomination’’ process in which the identified experts were asked to recommend other eligible experts (Fish & Busby 1996). We invited members with the eligibility criteria listed above to become part of the panel, included other proposed eligible experts and short listed all those who were willing to take part. All participants were assured that anonymity would be maintained and an e-mail consent was obtained. In Round 1, the panel members were asked to accept, reject, or question the inclusion of each statement of the preliminary inventory. They were invited to modify the statements, add new statements, or submit other comments, suggestions or remarks explaining their ratings. The panel was asked, using a Likert-type scale where 0 ¼ not necessary, 1 ¼ desirable, 2 ¼ important and 3 ¼ absolutely essential, to rate the inclusion of the statements. The survey was administered online with the panel members’ opinions collected independently and anonymously. Only statements with median scores 1 point or more were incorporated into the next-stage questionnaire, as a score below 1 indicated ‘‘not necessary’’. Free text comments were noted, and suggestions regarding format and clarification incorporated in the subsequent round. When several suggestions were given for a single item, the most commonly suggested modification was used. The results of the completed questionnaires were assessed by the expert group and a second inventory was developed for the Round 2 of the modified Delphi process. The Delphi panel accepted/rejected or modified/queried the inclusion of newly suggested statements by using the previous grading scheme and also reconsidered their response for the previous statements taking into consideration the Delphi group members’ percentage responses for each one of the statements. Following this, the statements with median scores 1 were removed from analysis. In the final round (Round 3), the responses to the newly suggested statements were reconsidered and the final inclusion of each statement was indicated through agreement/ disagreement/uncertainty. Panel agreement 4 75% was deemed necessary for inclusion of the statement into the final AMEET inventory. Subscales of the AMEET questionnaire were subsequently identified through group discussions among the expert group and confirmed using inputs from the Delphi panel (obtaining validity evidence based on test content) (American Education Research Association and American Psychological Association 1999). The Delphi panel was provided with a preliminary list of domains, the statements constituting it and requested to
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Development and validation of AMEET inventory
indicate agreement to the domain allocation or to reallocate the statement to their preferred domain. If at least 11 participants assigned a statement to the same domain, we considered the implication of the statement was clear and allocated the statement to that domain. The third stage (pilot study) was intended to refine the AMEET inventory involving the statements approved by the Delphi panel and evaluate its reliability. The inventory was piloted with a small group of faculty for face validity, and ambiguities in the statements were cleared. Finally, the selfadministered AMEET questionnaire was anonymously answered by all the teaching faculty of the institution. Each statement was scored on a Likert-type scale: 0 ¼ strongly disagree, 1 ¼ disagree, 2 ¼ uncertain, 3 ¼ agree and 4 ¼ strongly agree. The percentage responses, median and range for statements of Rounds 1 and 2 were calculated. The coefficient of range was also calculated to compare the reduction in the variation of scores from Round 1 to Round 2 (for preliminary statements) and from Round 2 to Round 3 (for newly suggested statements). The internal consistency of the inventory and its subscales were checked using Cronbach’s alpha. All the statistical calculations were performed using SPSS-20 (SPSS Inc., Chicago, IL). The study was approved by the institutional Ethics Committee and all participating faculty gave a written informed consent.
Results Stage I: Literature review and group discussions The literature search identified more than 100 articles related to EE and specifically seven recently developed inventories related to medical students’ EE. The Dundee Ready Education Environment Measure (DREEM) (Roff et al. 1997), the Postgraduate Hospital Educational Environment Measure (PHEEM) (Roff et al. 2005), the Dutch Residency Educational Climate Test (D-RECT) (Boor et al. 2011), DR-CLE questionnaire – diagnostic radiology clinical learning environment questionnaire (Bloomfield & Subramaniam 2008), STEEM – surgical theatre educational environment measure (Cassar 2004), ATEEM – the Anaesthetic Theatre Educational Environment Measure (Holt & Roff 2004) and the Clinical Learning Environment (CLE) (Saarikoski & Leino-Kilpi 1999), the Supervision and Dental Student Learning Environment Survey (DSLES) (Henzi et al. 2005) are some of the reported valid and reliable instruments for undergraduate medical, postgraduate medical, nursing, and dental students, respectively. Three review articles also gave insights into the various instruments for measuring the EE among different health professions student groups (McAleer et al. 2009; Soemantri et al. 2010; Scho¨nrock-Adema et al. 2012) and identified teaching, supervision, training, atmosphere, emotional climate, social support, learning opportunities, orientation to learning, workload, goal, direction, and emphasis on scholarship as common themes in the instruments. Scho¨nrock-Adema et al. (2012) gave insights into three domains (based on the framework formulated by Moos 1973,
1974) related to goal orientation (the content and aims of education, which involves concepts such as clarity about learning objectives, relevant learning content, and constructive criticism); relationships (an open and friendly atmosphere and affiliation, certain characteristics of which include student involvement, support freedom of expression, and involvement in educational settings); and organization/regulation (organization, role clarity, teacher control, innovation, work pressure, and physical comfort) as theoretical frameworks for almost all student EE inventories. Only two previous studies identified what medical faculty perceived about the students’ experiences of their EE (Miles & Leinster 2009; Shehnaz et al. 2012). In contrast, much research had been done in assessing school teachers’ climate (Rentoul & Fraser 1983; Burden & Fraser 1994; Johnson & Stevens 2001; Johnson et al. 2007), which was based on Moos’s three dimensions (Moos 1973, 1974). As almost all the EE inventories for health professions students and school teachers had the same three domains (goal orientation; relationships, and organization/regulation) as the key elements, we postulated that this framework was also applicable to the EE of medical faculty and we have used it as theoretical underpinning for the development of our preliminary inventory. The preliminary inventory of statements, based on the three domains of theoretical frameworks, following extensive literature search and the four expert group discussions review, consisted of 50 statements.
Stage II: Modified Delphi process In September 2012, we invited 10 medical educationalists, 20 specialists (clinical and non-clinical specialty tutors), and 10 psychologists to be members in our Delphi panel. The 40 panelists were chosen considering their involvement in undergraduate education, possession of a certificate/diploma level of medical educational training (for clinical/non-clinical faculty and psychologists), and worldwide geographical representation (USA, UK, Australia, India, and Sri Lanka). No monetary incentives were offered for involvement in the complete Delphi procedure and voluntary participation was emphasized. A brief introduction to the study, research objectives and details of all three rounds of the modified Delphi were given at the outset, and consent solicited assuring confidentiality for all participants. Twenty-five faculty members consented to participate in the Delphi panel (nine medical educationalists, seven clinical faculty, six non-clinical faculty, and three psychologists). After one month, the preliminary inventory was sent to the consenting panel members, who modified and graded each statement for its relevance with respect to faculty EE. Completed inventories were returned by 20 of the 25 participants in Round 1 to SS who summarized the remarks. Five of the participants (one medical educationalist, two clinicians, one non-clinical faculty and one psychologist) did not reply, despite telephone/e-mail reminders and the offer of another month for completion of the inventory. Based on the experts’ suggestions, five new statements were added to Round 2 inventory and certain statements were simplified to improve clarity.
3
S. I. Shehnaz et al.
The second inventory with 55 statements was sent to the 20 experts who had answered in Round 1. The five new statements added were modified, graded and the responses to the previous 50 statements were reconsidered in Round 2. Four panel members did not complete Round 2 in spite of repeated reminders. Table 1 displays the outcomes of Rounds 1 and 2. Following this round, three statements (Nos. 4, 20, and 26) were removed as their median scores were less than 1. In the last round (Round 3), the responses for the five new statements were reconsidered (Table 2) and agreement/disagreement/uncertainty solicited from the 16 panel members for the final inclusion of the 52 statements. There were no significant differences in the median panel responses between Rounds 1 and 2 for the 50 statements and
between Rounds 2 and 3 for the five new statements. The coefficient of range showed that the deviation was reduced for 15 preliminary statements (from Round 1 to Round 2) and three newly suggested statements (from Round 2 to Round 3). No change was observed for the rest of the 35 preliminary statements and two newly suggested statements (Tables 1 and 2). The minimum reduction observed for the 18 statements was 30% and the maximum reduction observed was 80%. Analysis after Round 3 resulted in deletion of two more statements as panel agreement was 575% (Table 3). Consensus was reached in this round: 39 statements garnered more than 90% agreement. The final AMEET inventory comprised 50 statements. Since non-participation influences changes in the median, the final analysis was restricted to only those 16 panel members who participated in all three rounds.
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Table 1. Percentage ratings and median (range) for Rounds 1 and 2 (n ¼ 16).
Round 1
Round 2
Percentage responses
S. no. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
23 24 25 26 27 28 29
Percentage responses
Statements
NN 0
D1
I2
AE 3
Median (range)
NN 0
D1
I2
AE 3
Median (range)
The teacher finds his/her role as a teacher interesting There is too much of emphasis on learning factual information The teachers are domineering and authoritative The teachers make the students look foolish The comments given by the teachers help students to improve themselves The preparation that the teachers undertake for doing their classes is adequate The emphasis in classes is on what the teacher does rather than on what the student learns The content and the teaching methods match the learning objectives of the course The teacher respects the students’ viewpoints The learning is ‘‘student centered’’ Students have sufficient opportunities to develop competence Students have sufficient opportunities to develop confidence The students have a clear idea of what they are expected to gain from the course The class environment makes the students want to get actively involved The students are encouraged to be life-long learners Students are encouraged to understand and apply what they learn rather than remember isolated facts The students are well mannered and respectful The students complete their assignments on time The students are motivated to learn The students’ attitude and activities annoy the teachers The students come well prepared for their learning activities The students are not sufficiently prepared for the formative assessments (the assessments conducted for giving feedback/practice) The students feel comfortable in learning with other students coming from varied background(s) The atmosphere in the hospital setting is conducive to learning The time allotted to different learning sessions is appropriate Cheating at examinations is widespread at the institution The atmosphere during most of the teaching/learning activities is conducive to learning The students have sufficient opportunities to improve their communication skills The students find the learning experience disappointing
18.8 6.3 31.3 56.3 –
18.8 43.8 18.8 12.5 6.3
18.8 31.3 37.5 18.8 43.8
43.8 18.8 12.5 12.5 50.0
2.0(3) 1.5(3) 1.5(3) 0.0(3) 2.5(2)
6.3 – 18.8 62.5 –
18.8 50.0 25.0 18.8 –
25.0 31.3 50.0 12.5 25.0
50.0 18.8 6.3 6.3 75.0
2.5(3) 1.5(2)a 2.0(3) 0.0(3) 3.0(1)a
–
18.8
31.3
50.0
2.5(2)
–
6.3
37.5
56.3
3.0(2)
6.3
18.8
25.0
50.0
2.5(3)
–
12.5
43.8
43.8
2.0(2)a
6.3
6.3
25.0
62.5
3.0(3)
–
–
31.3
68.8
3.0(1)a
– – – – –
12.5 6.3 12.5 31.3 6.3
31.3 31.3 31.3 18.8 31.3
56.3 62.5 56.3 50.0 62.5
3.0(2) 3.0(2) 3.0(2) 2.5(2) 3.0(2)
– – – – –
6.3 6.3 – 25.0 –
43.8 25.0 31.3 31.3 25
50.0 68.8 68.8 43.8 75
2.5(2) 3.0(2) 3.0(1)a 2.0(2) 3.0(1)a
–
12.5
18.8
68.8
3.0(2)
–
6.3
25.0
68.8
3.0(2)
– –
6.3 –
25.0 25.0
68.8 75.0
3.0(2) 3.0(1)
– –
6.3 –
18.8 18.8
75.0 81.3
3.0(2) 3.0(1)
18.8 6.3 – 50 – 18.8
25.0 25.0 6.3 18.8 31.3 25.0
37.5 62.5 37.5 18.8 43.8 37.5
18.8 6.3 56.3 12.5 25.0 18.8
2.0(3) 2.0(3) 3.0(2) 0.5(3) 2.0(2) 2.0(3)
18.8 – – 56.3 – 18.8
18.8 25.0 6.3 12.5 25.0 18.8
43.8 68.8 31.3 18.8 50.0 50.0
18.8 6.3 62.5 12.5 25.0 12.5
2.0(3) 2.0(2)a 3.0(2) 0.0(3) 2.0(2) 2.0(3)
–
12.5
50.0
37.5
2.0(2)
–
6.3
56.3
37.5
2.0(2)
– – 50 –
25.0 25.0 18.8 12.5
25.0 37.5 18.8 56.3
50.0 37.5 12.5 31.3
2.5(2) 2.0(2) 0.5(3) 2.0(2)
– – 56.3 –
6.3 6.3 12.5 12.5
37.5 56.3 18.8 56.3
56.3 37.5 12.5 31.3
3.0(2) 2.0(2) 0.0(3) 2.0(2)
–
18.8
31.3
50.0
2.5(2)
–
–
43.8
56.3
3.0(1)a
25.0
43.8
18.8
12.5
1.0 (3)
25.0
43.8
18.8
12.5
1.0(3)
(continued )
4
Development and validation of AMEET inventory
Table 1. Continued.
Round 1
Round 2
Percentage responses
Statements
NN 0
D1
I2
AE 3
Median (range)
NN 0
D1
I2
AE 3
Median (range)
The teaching learning experience is very stressful for the students The students find studying medicine enjoyable The students’ feel comfortable about asking any questions they want The students’ feedback about the curriculum is taken into consideration Teachers are encouraged to provide suggestions regarding curriculum Teachers are involved in collaborative efforts related to educational activities Teachers are encouraged to teach a trans-disciplinary approach to health problems Teachers have good interpersonal communication skills for dealing with peers and students Teachers are flexible and open to change/suggestions The attitudes of the teachers are conducive to effective team work Teachers dominate over subordinates, peers, or students I am satisfied with the opportunities that I get for working with other faculty There is a formal support system for faculty who get stressed Teachers have sufficient time to plan their teaching activities I am knowledgeable in educational concepts for my role as a teacher I possess the necessary teaching skills for undertaking my duties I have effective communication skills for dealing with hospital patients Opportunities are available for trying out alternative approaches in teaching I have opportunities for learning research methodology and research communication skills I have opportunities for developing the skills of educational administration The Faculty Development Programs are adequate in preparing me for the tasks expected of me
18.8
25.0
43.8
12.5
2.0(3)
31.3
18.8
43.8
6.3
1.5(3)
6.3 6.3
37.5 6.3
18.8 43.8
37.5 43.8
2.0(3) 2.0(3)
6.3 6.3
43.8 6.3
18.8 37.5
31.3 50.0
1.5(3) 2.5(3)
–
12.5
43.8
43.8
2.0(2)
–
6.3
50.
43.8
2.0(2)
–
6.3
31.3
62.5
3.0(2)
–
–
31.3
68.8
3.0(1)a
–
6.3
37.5
56.3
3.0(2)
–
–
43.8
56.3
3.0(1)a
–
12.5
31.3
56.3
3.0(2)
–
6.3
37.5
56.3
3(2)
–
12.5
31.3
56.3
3.0(2)
–
6.3
25.0
68.8
3.0(2)
– –
6.3 –
50.0 43.8
43.8 56.3
2.0(2) 3.0(1)
– –
– –
56.3 31.3
43.8 68.8
2.0(1)a 3.0(1)
18.8 –
25.0 18.8
37.5 31.3
18.8 50.0
2.0(3) 2.5(2)
25.0 –
31.3 18.8
31.3 37.5
12.5 43.8
1.0(3) 2.0(2)
– – –
25.0 18.8 12.5
43.8 50.0 43.8
31.3 31.3 43.8
2.0(2) 2.0(2) 2.0(2)
– – –
18.8 18.8 6.3
56.3 50.0 50.0
25.0 31.3 43.8
2.0(2) 2(2) 2(2)
– 6.3
6.3 12.5
43.8 43.8
50.0 37.5
2.5(3) 2.0(3)
– –
6.3 6.3
31.3 50.
62.5 43.8
3(3) 2.0(2)a
6.3
25.0
31.3
37.5
2.0(3)
–
6.3
50.0
43.8
2.0(2)a
6.3
31.3
12.5
50.0
2.5(3)
–
31.3
18.8
50.0
2.5(2)a
6.3
25.0
25.0
43.8
2.0(3)
–
31.3
18.8
50.0
2.5(2)a
–
–
62.5
37.5
2.0(1)
–
–
56.3
43.8
2.0(1)
S. no. 30 31 32 33 34 35
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
Percentage responses
NN, not necessary; D, desirable; I, important; AE, absolutely essential. Statements in bold: median score 5 1. a Statements with reduced coefficients of range.
The expert group initially suggested six subscales and some statements were reallocated to different domains on the advice of the Delphi panel. The statements were allocated to six domains according to the content of the statements: Teachers’ Perceptions of Teaching (nine statements); Teachers’ Perceptions of Learning Activities (nine statements); Teachers’ Perceptions of Students (six statements); Teachers’ Perceptions of Learning Atmosphere (10 statements); Teachers’ Perceptions of Collaborative Atmosphere (eight statements); Teachers’ Professional Self-Perceptions (eight statements).
Stage III: Pilot study The 50-item AMEET inventory was piloted with a representative small group of faculty (n ¼ 10) to confirm the face validity. As a result, three statements were re-written for better clarity. The inventory was then self-administered to the teaching faculty of the institution (n ¼ 62; response rate ¼ 89%). The
resulting internal consistency coefficient Alpha computed from the data is shown in Table 4. Figure 1 summarizes the methodological process and findings of the three stages of the development of the AMEET inventory.
Discussion The development of the AMEET inventory, a questionnaire to evaluate the teacher’s educational climate, was directed through a diverse methodology involving theoretical frameworks based on earlier findings (Scho¨nrock-Adema et al. 2012), a modified Delphi procedure and a pilot test. The key features of the Delphi are the multiple consultation rounds, structured flow of information, analysis of the collective panel responses in each round which is then updated to the Delphi members to facilitate the development of the questionnaire of the next round, the opportunity for experts to give qualitative inputs and reconsider their own ratings in the
5
S. I. Shehnaz et al.
Table 2. Percentage ratings and median (range) for five new statements introduced in Round 2 (n ¼ 16).
Round 2
Round 3
Percentage responses
S. no. 51 52 53
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
54 55
Statements The teachers adopt a variety of teaching methods to cater to diverse styles of learning Adequate teaching and learning resources (including physical and material) are available The students have sufficient opportunities to discuss their learning problems with the faculty My peers appreciate when I make professional progress I am encouraged to go to conferences that improve my knowledge and skills
NN 0
D1
I2
AE 3
6.3
31.3
31.3
31.3
–
18.8
43.8
–
12.5
12.5 6.3
43.8 37.5
Percentage responses Median (range)
D1
2.0(3)
–
18.8
50
31.3
2.0(2)a
37.5
2.0(2)
–
12.5
56.3
31.3
2.0(2)
56.3
31.3
2.0(2)
–
6.3
68.8
25
2.0(2)
43.8 43.8
– 12.5
1.0(2) 2.0(3)
6.3 –
50 50
–
2.0(1)a 2.0(2)a
43.8 43.8
I2
AE 3
Median (range)
NN 0
6.3
NN, not necessary; D, desirable; I, important; AE, absolutely essential. a Statements with reduced coefficients of range.
light of the regular summarized feedback ratings of the complete panel, the anonymity of the panel providing unbiased opinions in a non-threatening environment, and participation of experts across geographical barriers precluding their physical presence (Jones & Hunter 1995; Clayton 1997; Hasson et al. 2000). Many modifications to the classical Delphi technique have been proposed and one such modification involves the generation of initial statements from literature review rather than proposed by the expert panel (de Villiers et al. 2005). In our study too, the initial statements were not generated by the expert panel but from previous research and the concept of explicating the most important issues by deleting the less important ones through several rounds was maintained (Keeney et al. 2005; Thangaratinam & Redman 2005; Wallengren 2011). The selection of the Delphi expert panel is paramount for the validity of the process. To select our Delphi panel experts, we used a purposive sampling technique which utilizes the researcher’s expert knowledge about the general population to identify the cases to be included in the sample (Polit et al. 2001). Our final Delphi panel had 16 members who were medical educationalists, clinical, and non-clinical faculty or psychologists. It has been suggested that adequate panel size ranged from 15 to 30 members since groups larger than 30 is unmanageable and rarely improves the outcome, and smaller samples produce invalid numbers in the final rounds due to dropouts (Clayton 1997). We selected the Delphi technique to content validate the inventory through the opinions of a heterogenic group of experts in spite of geographical barriers. The iterative nature of the Delphi process also augmented the concurrent validity and the good response rates in the three rounds have not affected the validity adversely (Hasson et al. 2000). For analysis of the responses in Rounds 1 and 2, the percentage responses along with the medians and ranges were used (rather than mean and standard deviation) because a Likert-type scale (an ordinal scale) was used to grade the statements. The collected opinion had been subsequently 6
communicated back to the panel at the end of Rounds 1 and 2 (Hasson et al. 2000). This ensured that the panel members were aware of their own responses in relation to those of the group and also verified the level of agreement among respondents (Jones & Hunter 1995). The reduction in coefficient of range for 18 statements (30–80%) indicated that the agreement between the participants increased as the rounds progressed. Moreover, the agreement between the participants was not reduced in the rest of the 35 preliminary statements and the two newly suggested statements as no increase in the coefficient of range was detected. In order to maintain a manageable list of statements, some studies have also restricted statements from progressing to the next round if they did not meet a predetermined level of consensus (Wallengren 2011). In our study, the level of consensus was set at a median score of 1. The results of the final round indicated that the stable consensus for almost all statements had been attained as around 80% of the statements were unanimously agreed on by the panel (90–100% agreement). The achievement of consensus is the quintessence of the Delphi technique and offers scope for maximum contention for this research technique. Hence, it is imperative that consensus be predetermined and non-consensus clearly interpreted. The convergence of opinions with minimal divergence typically identifies that consensus is attained and this is characteristically reached after three rounds with maximum variations occurring between Rounds 1 and 2 (Bellamy et al. 1991). However, there is no unanimously agreed level, with different studies reporting consensus attained anywhere between 51% and 80% agreement among Delphi panel members (Green et al. 1999; de Villiers et al. 2005) or indicated by stability of the response through the iterative process (Crisp et al. 1997). We had pre-specified a panel agreement of more than 75% as attainment of consensus. It is generally agreed that the minimum acceptable value of Cronbach’s alpha is 0.70. The high alpha coefficients of the AMEET inventory and most of its domains in our pilot study indicated that this instrument has adequate reliability for
Development and validation of AMEET inventory
Table 3. Percentage agreement/disagreement and finalized domains in Round 3.
Statements 1 2 3 4 5 6 7
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
8 9 10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
Teachers’ perceptions of teaching I find my role as a teacher interesting There is too much of emphasis on learning factual information The teachers are domineering and authoritative The teachers make the students look foolish The comments given by the teachers help students to improve themselves The preparation that the teachers undertake for doing their classes is adequate The emphasis in classes is on what the teacher does rather than on what the student learns The content and the teaching methods match the learning objectives of the course The teacher respects the students’ viewpoints Teachers’ perceptions of learning activities The learning is ‘‘student centered’’ Students have sufficient opportunities to develop competence Students have sufficient opportunities to develop confidence The students have a clear idea of what they are expected to gain from the course The class environment makes the students want to get actively involved The students are encouraged to be life-long learners Students are encouraged to understand and apply what they learn rather than remember isolated facts Teachers’ perceptions of students The students are well mannered and respectful The students complete their assignments on time The students are motivated to learn The students’ attitude and activities annoy the teachers The students come well prepared for their learning activities The students are not sufficiently prepared for the formative assessments (the assessments conducted for giving feedback/practice) The students feel comfortable in learning with other students coming from varied background(s) Teachers’ perceptions of learning atmosphere The atmosphere in the hospital setting is conducive to learning The time allotted to different learning sessions is appropriate Cheating at examinations is widespread at the institution The atmosphere during most of the teaching/learning activities is conducive to learning The students have sufficient opportunities to improve their communication skills The students find the learning experience disappointing The teaching learning experience is very stressful for the students The students find studying medicine enjoyable The students’ feel comfortable about asking any questions they want The students’ feedback about the curriculum is taken into consideration Teachers are encouraged to provide suggestions regarding curriculum Teachers’ perceptions of collaborative atmosphere Teachers are involved in collaborative efforts related to educational activities Teachers are encouraged to teach a trans-disciplinary approach to health problems Teachers have good interpersonal communication skills for dealing with peers and students Teachers are flexible and open to change/suggestions The attitudes of the teachers are conducive to effective team work Teachers dominate over subordinates, peers or students I am satisfied with the opportunities that I get for working with other faculty There is a formal support system for faculty who get stressed Teachers have sufficient time to plan their teaching activities Teachers’ professional self-perceptions I am knowledgeable in educational concepts for my role as a teacher I possess the necessary teaching skills for undertaking my duties I have effective communication skills for dealing with hospital patients Opportunities are available for trying out alternative approaches in teaching I have opportunities for learning research methodology and research communication skills I have opportunities for developing the skills of educational administration The Faculty Development Programs are adequate in preparing me for the tasks expected of me The teachers adopt a variety of teaching methods to cater to diverse styles of learning Adequate teaching and learning resources (including physical and material) are available The students have sufficient opportunities to discuss their learning problems with the faculty My peers appreciate when I make professional progress I am encouraged to go to conferences that improve my knowledge and skills
Percentage agreement
Percentage disagreement
Percentage uncertainty
93.8 100 68.8 Deleted 100 100 87.5
– – 6.3
6.3 – 25
– – 6.3
– – 6.3
100 100
– –
– –
100 100 100 100 100 100 100
– – – – – – –
– – – – – – –
12.5 – –
6.3 – –
– 6.3
6.3 12.5
–
–
81.3 100 100 Deleted 93.8 81.3 100
93.8 93.8 Deleted 100 100 81.3 81.3 87.5 100 93.8 100
– –
6.3 6.3
– – 12.5 18.8 6.3 – – –
– – 6.3 – 6.3 – 6.3 –
100 93.8 100
– – –
– 6.3 –
100 100 75 87.5 93.8 100
– – 18.8 6.3 – –
– – 6.3 6.3 6.3 –
100 93.8 93.8 100 100
– – – – –
– 6.3 6.3 – –
93.8 100
6.3 –
– –
100 100 100
– – –
– – –
25.0 6.3
6.3 12.5
68.8 81.3
Statements in bold: agreement5 75%.
7
S. I. Shehnaz et al.
measurement at our institution, although the coefficient for perceptions of teaching was slightly low (Table 4). The AMEET inventory reveals the medical teachers’ perceptions of EE and also sheds light on other dependent factors such as professional self-development, inter-personal relationships, and student achievement. This instrument may, therefore, be useful in the following: The diagnosis of the strengths and weaknesses of curricula in a health professions setting. This may help
Table 4. Cronbach’s alpha for domains and full meet inventory.
Number Cronbach’s of items alpha
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Domains Teachers’ perceptions of teaching Teachers’ perceptions of learning activities Teachers’ perceptions of students Teachers’ perceptions of learning atmosphere Teachers’ perceptions of collaborative atmosphere Teachers’ professional self-perceptions Full AMEET inventory
9 9 6 10 8 8 50
0.60 0.79 0.75 0.71 0.75 0.68 0.93
to evaluate the curriculum and will be useful in improving the curriculum delivered. The comparative analyses of faculty’ perceptions of education environments within an institution and between institutions. Providing insights into the extent of alignment between expected outcomes of faculty development programs and actual benefits perceived by the faculty so as to make suggestions for future improvements. Providing information to the faculty themselves in identifying areas of EE they desire to improve upon. We also anticipate that studying teachers’ perceptions of the EE will provide useful inputs for creating a productive environment for the teachers which will, in turn, contribute to the continuous quality improvement and innovation at the institution. The strength of this study was the modified Delphi process wherein three different strategies were used in developing the inventory and a diverse group of experts from different geographical regions validated the inventory. Thus, we have endeavored to develop an instrument which can be presumed to be used in medical institutions in different countries, cultures, and contexts. However, further studies confirming
STAGE I A. Literature review and search for theoretical frameworks B. Four group discussions (n=5)
STAGE II: DELPHI TECHNIQUE (n=16) ROUND 1 A. Panel accepts / rejects, queries the inclusion of each statement using a grading scheme (0-3) B. Panel modifies the statements or sends comments or suggestions C. Panel suggests five new statements addressing missed out areas.
55 Statements
ROUND 2 A. Panel accepts / rejects, modifies /queries inclusion of five newly suggested statements using previous grading scheme. B. Panel reconsiders their response for previous 50 statements taking into consideration the Delphi members’ percentage responses C. Three statements removed from analysis as Median was < 1.
52 Statements
ROUND 3 A. Panel reconsiders their response for five newly suggested statements taking into consideration Delphi members’ responses. B. Panel indicates through agreement/disagreement/uncertainty about final inclusion of all 52 statements. C. Two statements removed from analysis as panel agreement was less than 75%. D. Domains identified through inputs from Delphi members.
50 Statements Six domains
STAGE III: PILOT STUDY A. Among small group of faculty for confirming the face validity B. Among all teaching faculty (n=62) for testing the reliability
Figure 1. 8
Preliminary 50 statements
Flow chart of methodology process.
Final AMEET
Development and validation of AMEET inventory
the consistency of reliability are justified. A limitation of our study was the in-house pilot study with a small sample size (62 faculty). Moreover, although validity evidence on the test content was obtained, validity evidence on internal structure (exploratory factor analysis) was not possible as at least five faculty per item (250 faculty) are necessary for a sound analysis (Streiner 1994) and the pilot study had a limited number of participants. Further research evaluating the psychometrics of this inventory and perceptions of faculty in different countries are warranted for validation in international settings, exploratory and confirmatory analyses, reliability including generalisability theory (Crossley et al. 2002), and subsequent widespread use.
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
Conclusion The educational environment experienced by teachers involves all conditions affecting teaching and learning activities. The faculty EE is closely linked to that of the students and makes a strong impact on the students’ achievements and level of satisfaction. Our study resulted in the development and validation of the AMEET questionnaire, a valid, highly reliable and practically useful instrument to assess medical faculty members’ perceptions of the educational environment. However, the construct validity and international applicability of this inventory are areas of further research.
Notes on contributors SYED ILYAS SHEHNAZ, MBBS, MD (Pharmacology), Grad Dip HPE, is an Assistant Professor of Pharmacology at Gulf Medical University, Ajman, United Arab Emirates. She is also the coordinator of the Medical Education Unit of the university. GAMINI PREMADASA, MD (Medical Education), is a medical educationist at Gulf Medical University, Ajman, United Arab Emirates. He had served previously as Director, RTTC of WHO (SEARO) at the University of Peradeniya, Sri Lanka; Assistant Director, Medical Education Unit, Faculty of Medicine, Kuwait University and Director, CME Center, Kuwait Institute for Medical Specialization. MOHAMED ARIFULLAHAS received PhD in Pharmacology and a postdoctoral fellowship for two years at Texas Medical University, Lubbock, USA. He is currently the Dean for Admissions and Registers and Head of Department of Pharmacology in the Gulf Medical University, Ajman, United Arab Emirates. JAYADEVAN SREEDHARAN has a PhD in Biostatistics (India), Diploma in Cancer Prevention (USA) and Graduate Diploma in Health Professions Education (UAE). He is also a fellow of UICC. At present he is working as an Assistant Director, Statistical Support Facility and Professor of Biostatistics at Gulf Medical University, Ajman, UAE. KADAYAM GURUSWAMI GOMATHI has a PhD in Biochemistry and a Graduate Diploma in Health Professions Education. She is currently a Professor of Biochemistry and an Associate Dean for Admissions and Registers in the Gulf Medical University, Ajman. She is also a member of the Medical Education Unit of the university.
Acknowledgements The Delphi panel members are gratefully acknowledged. The authors also thank all the faculty who participated in the study.
Declaration of interest: The authors report no declarations of interest.
References American Education Research Association and American Psychological Association. 1999. Standards for educational and psychological testing. Washington, DC: American Education Research Association. Bellamy N, Anastassiades TP, Buchanan WW, Davis P, Lee P, McCain GA, Wells GA, Campbell J. 1991. Rheumatoid arthritis anti-rheumatic drug trials, III: Setting the Delta for clinical trials of anti-rheumatic drugs— results of a consensus development (Delphi) exercise. J Rheumatol 118(12):1908–1915. Best Evidence Medical Education Collaboration. 2003. Guide for topic review groups on carrying out BEME systematic reviews. Retrieved February 3, 2014. Available from http://www.bemecollaboration.org. Bloomfield L, Subramaniam R. 2008. Development of an instrument to measure the clinical learning environment in diagnostic radiology. J Med Imaging Radiat Oncol 52:262–268. Boor K, Van Der Vleuten C, Teunissen P, Scherpbier A, Scheele F. 2011. Development and analysis of D-RECT, an instrument measuring residents’ learning climate. Med Teach 33(10):820–827. Burden R, Fraser B. 1994. Examining teachers’ perceptions of their working environments: Introducing the School Level Environment Questionnaire. Educ Psychol Pract 10:67–71. Cassar K. 2004. Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees. Med Teach 26(3):260–264. Clayton MJ. 1997. Delphi: A technique to harness expert opinion for critical decision-making tasks in education. Educ Psychol 17(4):373–384. Crisp J, Pelletier D, Duffield C, Adams A, Nagy S. 1997. The Delphi method? Nurs Res 46:116–118. Crossley J, Davies H, Humphris G, Jolly B. 2002. Generalisability: A key to unlock professional assessment. Med Educ 36(10):972–978. de Villiers MR, de Villiers PJ, Kent AP. 2005. The Delphi technique in health sciences education research. Med Teach 27(7):639–643. Fish LS, Busby DM. 1996. The Delphi method. In: Prenkl DH, Moon SM, editors. Research methods in family therapy. New York: The Guilford Press. pp 469–482. Genn JM. 2001. AMEE medical education guide no. 23 (part 2): Curriculum, environment, climate, quality and change in medical education – A unifying perspective. Med Teach 23(5):445–454. Green B, Jones M, Hughes D, Williams A. 1999. Applying the Delphi technique in a study of GPs’ information requirements. Health Soc Care Commun 7(3):198–205. Harden RM. 2001. The learning environment and the curriculum. Med Teach 23:335–336. Hasson F, Keeney S, McKenna H. 2000. Research guidelines for Delphi survey technique. J Adv Nurs 32(4):1008–1015. Henzi D, Davis E, Jasinevicius R, Hendricson W, Cintron L, Isaacs M. 2005. Appraisal of the dental school learning environment: The students’ view. J Dent Educ 69(10):1137–1143. Holt MC, Roff S. 2004. Development and validation of the Anaesthetic Theatre Educational Environment Measure (ATEEM). Med Teach 26(6):553–558. Johnson B, Stevens JJ, Zvoch K. 2007. Teachers’ perceptions of school climate: A validity study of scores from the revised school level environment questionnaire. Educ Psychol Measur 67: 833–844. Johnson B, Stevens JJ. 2001. Exploratory and confirmatory factor analysis of the School Level Environment Questionnaire (SLEQ). Learn Environ Res 4:325–344. Jones J, Hunter D. 1995. Consensus methods for medical and health services research. Br Med J 311(7001):376–380. Keeney S, Hasson F, McKenna HP. 2005. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 38: 195–200.
9
Med Teach Downloaded from informahealthcare.com by Tulane University on 10/18/14 For personal use only.
S. I. Shehnaz et al.
McAleer S, Soemantri D, Roff S. 2009. Educational environment. In: Dent JA, Harden RM, editors. A practical guide for medical teachers. Edinburgh: Churchill Livingstone. pp 64–70. Miles S, Leinster SJ. 2009. Comparing staff and student perceptions of the student experience at a new medical school. Med Teach 31:539–546. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. 2009. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Open Med 3:123–130. Moos RH. 1973. Conceptualizations of human environments. Am Psychol 28:652–655. Moos RH. 1974. The social climate scales: An overview. Palo Alto, CA: Consulting Psychologists Press. Polit DF, Beck CT, Hungler BP. 2001. Essentials of nursing research: Methods, appraisal, and utilisation. Philadelphia: Lippincott. Rentoul AJ, Fraser BJ. 1983. Development of a school-level environment questionnaire. J Educ Admin 21(1):21–39. Roff S, McAleer S, Harden RM, Al-Qahtani M, Uddin AA, Deza H, Groenen G, Primparyon P. 1997. Development and validation of the Dundee Ready Education Environment Measure (DREEM). Med Teach 19(4):295–299. Roff S, McAleer S, Skinner A. 2005. Development and validation of an instrument to measure the postgraduate clinical learning and teaching
10
educational environment for hospital-based junior doctors in the UK. Med Teach 27(4):326–331. Saarikoski M, Leino-Kilpi H. 1999. Association between quality of ward nursing care and students’ assessment of the ward as a learning environment. J Res Nurs 4(6):467–474. Scho¨nrock-Adema J, Bouwkamp-Timmer T, van Hell EA, Cohen-Schotanus J. 2012. Key elements in assessing the educational environment: Where is the theory? Adv Health Sci Educ Theory Pract 17(5):727–742. Shehnaz SI, Sreedharan J, Gomathi KG. 2012. Faculty and students’ perceptions of student experiences in a medical school undergoing curricular transition in the United Arab Emirates. Sultan Qaboos Univ Med J 12(1):628–636. Soemantri D, Herrera C, Riquelme A. 2010. Measuring the educational environment in health professions studies: A systematic review. Med Teach 32:947–952. Streiner DL. 1994. Figuring out factors: The use and misuse of factor analysis. Can J Psychiat 39(3):135–140. Thangaratinam S, Redman CWE. 2005. The Delphi technique. Obstet Gynaecol 7:120–125. Wallengren J. 2011. Identification of core competencies for primary care of allergy patients using a modified Delphi technique. BMC Med Educ 11:12.
