HHS Public Access Author manuscript Author Manuscript
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01. Published in final edited form as: Acad Emerg Med. 2015 September ; 22(9): 1103–1112. doi:10.1111/acem.12735.
Interview-Based Qualitative Research in Emergency Care Part II: Data Collection, Analysis and Results Reporting
Author Manuscript
Megan L. Ranney, MD, MPH, Zachary Meisel, MD, MPH, Esther K. Choo, MD, MPH, Aris Garro, MD, MPH, Comilla Sasson, MD, MS, and Kathleen Morrow, PhD Departments of Emergency Medicine: Rhode Island Hospital/Alpert Medical School, Brown University, (MLR, EKC, AG) Providence, RI; University of Pennsylvania, (ZM) Philadelphia, PA; University of Colorado, (CS) Aurora, CO; Department of Psychiatry & Human Behavior, The Miriam Hospital/Alpert Medical School, Brown University, (KM) Providence, RI
Abstract
Author Manuscript
Qualitative methods are increasingly being used in emergency care research. Rigorous qualitative methods can play a critical role in advancing the emergency care research agenda by allowing investigators to generate hypotheses, gain an in-depth understanding of health problems or specific populations, create expert consensus, and develop new intervention and dissemination strategies. In Part I of this two-article series, we provided an introduction to general principles of applied qualitative health research and examples of its common use in emergency care research, describing study designs and data collection methods most relevant to our field (observation, individual interviews, and focus groups). Here in Part II of this series, we outline the specific steps necessary to conduct a valid and reliable qualitative research project, with a focus on interviewbased studies. These elements include building the research team, preparing data collection guides, defining and obtaining an adequate sample, collecting and organizing qualitative data, and coding and analyzing the data. We also discuss potential ethical considerations unique to qualitative research as it relates to emergency care research.
INTRODUCTION
Author Manuscript
In Part I of this series, we discussed the role of qualitative methods in emergency care research, and introduced types of qualitative studies and their applications (article immediately preceding in this issue). Here in Part II, we provide an outline of rigorous qualitative methods, define basic qualitative terminology, and provide resources for researchers interested in undertaking their own qualitative work. This guide should not serve as a substitute for consultation with an experienced qualitative researcher, but should serve as an overview for reviewers of qualitative studies, and may help with early phases of study
Corresponding Author: Megan L. Ranney, MD, MPH, 55 Claverick Street, 2nd Floor, Providence, RI 02906, Tel: (401) 444-2557/ Fax: (401) 444-2249, [email protected]. Presentations: American Public Health Association, November 2013, Boston, MA Disclosures: Dr. Choo is supported by NIDA grant K23DA031881; Dr. Ranney is supported by NIMH grant K23 MH095866; Dr. Morrow is supported by NICHD grant K24HD062645; Dr. Garro is supported by American Lung Association ACP Grants Program (ACP-231928-N). Drs. Choo and Meisel, both associate editors at this journal, had no role in the peer-review process or publication decision for this paper.
Ranney et al.
Page 2
Author Manuscript
design. For a high-quality checklist of important components of a qualitative manuscript, we refer readers to the COnsolidated criteria for REporting Qualitative research (COREQ).1 For reviewers, we also recommend the Critical Appraisal Skills Programme’s guide to qualitative research.2
DATA COLLECTION Rigorous data collection is an integral part of qualitative research. The majority of effort in a survey-based quantitative study is typically spent designing a valid and reliable survey instrument and obtaining a non-biased, generalizable sample.3 In a qualitative study, the majority of effort in the design phase should be spent developing a systematic, welldeveloped, data collection protocol.
Author Manuscript
There are three essential components to a qualitative data collection protocol: 1) developing a clear collection strategy, 2) appropriately identifying and sampling the population-ofinterest, and 3) obtaining data (whether by observation, interview, or focus group) in a reproducible and “rich” (detailed and complete) manner. As described in Part I of this series, numerous qualitative data collection strategies exist, ranging from observational to focus groups to analysis of existing data sources.
Author Manuscript
Below we discuss each component in detail, focusing on strategies for interview-based qualitative inquiry. For further information on both interview-based and other qualitative methods, we refer readers to qualitative methods textbooks such as Denzin and Lincoln,4 Crabtree,5 or excellent online resources such as www.qualres.org (produced by the Robert Wood Johnson Foundation), and Family Health International’s “Qualitative Research Methods” (www.fhi360.org/resource/qualitative-research-methods-data-collectors-fieldguide). Developing Data Collection Strategies Qualitative studies usually use written, templated data collection guides that include topic headings, open-ended questions within each topic, and probes that can be used to follow up on critical questions. The data collection guide should be created for each individual study to specifically address that study’s research questions. Data collection guides should be developed in conjunction with topical and methodological experts to ensure accurate content, clarity, validity, appropriate language, and length. A hastily assembled data collection guide and inconsistent data collection (whether via interviews, focus groups, or observation) will result in flawed data, with poor credibility and confirmability.5,6
Author Manuscript
Regardless of the data collection strategy – observational, interview, or analysis of preexisting text - a rigorous qualitative manuscript should explicitly discuss how its data collection strategy was designed.1 As many journals request to publish the data collection guide as an appendix, the guide should also be interpretable to a general audience. Observational Studies—As discussed in Part I, observation may occur in person or remotely (e.g. through video or audio taping, via photographs, or by observing environmental patterns). Other researchers will collect pre-existing text (for instance,
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 3
Author Manuscript
looking at free text in discharge papers or electronic health records). In observational qualitative research, data collection guides may be applied in real time in the field, or may be used distally, to extract data in remote studies or from existing data sources. Even when using unstructured “field notes” instead of a formal data collection protocol, note-taking should be steered by pre-determined guidelines regarding what deserves the researcher’s attention. For a detailed discussion of the development of observational data collection strategies, we refer readers to the resources already mentioned, or to observation-specific textbooks (e.g. Schensul et al.,7 or Kawulich8).
Author Manuscript
Interviews and Focus Groups—Many emergency care qualitative researchers use interviews or focus groups for data collection. An ideal qualitative interview will feel like an extended conversation for the participant – yet will yield data on each topic area outlined in the interview guide.9 Most interview or focus group guides will contain some information that serve as guidelines for the interviewer.
Author Manuscript
There are many approaches to the organization of interviews and focus groups, but one method that we have found to be useful especially for novice researchers is to use an outline format. It begins with a spoken introduction that reviews the overarching goals of the study, rules of the interview / focus group, a discussion of the flow of the session, and a confidentiality statement. This introduction is often read verbatim, but can be paraphrased as long as all pertinent points are discussed. An interview or focus group then often proceeds to an “icebreaker” question – one that is easy, non-controversial, and makes the participant feel comfortable sharing information. The introduction/icebreaker question is usually followed by topic headings, each of which will contain an opening question, main questions, followup questions, and probes (see Table 1). Questions are usually divided according to overarching theme. Each theme (and, sometimes, each question) will be accompanied by an “intent statement,” written for the research staff. The intent statement is meant to clarify the goals of the question, maximize reproducibility of methods, and ensure that all needed information is gathered [Morrow and Rosen, unpublished data]. Major thematic questions are used to begin discussion of each new major topic or subtopic, and are generally open-ended to avoid brief or yes/no responses.
Author Manuscript
Follow-up questions and “probes” typically follow the major thematic questions. Follow-up questions ask for more detail within preplanned areas of inquiry. Probes are short questions that allow an interviewer to delve deeper into what the study participant is saying (e.g. “Who did this? How did you feel?”). A general rule is that more sensitive or difficult questions should be used later in the interview, once the participants have developed comfort and rapport with the interviewer; or should be approached following the participant’s lead. Although an interviewer is expected to address each major thematic question (even if not asking the question verbatim), the probes may or may not be asked, depending on whether or not a participant spontaneously discusses the information requested by the probes. Indeed, although the interview guide provides an overview of the desired content, it is not meant to
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 4
Author Manuscript
be used verbatim. A skilled moderator will almost always alter the order and specific wording of questions, prompts, and probes, departing from the guide as needed to follow interesting and relevant lines of inquiry. Interview and focus group sessions close with the moderator providing a brief summary of the conversation, allowing the participants to clarify or add information. It is often beneficial to have a debriefing after the participants leave, but before the research staff departs a session. This allows the moderator and note taker to record initial impressions while the verbal and non-verbal data are fresh in memory, clarify any ambiguous data that were collected, and record any pertinent details of the session that may not be captured in the notes or recordings.
Author Manuscript
The length and exact content of an interview guide will depend on the type of study, the goals of the study, and the amount of time that is available for the interview session. Most interviews and focus groups should last no more than 90 minutes, as longer sessions can become stressful for participants and may affect the later data quality. Selecting and Training the Data Collectors
Author Manuscript
Because the data collectors themselves become part of, and influence, the process, it is critical to select data collectors carefully. For some studies, it may be desirable for the data collector to have similarities with the participants. For instance, physicians may be more likely to disclose personal attitudes and beliefs in an interview with another physician, than with, for example, a student. For other study topics, using a data collector from a different demographic group or work environment than the participants is helpful.10 For example, an observer who is external to a hospital may be able to observe the hospital employees’ workflow in a more objective, non-partisan manner, and with less effect on “normal” lived experience. Potential authority differentials should also be recognized a priori. For example, a residency director would not be a good choice as a qualitative interviewer to ask residents to discuss work hour issues. Lastly, the sex of the data collector may be important. For instance, women may be less open about a history of partner violence with a male interviewer. Interviewers with a clinical background must avoid inserting their clinical perspective into the data collection. Some interviewers, in fact, prefer to not describe themselves as an MD or PhD when interviewing, in order to minimize bias. Referring to oneself as a “researcher” in this situation is often useful.
Author Manuscript
Data collectors must be aware of their background and biases, and seek to minimize their influence on the data collected.11 Qualitative manuscripts should therefore always describe the training of its data collectors, and acknowledge the ways in which the collectors’ own biases may have influenced data collection, in the manuscript.1 In interview- or focus-group-based studies, many pitfalls for interviewers/moderators can be avoided (see Table 2) to limit the introduction of the moderator’s own viewpoints and opinions into the data. Some recommended techniques include asking open-ended questions, allowing pauses in the conversation, and using reflective statements.5 Closed-ended questions (e.g. “yes/no” questions) lead to conversational dead ends and do not encourage the participant(s) to explain what they mean. Double-barreled questions (i.e. two questions
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 5
Author Manuscript
in one) can be confusing to the participant(s), and often only one of the questions is answered. Leading questions are phrased in a way that implies a certain answer is expected and may result in false statements due to perceived social desirability.9 Not allowing for silence may shut down participant’s thought processes, and may result in a more superficial data set. Reflective techniques allow for participants to confirm a correct understanding, enhance rapport with the moderator, and encourage elaboration on critical themes.
Author Manuscript
Prior to conducting an interview or focus group, the interviewer should practice the process multiple times with the principal investigator, and with researchers or colleagues who are not involved in the study. Practice interviews will help assess clarity and usefulness of questions further; identify areas of potential misunderstanding, awkwardness, or problems with tone or language; and allow the interviewer to learn the cadence and flow of the interview. Practice also allows the whole team to role-play in preparation for difficult interviews, such as those that involve participants who are excessively quiet, excessively verbose, have difficulty staying on topic, or are adversarial. Strategies for engaging such participants or minimizing disruption to the data are amply outlined elsewhere.12 Sampling Strategies
Author Manuscript
A qualitative manuscript must describe its sampling process in detail, and should justify its choice of sampling strategies. Sampling strategies for qualitative research include random, convenience, or purposeful samples of participants. Purposeful sampling is unique to qualitative research. It is often used when a researcher is interested in a specific participant characteristic (e.g. ED patients with low medical literacy would be purposely sampled for a study of discharge instruction comprehension), or to be sure to capture equal numbers of participants from specific strata (e.g. sex, for a study on chest pain discharge instructions).13 Numerous, well-validated techniques for purposeful sampling exist, such as snowball techniques, and critical case sampling.13,14
Author Manuscript
Unlike quantitative studies, sample frames do not need to mimic general populations to be valid. Although large sample sizes are usually not necessary, and may even harm the quality of the data analysis, obtaining an adequate and diverse sample is still crucial for data credibility.5 Also unlike survey or other quantitative studies, it is inadvisable to determine a priori an exact sample size for a qualitative study. To facilitate budgeting and grant applications, researchers often present an expected range of participants based on the study questions, researcher expertise with a specific line of inquiry or population, and published rules-of-thumb. During the conduct of the study itself, however, researchers should determine appropriate sample size by identifying a point at which they will consider the data “saturated.” In general, saturation is defined as the point during data collection where additional data do not provide new information to the researchers.14 Although the precise definition of “saturation” depends on the research project and the qualitative discipline,5,15 the concept implies that the research team is conducting at least a preliminary analysis in real time, to determine when the point of “no new data” is reached (see below Data Management and Analysis for further discussion of the analytic process). This expectation for an ongoing qualitative analytic process -- which may, in turn, modify both sample size and data Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 6
Author Manuscript
collection processes -- starkly contrasts with guidelines for quantitative research, in which any interim analyses must be pre-specified and must be independent of the data collection process. Unlike quantitative research, in which outliers complicate data analysis, in qualitative research outliers (participants with unique stories) are informative. They can be used to challenge themes that have been identified, providing a context for when these themes may not hold true.
Author Manuscript
Example: A qualitative study exploring parents’ adherence to their children’s asthma medication plans identified that most parents adhere to their physicians’ recommendations about duration of therapy, because they “wanted to do what was best for their kids” and they “always follow the doctor’s advice.” Two families, however, discussed that they had developed their own plans for administering their kids’ medications. These plans were explicitly different from those discussed with their physician. On further examination of these outliers, researchers realized that one of these families had a mother who worked in health care; during the interview, this family explained that they had modified their medication plans based on the mother’s perception that she had a greater understanding of her child’s asthma than the doctor. In the other family, a parent had severe asthma; this parent explained that he had prior “bad outcomes” from certain asthma medications, which he wanted to avoid for his kid. These outlier viewpoints corroborate the greater theme, but also illustrate unique situations in which this theme may need to be modified. Qualitative Study Logistics
Author Manuscript Author Manuscript
The logistics of a qualitative study differ slightly between observation studies, interviews, and focus groups. A few logistical elements are, however, universal (see Table 3). For instance, we recommend using at least two recording devices (whether audio or video). If participants are invited into the research setting (e.g., for an interview or focus group), the room(s) used should be vetted prior to the actual data collection, to ensure adequate privacy, control of excessive noise (for instance, from an air conditioning unit), temperature, or any other factors that may preclude good data collection. The location in which an interview or focus group is conducted may influence the quality and type of information provided. Conducting interviews by phone or in community locations may be necessary for practical reasons – as when research subjects have difficulty obtaining transportation to interviews – but should be performed with caution: the quality of a cell phone connection may be poor, or an interview in a library or coffee shop may preclude disclosure of sensitive topics. In some settings and for some topics, it may be advisable to use a white noise machine during the interview. Providing food (but not alcohol) is a good way to ease participants into the experience and provide a natural “ice-breaker.” Depending on the topic under discussion, investigators may wish to have a box of tissues, and/or mental health crisis services available in case participants become emotional during the study. If there is an observational component to the study, the observations should be conducted in a way that minimizes interference with the “real” experience.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 7
Author Manuscript
Example: A research team wishes to learn more about electronic medical records’ effect on physician decisions about asthma treatment. They arrange to conduct observations of physician documentation in real time. They wear research badges, but dress in scrubs and sit inconspicuously in the corner of the physician workroom in order to allow observation of behavior as it would naturally occur without their presence. An acknowledgement of these logistical components (e.g., in what setting the data was collected) is an important part of a qualitative manuscript. Complementing Audio/Visual Data with Written Data
Author Manuscript
Most qualitative data collection includes some form of note-taking in addition to audio or video recording.16 In one-on-one interviews, the note-taker is usually the interviewer. For focus groups, a dedicated note-taker is usually present (in addition to the focus group leader). This note-taker can also serve a dual role of assisting with focus group logistics on the day of the session, such as directing lost participants by telephone, assisting with consent processes, and greeting late arrivals. For observational studies, a data collector may take notes in real time (“field notes”), or after leaving the study site.
Author Manuscript
There are three primary reasons to take notes during and after data collection. The first is that, even with two recording devices, data collection sometimes fails. Good notes may be the only way to keep a specific data set from being completely lost. The second reason is that, particularly with focus groups and ethnographic studies, the audio recording may not do justice to the full complexities of a discussion. For instance, audio cannot capture hand gestures or facial expressions, may not capture who is speaking at a particular time, and may miss softer-speaking participants. By complementing a good note-taker’s written observations with an audio transcript, additional complexity of the data may be captured.5 The third reason is that note-taking can serve as an initial form of analysis. For instance, in many qualitative studies a summary document is created immediately at the conclusion of the session. Indeed, a good practice is to have the researchers “debrief” with one another in person or in written form after an episode of data collection is completed. This debriefing allows corrections to the process if problems are observed with the data collection guide and/or the logistics, helps with iterative refinement of the interview/data collection guide as new data is acquired, and serves as an interim form of analysis, which is particularly important if researchers are trying to determine data saturation in a short period of time.1 Qualitative manuscripts should describe the process of note-taking and its role in informing both changes in data collection strategies, and in analysis.1
Author Manuscript
DATA MANAGEMENT AND ANALYSIS Rigor in qualitative research is ensured through transparency in the coding and analysis process. By acknowledging the background of the researchers performing coding and analysis, and adequately describing the coding and analytic process that was used, researchers of similar backgrounds, using similar methods, should be able to come to the same conclusions if provided the data set. The following guidelines facilitate this level of rigor in qualitative coding and analysis.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 8
Managing the Data
Author Manuscript
Data from recordings should be transcribed and de-identified. It can be transcribed by the researchers themselves, although so doing is time-consuming, or by a HIPAA-compliant professional transcription service. Some qualitative researchers believe that the analysis process starts in the process of transcribing. Determining how to transcribe and code pauses, verbal inflections, etc., is actually a decision about data analysis. Early-stage qualitative researchers who are learning the methodology will benefit from transcribing one or more sessions initially, as it provides insight into the qualitative process, logistics of recording data, and allows data collectors to be self-reflective about their data collection techniques.
Author Manuscript
Transcripts should be formatted in a way that allows easy coding and that matches the standards of the intended software used to support the data analysis. An example transcript format (with codes applied) is available in Data Supplement 1. It is often easiest to put line numbers on the transcript for coding purposes, although these should be removed before the transcript is uploaded to data management software.17 Many researchers prefer to use data management software to organize transcripts, notes, memos, and coding. Such software is discussed further below. Coding
Author Manuscript
What are codes, anyhow? “Codes” are labels that the research team applies to the data.10 Coding is designed specifically to categorize the data in a way that can be analyzed further. Each code corresponds to an idea or concept. Codes differ from the themes that will emerge after the analysis is complete. Some codes will need to be applied to large amounts of data; others may be more appropriately applied to specific words or phrases. Although one may code the researcher’s questions in order to provide context during the analysis process, ultimately only the participants’ words or actions serve as analyzable data. The Coding Structure—Creating the coding structure is perhaps the most challenging part of qualitative research, and is what defines the qualitative approach.18 After reading the transcripts in part or in whole, the researchers will work together to create and then refine a codebook. In the process of coding and re-coding early transcripts, coding should be done as a group, so that the codes themselves (and their definitions) can be clearly delineated and agreed upon. The eventual codebook will include an organized list of the “codes” as well as specific, well-articulated definitions and examples for each code. The codebook will serve to guide the staff tasked with coding the entire set of transcripts in a consistent and reliable fashion.19 A sample coding structure is included with Data Supplement 1.
Author Manuscript
In order to develop a coding scheme, investigators can use a variety of approaches. A common method in emergency care research is “deductive,” or informed by pre-existing hypotheses and pre-existing research, such as in “framework” or “content” analysis.20 In this coding and analytic approach, the researcher creates an initial coding structure based off of the interview guide, and then refines the structure (iteratively) as interviews are completed. Some qualitative research, such as that using large pre-existing data sources, will also quantitatively code (e.g. word counts).
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 9
Author Manuscript
In contrast, in “inductive” analytic approaches, such as those informed by phenomenology or grounded theory, investigators do not a priori impose a theoretical construct or framework on the data, but are entirely open to the ideas that arise from the participants.21 This inductive approach allows for codes, themes, and ideas to arise from the narrative -- but nevertheless starts with some a priori hypotheses and perspectives. This approach requires having a greater number of data points prior to development of a coding structure. In a more inductive approach to coding and analysis, a researcher or team of researchers will read multiple transcripts and begin to create lists of themes, and memos about these themes, that will then coalesce into a coding structure. This, too, will be iteratively revised as coding progresses. Inductive analytic approaches are generally more difficult and time-consuming than deductive approaches, but are extremely valuable to the field.
Author Manuscript
A strong coding structure will inform the researcher’s analysis but is also, necessarily, informed by the analytic approach. For instance, researchers who are using a deductive approach will decide on many codes prior to reading their transcripts, whereas those using an inductive approach will read a number of transcripts prior to identifying any potential codes. Similarly, qualitative studies that reflect a certain theoretical background should be sure to include codes for all of the elements of this theory; a study that is informed by the Transtheoretical Model, for instance, should include codes for specific “stages of change,” “decisional balance,” “self-efficacy,” etc. The analytic approach should therefore be defined a priori (see Analysis, below for further details about the analytic approach). Many researchers find that their earlier codes are more descriptive (who, what, where), whereas codes developed later in the process are more analytic (how this is important, how it relates to theory).
Author Manuscript
Regardless of the methods used to create the initial codebook, qualitative researchers should maintain flexibility to revise the codebook; adding, subtracting, modifying, and clarifying coding categories as the work progresses. This process involves a “constant comparison” of new data in the context of existing coding categories. It is important for the credibility of the work to keep an “audit trail” of changes in the codebook as the coding and analysis progresses. The Coding Process. As the codebook is created, the researchers will apply it to their data. The process of applying codes to the content of the transcripts is often stylistic; some researchers prefer to “chunk” larger passages with numerous codes, while others prefer to “split” passages into small sections, each of which is assigned only a few, or single codes. “Chunking” saves time during the coding process and maintains more context, while “splitting” allows for easier identification of coded statements without having to sort through large sections of text and may save time during the analysis phase. See Data Supplement 1 to see an example of how codes have been applied to a transcript.
Author Manuscript
Much of the choice about coding style depends on the research questions and analytic goals. For instance, a researcher may choose to “chunk” broadly when using framework analysis to improve intervention development; or may “split,” coding line-by-line or even word-byword, when using grounded theory to develop a theoretical explanation for why some caregivers have negative experiences during their loved ones’ ED visit. Many studies use both styles.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 10
Author Manuscript
Most projects increase rigor by using double-coding, with more than one researcher independently assigning pre-specified codes to the data. Once the coding structure becomes well defined, some research projects will proceed using a single coder for remaining transcripts, being sure to check in to avoid coder drift throughout the coding process. This saves time. Other studies will double-code ALL transcripts. The former approach is more appropriate for large datasets or for coding structures that are minimally interpretative. The latter approach may be more appropriate for more inductive work, and/or small data sets. Some studies also use advanced data management software to apply preliminary or structural codes to data, for instance based on word counts or question order.
Author Manuscript
Some journals require tracking of concordance between the multiple coders, although this concept is controversial with qualitative researchers.22 Some journals will also request a report of what strategies were applied when discordance occurred to preserve the integrity of the analysis. Concordance is generally calculated as a kappa or inter-rater reliability scale, in which the number of agreed-upon codes is divided by the total number of codes. It is possible to calculate concordance by hand, or using software (although we urge researchers to be aware of the limitations of their data management software, if so doing). In summary, during coding, the investigators will be engaged in three tasks: 1) applying the pre-determined codes; 2) iteratively expanding on, and refining, the coding structure; and 3) taking notes about the data during the coding process. The notes can be attached to pieces of data (such as a quotation), to explain what conclusions the researchers drew, or they may be written as a reflection on an entire code. These notes should be saved and indexed, to inform the broader analysis. Data Analysis
Author Manuscript Author Manuscript
Software—There is a variety of data management software packages for qualitative analysis. It is important to note that these programs facilitate data analysis, but do not perform any analytical techniques themselves. Commonly used software packages include NVivo (formerly NUDIST), Atlas.ti, Microsoft Excel (for simpler datasets), Anthropac, and others. Advantages to using data management software include the ability to quickly identify and categorize data into major categories and themes, and the ability to assign demographic and clinical characteristics to the study participants. Advanced software programs allow for integration of data across categories, can provide a basis for developing research frameworks for a particular topic (see “Coding” section above), and can even auto-code some information (e.g. identity of speaker; sections of an interview) if a transcript is formatted correctly prior to upload.23 Excellent written and online training guides exist for all software types; experienced qualitative researchers can also be a good resource for readers interested in further guidance. The Analytic Process—The analysis of qualitative research is designed to understand the data in context. As described above, it begins with debriefs or field notes, extends into the transcription process and development of coding, and culminates in a consensus-based development of themes.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 11
Author Manuscript
Theme development is the process of identifying common threads between the pieces of the data, which have been artificially divided and categorized by codes. Like the coding process, it is often driven by specific frameworks including inductive or deductive theories, as described above and elsewhere.24 The process of theme development usually occurs in the setting of team meetings during which investigators consider the codes, re-read portions of the transcripts, and describe, characterize, or interpret the themes present in the text. Just like codebook development, theme development represents the collective opinion of the qualitative researchers. Critical, theory-informed immersion in the coding categories is a necessary first step toward developing themes.
Author Manuscript
By comparing data between coding categories, looking for differences in opinions between various types of participants, applying the researchers’ own lived experiences and theoretical perspectives, and collating and refining themes in an iterative manner, a valid and reliable analysis can be achieved. Some researchers explicitly operationalize this process by: 1) starting with a review of the text within a coding category; 2) using data management software to compare the codes for different types of participants (for instance, using Boolean search terms, matrices of participant characteristics, or network diagramming); 3) developing iterative, evolving lists of emerging themes, and revising the codes accordingly; and 4) collaborating with other researchers to compare and contrast emerging themes, finally achieving consensus regarding overarching theoretical constructs. Final themes or theories can be connected to pre-existing hypotheses, theory, or some other external framework, depending on the analytic paradigm used (e.g., framework analysis).
Author Manuscript
A number of well-described techniques exist to ensure the trustworthiness of analysis, such as peer review, member checking, triangulation, and explicit consideration of researcher bias. A good overview of these techniques can be found in all of the textbooks referenced, as well as in Malterud.18 These techniques should be described in the research manuscript.
Author Manuscript
Example: Researchers were interested in how emergency physicians (EPs) defined, used, and perceived asthma management guidelines. They interviewed EPs from a variety of practice and geographic settings, recorded, transcribed, and coded the narrative data. Using inductive techniques, they categorized the main themes and mapped them back to the central domains they were investigating. For definition/ awareness, they identified the themes of engagement and relationships between local and national recommendations. For use, they identified communication and expectation management. For attitudes, they identified physician autonomy, standardization, and unintended consequences. After identifying the themes, they then presented the themes back to a group of EPs to ensure that they were adequately representing their beliefs. Unlike quantitative analysis, qualitative analysis occurs contemporaneously with data collection. The two forms of analysis also contrast in their explicit acknowledgement of bias. Quantitative methods approach factors external to the hypothesis being tested as “noise” that needs to be “controlled” for, using various methodologic and statistical approaches. In contrast, qualitative methods embrace these external factors without attempting to eliminate them, and the analysis acknowledges these contextual factors and their effect on the topic being studied. While quantitative analysis often focuses on Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 12
Author Manuscript
correlations and sorting of data independent of context, qualitative analysis identifies connections between participant statements and the observed events in order to describe a coherent and whole phenomenon. Qualitative analyses may produce ideas that can lead to testable hypotheses, but rarely test specific hypotheses on their own. Presenting the Data
Author Manuscript
There are many ways to present qualitative data, including visual, audio, or written descriptions. Many qualitative studies will divide the results section of a paper according to the themes that emerged from the coding analysis. Specific quotations, representing archetypical or particularly rich descriptions of the themes that have emerged, should be provided. Tables can be used to present quotations or higher-level themes and ideas. Some studies will also provide figures showing the network of their constructs and themes. COREQ guidelines encourage manuscripts to include quotations from a wide variety of study participants, to allow independent verification of the credibility of the data.1 It is generally not appropriate to present qualitative findings using classic descriptive statistics, as this implies psychometric and statistical validity that cannot be claimed by qualitative methods.25 Instead, the description of the data should highlight where there is consensus among study participants (“many,” “most”) and where there are important outlier opinions or concepts (“with the exception,” “a few,” “one participant stated”). Other forms of data presentation include word trees, tree maps, and other rapidly emerging strategies. A classic overview of qualitative data visualization methods is Slone26; the various types of data management software are also frequently updating their abilities to provide new data visualization techniques.
Author Manuscript
CHALLENGES AND BARRIERS Potential Threats to Validity of a Qualitative Study
Author Manuscript
Just as with quantitative research, there are expected threats to the validity of qualitative research which can occur at any stage of the study.18 During study design, if researchers create an overly focused interview guide, they may unwittingly limit the data to meet their expectations, and obviate the possibility of considering alternative concepts. This is why some researchers choose to use an inductive method to create their interview guide, starting with open-ended questions to allow the data to guide development of topics and questions. During data collection itself, interviewers sometimes use leading questions, insert their own opinions into the discussion, explicitly affirm or negate participants’ statements, or otherwise influence what participants disclose; these errors may lead to answers that are one-sided, socially desirable, or otherwise “thin.” During data analysis, anecdotalism represents a threat to validity: this occurs when researchers may erroneously attribute increased importance to interesting but potentially non-representative statements. Methods to avoid anecdotalism include using multiple coders, “triangulating” data (when one’s own data or theory is compared with other data sources/theories to corroborate findings), and verifying data by seeking out statements that might contradict or disprove the findings.5 Qualitative studies may not be generalizable to other settings, depending on the researchers’ sampling and analytic methods. Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 13
Author Manuscript
Finally, the sheer amount of data available from a qualitative study (“data overload”) may preclude adequate analysis. Rigorous, credible qualitative analysis requires that researchers fully immerse themselves in the data. Cutting short the analytic process can result in superficial or inaccurate conclusions. Human Subjects Concerns
Author Manuscript
Often the greatest risk for subjects in a qualitative research study is confidentiality.27 A participant’s voice or face on a recording is considered protected health information. Maintaining confidentiality of qualitative recordings is therefore critical. Additionally, during data collection, participants often reveal personal, identifiable information (for instance, regarding their school, their work, or their treatment history), which must be carefully redacted from the transcript. It is particularly important to be aware of potential violations of “third party” confidentiality (the confidentiality of a participant’s friends and family) whose names and stories may be mentioned without their consent. Focus groups offer an additional threat to confidentiality in that study participants may have access to other participants’ data (because it is shared in a group setting). Therefore, focus group leaders must remind all participants that information discussed in the group must remain confidential. One can also caution focus group participants to not disclose particularly private information if they are concerned about confidentiality, as other group members are not strictly bound to confidentiality rules. It is helpful to provide, in the protocol, an avenue for focus group participants to provide such data privately in the course of their research participation.
Author Manuscript
Institutional review board (IRB) approval is always needed for original qualitative research, even for research with peers (e.g., with other physicians).28 Some forms of secondary data analysis (e.g., analysis of pre-existing social media) are often exempt, but researchers should confirm with their local review board. Most IRBs are comfortable with the fluidity of qualitative interview guides, recognizing that interview questions will evolve over the course of a study. Some researchers include photo and voice release forms in their consent documents, depending on the eventual purpose of the data and based on individual institutional and state policies and laws. If researchers do wish to de-identify a recording, to use it in future presentations or products, then video recordings should be pixilated and audio software should be used to alter voices. Regardless, all researchers should specify when the data will be destroyed.
CONCLUSIONS Author Manuscript
Qualitative research offers opportunities for emergency care researchers to explore novel topics, define new survey or assessment methods, generate hypotheses, and develop interventions. As with all forms of research, it must be conducted with rigor at every stage of the process, from initial interview development to participant sampling, creation of a coding structure, applying codes to transcripts, and identifying key themes. With appropriate training and practice, emergency physicians can conduct high-quality qualitative studies and become critical evaluators of and contributors to the emerging body of qualitative emergency care literature.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 14
Author Manuscript
Supplementary Material Refer to Web version on PubMed Central for supplementary material.
References
Author Manuscript Author Manuscript Author Manuscript
1. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007; 19(6):349–57. [PubMed: 17872937] 2. [Accessed Jun 28, 2015] Critical Appraisal Skills Programme. 10 Questions to Help You Make Sense of Qualitative Research. Available at: http://www.caspinternational.org/mod_product/ uploads/CASPQualitative_Research_Checklist_31.05.13.pdf 3. Mello MJ, Merchant RC, Clark MA. Surveying emergency medicine. Acad Emerg Med. 2013; 20:409–12. [PubMed: 23701351] 4. Denzin, N.; Lincoln, Y. Handbook of Qualitative Research. 2. Thousand Oaks, CA: Sage Publications, Inc; 2000. 5. Crabtree, BF.; Miller, WL. Doing Qualitative Research. 2. Thousand Oaks, CA: Sage Publications, Inc; 1999. 6. Lincoln, YS.; Guba, EG. Naturalistic Inquiry. Thousand Oaks, CA: Sage Publications, Inc; 1985. 7. Schensul, SL.; Schensul, JJ.; LeCompte, MD. Essential ethnographic methods: observations, interviews, and questionnaires (Book 2 in Ethnographer’s Toolkit). Walnut Creek, CA: AltaMira Press; 1999. 8. Kawulich BB. Participant observation as a data collection method. Forum Quar Social Res. 2005; 6(2):article 43. 9. Roulston K, deMarrais K, Lewis JB. Learning to interview in the social sciences. Qual Inquiry. 2003; 9(4):643–68. 10. Miles, MB.; Huberman, AM. Qualitative Data Analysis: An Expanded Sourcebook. 2. Thousand Oaks, CA: Sage Publications, Inc; 1994. 11. Richards H, Emslie C. The ‘doctor’ or the ‘girl from the University’? Considering the influence of professional roles on qualitative interviewing. Fam Pract. 2000; 17(1):71–5. [PubMed: 10673494] 12. Ulin, PR.; Robinson, ET.; Tolley, EE. Qualitative Methods in Public Health: A Field Guide for Applied Research. San Francisco, CA: Jossey-Bass; 2005. 13. Curtis S, Gesler W, Smith G, Washburn S. Approaches to sampling and case selection in qualitative research: examples in the geography of health. Soc Sci Med. 2000; 50(7–8):1001–14. [PubMed: 10714922] 14. Morse, JM. Strategies for sampling. In: Morse, J., editor. Qualitative Nursing Research: A Contemporary Dialogue. Thousand Oaks, CA: Sage; 1991. p. 127-31.Rev. Ed 15. Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006; 18(1):59–82. 16. Maloney RS, Paolisso M. What can digital audio data do for you? Field Methods. 2001; 13(1):88– 96. 17. McLellan E, MacQueen KM, Neidig JL. Beyond the qualitative interview: data preparation and transcription. Field Methods. 2003; 15(1):63–84. 18. Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet. 2001; 358(9280): 483–8. [PubMed: 11513933] 19. Carey JW, Morgan M, Oxtoby MJ. Intercoder agreement in analysis of responses to open-ended interview questions: examples from tuberculosis research. Field Methods. 1996; 8(3):1–5. 20. Krippendorff, K. Content Analysis: An Introduction to Its Methodology. 3. Thousand Oaks, CA: Sage Publications; 2012. 21. Starks H, Brown Trinidad S. Choose your method: a comparison of phenomenology, discourse analysis, and grounded theory. Qual Health Res. 2007; 17(10):1372–80. [PubMed: 18000076] 22. Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ. 2001; 322(7294):1115–7. [PubMed: 11337448]
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 15
Author Manuscript
23. Fielding, N.; Lee, R. Computer Analysis and Qualitative Research. Thousand Oaks, CA: Sage Publications; 1998. 24. Sandelowski M. Theory unmasked: the uses and guises of theory in qualitative research. Res Nurs Health. 1993; 16(3):213–8. [PubMed: 8497673] 25. Sandelowski M, Barroso J. Classifying the findings in qualitative studies. Qual Health Res. 2003; 13(7):905–23. [PubMed: 14502957] 26. Sloane DJ. Visualizing qualitative information. Qualitative Rep. 2009; 14(3):488–97. 27. Richards HM, Schwartz LJ. Ethics of qualitative research: are there special issues for health services research? Fam Pract. 2002; 19(2):135–9. [PubMed: 11906977] 28. Lincoln YS, Tierney WG. Qualitative research and institutional review boards. Qual Inquiry. 2004; 10(2):219–34.
Author Manuscript Author Manuscript Author Manuscript Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 16
Table 1
Author Manuscript
Outline of an Interview Guide for Focus Groups or Semi-Structured Interviews Section
Purpose
Introduction
Introduce yourself, establish the ground rules for the discussion (no interruptions, respect each other), reiterate privacy and confidentiality, allow questions.
Main Topics Opening questions
Simple, straightforward, non-sensitive question to get participants talking, overcome awkwardness, and to help them adjust to the “narrative” (vs survey) style of inquiry
Main (“grand tour”) questions
Introduce the major topics of the research project. Often more controversial topics come later so as not to induce early participant discomfort
Follow up questions (“prompts”)
Moves conversation from the main question to a deeper level, asking for more detail, usually within preplanned areas of inquiry.
Probes
Conversational tools that invite further discussion or elaboration. Probes may steer participants to specific areas, ask for more depth or context, or clarify earlier statements.
Author Manuscript
Summary
Closing statement summarizing some of the key points and allowing an opportunity for participants to clarify these key points or add additional pertinent data. Summaries can occur at the end of brief interviews, or after each major topic.
Debrief
Research staff records impressions of session, clarifies ambiguous data, and records data not captured in notes/recordings
Author Manuscript Author Manuscript Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 17
Table 2
Author Manuscript
Qualitative Questions
Author Manuscript
Goal
“Bad” Question
Explanation
“Better” Question
To elicit participants’ conception of prescribed medicines
“Let’s talk about how your child’s medications have helped her asthma.”
The question is leading, and is likely to result in the participant providing a biased and uninformative response. As qualitative data is made up of a participant’s spontaneous answers, this question will likely lead to uncodeable, and therefore unusable, data.
“Tell me about your child’s asthma medications?” [[[pause for response, then probe:]]] “What, if anything, has changed since she started the medications?” [[[Pause for response, then:]]] OK. I hear you saying that even though she has fewer symptoms now, you worry that your kid will get dependent on her meds if she takes them every day? Is this right? Tell me more!” [[[[Pause]]]
•
Leading question.
•
Too general; fails to obtain useable data to inform intervention design.
“OK, here are some screenshots of the intervention. Let’s start with the first one: tell me what this means to you?”
To hear a variety of opinions about the design of an intervention
Author Manuscript
To elicit participants’ perceived barriers to medication use
“Why do you think this program is a good idea?”
“You’re not taking your meds, huh? Are they too expensive?”
•
This is a leading question.
•
The question is judgment-laden.
•
This question is closed-ended, and invites a “yes/no” answer.
“Let’s talk about the last day when you didn’t take your medicines. Can you tell me about what was going on that day?”
Author Manuscript Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Explanation •
These questions are open-ended and do not impose the interviewer’s viewpoint on the participant.
•
They also use “reflective statements,” in which the interviewer paraphrases the participant’s statements to ensure that they are understood correctly, and to prompt further data collection.
•
Elicits specific feedback about intervention design components.
•
Allows participants to describe their thought process about the design in their own words.
•
Allows interviewers to observe the logic of participants’ use of a resource, as well as their verbal responses.
•
Allows participant to describe barriers in their own words, even when unaware that something is a barrier.
•
Non-judgmental.
Ranney et al.
Page 18
Table 3
Author Manuscript
Interview/Focus Groups Logistics Check-List •
Use at least two recording devices
•
Check the room ahead of time (noise, temperature, set-up)
•
Bring facial tissues
•
Refreshments (nothing with a noisy wrapper, though!)
•
Paper and pencils for participants to write down thoughts (as applicable)
•
List of resources (as applicable, according to the interview topic)
•
Assure privacy of the room; post a sign on the interview door, or consider using a white noise machine to obscure the discussion
Author Manuscript Author Manuscript Author Manuscript Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01. Published in final edited form as: Acad Emerg Med. 2015 September ; 22(9): 1103–1112. doi:10.1111/acem.12735.
Interview-Based Qualitative Research in Emergency Care Part II: Data Collection, Analysis and Results Reporting
Author Manuscript
Megan L. Ranney, MD, MPH, Zachary Meisel, MD, MPH, Esther K. Choo, MD, MPH, Aris Garro, MD, MPH, Comilla Sasson, MD, MS, and Kathleen Morrow, PhD Departments of Emergency Medicine: Rhode Island Hospital/Alpert Medical School, Brown University, (MLR, EKC, AG) Providence, RI; University of Pennsylvania, (ZM) Philadelphia, PA; University of Colorado, (CS) Aurora, CO; Department of Psychiatry & Human Behavior, The Miriam Hospital/Alpert Medical School, Brown University, (KM) Providence, RI
Abstract
Author Manuscript
Qualitative methods are increasingly being used in emergency care research. Rigorous qualitative methods can play a critical role in advancing the emergency care research agenda by allowing investigators to generate hypotheses, gain an in-depth understanding of health problems or specific populations, create expert consensus, and develop new intervention and dissemination strategies. In Part I of this two-article series, we provided an introduction to general principles of applied qualitative health research and examples of its common use in emergency care research, describing study designs and data collection methods most relevant to our field (observation, individual interviews, and focus groups). Here in Part II of this series, we outline the specific steps necessary to conduct a valid and reliable qualitative research project, with a focus on interviewbased studies. These elements include building the research team, preparing data collection guides, defining and obtaining an adequate sample, collecting and organizing qualitative data, and coding and analyzing the data. We also discuss potential ethical considerations unique to qualitative research as it relates to emergency care research.
INTRODUCTION
Author Manuscript
In Part I of this series, we discussed the role of qualitative methods in emergency care research, and introduced types of qualitative studies and their applications (article immediately preceding in this issue). Here in Part II, we provide an outline of rigorous qualitative methods, define basic qualitative terminology, and provide resources for researchers interested in undertaking their own qualitative work. This guide should not serve as a substitute for consultation with an experienced qualitative researcher, but should serve as an overview for reviewers of qualitative studies, and may help with early phases of study
Corresponding Author: Megan L. Ranney, MD, MPH, 55 Claverick Street, 2nd Floor, Providence, RI 02906, Tel: (401) 444-2557/ Fax: (401) 444-2249, [email protected]. Presentations: American Public Health Association, November 2013, Boston, MA Disclosures: Dr. Choo is supported by NIDA grant K23DA031881; Dr. Ranney is supported by NIMH grant K23 MH095866; Dr. Morrow is supported by NICHD grant K24HD062645; Dr. Garro is supported by American Lung Association ACP Grants Program (ACP-231928-N). Drs. Choo and Meisel, both associate editors at this journal, had no role in the peer-review process or publication decision for this paper.
Ranney et al.
Page 2
Author Manuscript
design. For a high-quality checklist of important components of a qualitative manuscript, we refer readers to the COnsolidated criteria for REporting Qualitative research (COREQ).1 For reviewers, we also recommend the Critical Appraisal Skills Programme’s guide to qualitative research.2
DATA COLLECTION Rigorous data collection is an integral part of qualitative research. The majority of effort in a survey-based quantitative study is typically spent designing a valid and reliable survey instrument and obtaining a non-biased, generalizable sample.3 In a qualitative study, the majority of effort in the design phase should be spent developing a systematic, welldeveloped, data collection protocol.
Author Manuscript
There are three essential components to a qualitative data collection protocol: 1) developing a clear collection strategy, 2) appropriately identifying and sampling the population-ofinterest, and 3) obtaining data (whether by observation, interview, or focus group) in a reproducible and “rich” (detailed and complete) manner. As described in Part I of this series, numerous qualitative data collection strategies exist, ranging from observational to focus groups to analysis of existing data sources.
Author Manuscript
Below we discuss each component in detail, focusing on strategies for interview-based qualitative inquiry. For further information on both interview-based and other qualitative methods, we refer readers to qualitative methods textbooks such as Denzin and Lincoln,4 Crabtree,5 or excellent online resources such as www.qualres.org (produced by the Robert Wood Johnson Foundation), and Family Health International’s “Qualitative Research Methods” (www.fhi360.org/resource/qualitative-research-methods-data-collectors-fieldguide). Developing Data Collection Strategies Qualitative studies usually use written, templated data collection guides that include topic headings, open-ended questions within each topic, and probes that can be used to follow up on critical questions. The data collection guide should be created for each individual study to specifically address that study’s research questions. Data collection guides should be developed in conjunction with topical and methodological experts to ensure accurate content, clarity, validity, appropriate language, and length. A hastily assembled data collection guide and inconsistent data collection (whether via interviews, focus groups, or observation) will result in flawed data, with poor credibility and confirmability.5,6
Author Manuscript
Regardless of the data collection strategy – observational, interview, or analysis of preexisting text - a rigorous qualitative manuscript should explicitly discuss how its data collection strategy was designed.1 As many journals request to publish the data collection guide as an appendix, the guide should also be interpretable to a general audience. Observational Studies—As discussed in Part I, observation may occur in person or remotely (e.g. through video or audio taping, via photographs, or by observing environmental patterns). Other researchers will collect pre-existing text (for instance,
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 3
Author Manuscript
looking at free text in discharge papers or electronic health records). In observational qualitative research, data collection guides may be applied in real time in the field, or may be used distally, to extract data in remote studies or from existing data sources. Even when using unstructured “field notes” instead of a formal data collection protocol, note-taking should be steered by pre-determined guidelines regarding what deserves the researcher’s attention. For a detailed discussion of the development of observational data collection strategies, we refer readers to the resources already mentioned, or to observation-specific textbooks (e.g. Schensul et al.,7 or Kawulich8).
Author Manuscript
Interviews and Focus Groups—Many emergency care qualitative researchers use interviews or focus groups for data collection. An ideal qualitative interview will feel like an extended conversation for the participant – yet will yield data on each topic area outlined in the interview guide.9 Most interview or focus group guides will contain some information that serve as guidelines for the interviewer.
Author Manuscript
There are many approaches to the organization of interviews and focus groups, but one method that we have found to be useful especially for novice researchers is to use an outline format. It begins with a spoken introduction that reviews the overarching goals of the study, rules of the interview / focus group, a discussion of the flow of the session, and a confidentiality statement. This introduction is often read verbatim, but can be paraphrased as long as all pertinent points are discussed. An interview or focus group then often proceeds to an “icebreaker” question – one that is easy, non-controversial, and makes the participant feel comfortable sharing information. The introduction/icebreaker question is usually followed by topic headings, each of which will contain an opening question, main questions, followup questions, and probes (see Table 1). Questions are usually divided according to overarching theme. Each theme (and, sometimes, each question) will be accompanied by an “intent statement,” written for the research staff. The intent statement is meant to clarify the goals of the question, maximize reproducibility of methods, and ensure that all needed information is gathered [Morrow and Rosen, unpublished data]. Major thematic questions are used to begin discussion of each new major topic or subtopic, and are generally open-ended to avoid brief or yes/no responses.
Author Manuscript
Follow-up questions and “probes” typically follow the major thematic questions. Follow-up questions ask for more detail within preplanned areas of inquiry. Probes are short questions that allow an interviewer to delve deeper into what the study participant is saying (e.g. “Who did this? How did you feel?”). A general rule is that more sensitive or difficult questions should be used later in the interview, once the participants have developed comfort and rapport with the interviewer; or should be approached following the participant’s lead. Although an interviewer is expected to address each major thematic question (even if not asking the question verbatim), the probes may or may not be asked, depending on whether or not a participant spontaneously discusses the information requested by the probes. Indeed, although the interview guide provides an overview of the desired content, it is not meant to
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 4
Author Manuscript
be used verbatim. A skilled moderator will almost always alter the order and specific wording of questions, prompts, and probes, departing from the guide as needed to follow interesting and relevant lines of inquiry. Interview and focus group sessions close with the moderator providing a brief summary of the conversation, allowing the participants to clarify or add information. It is often beneficial to have a debriefing after the participants leave, but before the research staff departs a session. This allows the moderator and note taker to record initial impressions while the verbal and non-verbal data are fresh in memory, clarify any ambiguous data that were collected, and record any pertinent details of the session that may not be captured in the notes or recordings.
Author Manuscript
The length and exact content of an interview guide will depend on the type of study, the goals of the study, and the amount of time that is available for the interview session. Most interviews and focus groups should last no more than 90 minutes, as longer sessions can become stressful for participants and may affect the later data quality. Selecting and Training the Data Collectors
Author Manuscript
Because the data collectors themselves become part of, and influence, the process, it is critical to select data collectors carefully. For some studies, it may be desirable for the data collector to have similarities with the participants. For instance, physicians may be more likely to disclose personal attitudes and beliefs in an interview with another physician, than with, for example, a student. For other study topics, using a data collector from a different demographic group or work environment than the participants is helpful.10 For example, an observer who is external to a hospital may be able to observe the hospital employees’ workflow in a more objective, non-partisan manner, and with less effect on “normal” lived experience. Potential authority differentials should also be recognized a priori. For example, a residency director would not be a good choice as a qualitative interviewer to ask residents to discuss work hour issues. Lastly, the sex of the data collector may be important. For instance, women may be less open about a history of partner violence with a male interviewer. Interviewers with a clinical background must avoid inserting their clinical perspective into the data collection. Some interviewers, in fact, prefer to not describe themselves as an MD or PhD when interviewing, in order to minimize bias. Referring to oneself as a “researcher” in this situation is often useful.
Author Manuscript
Data collectors must be aware of their background and biases, and seek to minimize their influence on the data collected.11 Qualitative manuscripts should therefore always describe the training of its data collectors, and acknowledge the ways in which the collectors’ own biases may have influenced data collection, in the manuscript.1 In interview- or focus-group-based studies, many pitfalls for interviewers/moderators can be avoided (see Table 2) to limit the introduction of the moderator’s own viewpoints and opinions into the data. Some recommended techniques include asking open-ended questions, allowing pauses in the conversation, and using reflective statements.5 Closed-ended questions (e.g. “yes/no” questions) lead to conversational dead ends and do not encourage the participant(s) to explain what they mean. Double-barreled questions (i.e. two questions
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 5
Author Manuscript
in one) can be confusing to the participant(s), and often only one of the questions is answered. Leading questions are phrased in a way that implies a certain answer is expected and may result in false statements due to perceived social desirability.9 Not allowing for silence may shut down participant’s thought processes, and may result in a more superficial data set. Reflective techniques allow for participants to confirm a correct understanding, enhance rapport with the moderator, and encourage elaboration on critical themes.
Author Manuscript
Prior to conducting an interview or focus group, the interviewer should practice the process multiple times with the principal investigator, and with researchers or colleagues who are not involved in the study. Practice interviews will help assess clarity and usefulness of questions further; identify areas of potential misunderstanding, awkwardness, or problems with tone or language; and allow the interviewer to learn the cadence and flow of the interview. Practice also allows the whole team to role-play in preparation for difficult interviews, such as those that involve participants who are excessively quiet, excessively verbose, have difficulty staying on topic, or are adversarial. Strategies for engaging such participants or minimizing disruption to the data are amply outlined elsewhere.12 Sampling Strategies
Author Manuscript
A qualitative manuscript must describe its sampling process in detail, and should justify its choice of sampling strategies. Sampling strategies for qualitative research include random, convenience, or purposeful samples of participants. Purposeful sampling is unique to qualitative research. It is often used when a researcher is interested in a specific participant characteristic (e.g. ED patients with low medical literacy would be purposely sampled for a study of discharge instruction comprehension), or to be sure to capture equal numbers of participants from specific strata (e.g. sex, for a study on chest pain discharge instructions).13 Numerous, well-validated techniques for purposeful sampling exist, such as snowball techniques, and critical case sampling.13,14
Author Manuscript
Unlike quantitative studies, sample frames do not need to mimic general populations to be valid. Although large sample sizes are usually not necessary, and may even harm the quality of the data analysis, obtaining an adequate and diverse sample is still crucial for data credibility.5 Also unlike survey or other quantitative studies, it is inadvisable to determine a priori an exact sample size for a qualitative study. To facilitate budgeting and grant applications, researchers often present an expected range of participants based on the study questions, researcher expertise with a specific line of inquiry or population, and published rules-of-thumb. During the conduct of the study itself, however, researchers should determine appropriate sample size by identifying a point at which they will consider the data “saturated.” In general, saturation is defined as the point during data collection where additional data do not provide new information to the researchers.14 Although the precise definition of “saturation” depends on the research project and the qualitative discipline,5,15 the concept implies that the research team is conducting at least a preliminary analysis in real time, to determine when the point of “no new data” is reached (see below Data Management and Analysis for further discussion of the analytic process). This expectation for an ongoing qualitative analytic process -- which may, in turn, modify both sample size and data Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 6
Author Manuscript
collection processes -- starkly contrasts with guidelines for quantitative research, in which any interim analyses must be pre-specified and must be independent of the data collection process. Unlike quantitative research, in which outliers complicate data analysis, in qualitative research outliers (participants with unique stories) are informative. They can be used to challenge themes that have been identified, providing a context for when these themes may not hold true.
Author Manuscript
Example: A qualitative study exploring parents’ adherence to their children’s asthma medication plans identified that most parents adhere to their physicians’ recommendations about duration of therapy, because they “wanted to do what was best for their kids” and they “always follow the doctor’s advice.” Two families, however, discussed that they had developed their own plans for administering their kids’ medications. These plans were explicitly different from those discussed with their physician. On further examination of these outliers, researchers realized that one of these families had a mother who worked in health care; during the interview, this family explained that they had modified their medication plans based on the mother’s perception that she had a greater understanding of her child’s asthma than the doctor. In the other family, a parent had severe asthma; this parent explained that he had prior “bad outcomes” from certain asthma medications, which he wanted to avoid for his kid. These outlier viewpoints corroborate the greater theme, but also illustrate unique situations in which this theme may need to be modified. Qualitative Study Logistics
Author Manuscript Author Manuscript
The logistics of a qualitative study differ slightly between observation studies, interviews, and focus groups. A few logistical elements are, however, universal (see Table 3). For instance, we recommend using at least two recording devices (whether audio or video). If participants are invited into the research setting (e.g., for an interview or focus group), the room(s) used should be vetted prior to the actual data collection, to ensure adequate privacy, control of excessive noise (for instance, from an air conditioning unit), temperature, or any other factors that may preclude good data collection. The location in which an interview or focus group is conducted may influence the quality and type of information provided. Conducting interviews by phone or in community locations may be necessary for practical reasons – as when research subjects have difficulty obtaining transportation to interviews – but should be performed with caution: the quality of a cell phone connection may be poor, or an interview in a library or coffee shop may preclude disclosure of sensitive topics. In some settings and for some topics, it may be advisable to use a white noise machine during the interview. Providing food (but not alcohol) is a good way to ease participants into the experience and provide a natural “ice-breaker.” Depending on the topic under discussion, investigators may wish to have a box of tissues, and/or mental health crisis services available in case participants become emotional during the study. If there is an observational component to the study, the observations should be conducted in a way that minimizes interference with the “real” experience.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 7
Author Manuscript
Example: A research team wishes to learn more about electronic medical records’ effect on physician decisions about asthma treatment. They arrange to conduct observations of physician documentation in real time. They wear research badges, but dress in scrubs and sit inconspicuously in the corner of the physician workroom in order to allow observation of behavior as it would naturally occur without their presence. An acknowledgement of these logistical components (e.g., in what setting the data was collected) is an important part of a qualitative manuscript. Complementing Audio/Visual Data with Written Data
Author Manuscript
Most qualitative data collection includes some form of note-taking in addition to audio or video recording.16 In one-on-one interviews, the note-taker is usually the interviewer. For focus groups, a dedicated note-taker is usually present (in addition to the focus group leader). This note-taker can also serve a dual role of assisting with focus group logistics on the day of the session, such as directing lost participants by telephone, assisting with consent processes, and greeting late arrivals. For observational studies, a data collector may take notes in real time (“field notes”), or after leaving the study site.
Author Manuscript
There are three primary reasons to take notes during and after data collection. The first is that, even with two recording devices, data collection sometimes fails. Good notes may be the only way to keep a specific data set from being completely lost. The second reason is that, particularly with focus groups and ethnographic studies, the audio recording may not do justice to the full complexities of a discussion. For instance, audio cannot capture hand gestures or facial expressions, may not capture who is speaking at a particular time, and may miss softer-speaking participants. By complementing a good note-taker’s written observations with an audio transcript, additional complexity of the data may be captured.5 The third reason is that note-taking can serve as an initial form of analysis. For instance, in many qualitative studies a summary document is created immediately at the conclusion of the session. Indeed, a good practice is to have the researchers “debrief” with one another in person or in written form after an episode of data collection is completed. This debriefing allows corrections to the process if problems are observed with the data collection guide and/or the logistics, helps with iterative refinement of the interview/data collection guide as new data is acquired, and serves as an interim form of analysis, which is particularly important if researchers are trying to determine data saturation in a short period of time.1 Qualitative manuscripts should describe the process of note-taking and its role in informing both changes in data collection strategies, and in analysis.1
Author Manuscript
DATA MANAGEMENT AND ANALYSIS Rigor in qualitative research is ensured through transparency in the coding and analysis process. By acknowledging the background of the researchers performing coding and analysis, and adequately describing the coding and analytic process that was used, researchers of similar backgrounds, using similar methods, should be able to come to the same conclusions if provided the data set. The following guidelines facilitate this level of rigor in qualitative coding and analysis.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 8
Managing the Data
Author Manuscript
Data from recordings should be transcribed and de-identified. It can be transcribed by the researchers themselves, although so doing is time-consuming, or by a HIPAA-compliant professional transcription service. Some qualitative researchers believe that the analysis process starts in the process of transcribing. Determining how to transcribe and code pauses, verbal inflections, etc., is actually a decision about data analysis. Early-stage qualitative researchers who are learning the methodology will benefit from transcribing one or more sessions initially, as it provides insight into the qualitative process, logistics of recording data, and allows data collectors to be self-reflective about their data collection techniques.
Author Manuscript
Transcripts should be formatted in a way that allows easy coding and that matches the standards of the intended software used to support the data analysis. An example transcript format (with codes applied) is available in Data Supplement 1. It is often easiest to put line numbers on the transcript for coding purposes, although these should be removed before the transcript is uploaded to data management software.17 Many researchers prefer to use data management software to organize transcripts, notes, memos, and coding. Such software is discussed further below. Coding
Author Manuscript
What are codes, anyhow? “Codes” are labels that the research team applies to the data.10 Coding is designed specifically to categorize the data in a way that can be analyzed further. Each code corresponds to an idea or concept. Codes differ from the themes that will emerge after the analysis is complete. Some codes will need to be applied to large amounts of data; others may be more appropriately applied to specific words or phrases. Although one may code the researcher’s questions in order to provide context during the analysis process, ultimately only the participants’ words or actions serve as analyzable data. The Coding Structure—Creating the coding structure is perhaps the most challenging part of qualitative research, and is what defines the qualitative approach.18 After reading the transcripts in part or in whole, the researchers will work together to create and then refine a codebook. In the process of coding and re-coding early transcripts, coding should be done as a group, so that the codes themselves (and their definitions) can be clearly delineated and agreed upon. The eventual codebook will include an organized list of the “codes” as well as specific, well-articulated definitions and examples for each code. The codebook will serve to guide the staff tasked with coding the entire set of transcripts in a consistent and reliable fashion.19 A sample coding structure is included with Data Supplement 1.
Author Manuscript
In order to develop a coding scheme, investigators can use a variety of approaches. A common method in emergency care research is “deductive,” or informed by pre-existing hypotheses and pre-existing research, such as in “framework” or “content” analysis.20 In this coding and analytic approach, the researcher creates an initial coding structure based off of the interview guide, and then refines the structure (iteratively) as interviews are completed. Some qualitative research, such as that using large pre-existing data sources, will also quantitatively code (e.g. word counts).
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 9
Author Manuscript
In contrast, in “inductive” analytic approaches, such as those informed by phenomenology or grounded theory, investigators do not a priori impose a theoretical construct or framework on the data, but are entirely open to the ideas that arise from the participants.21 This inductive approach allows for codes, themes, and ideas to arise from the narrative -- but nevertheless starts with some a priori hypotheses and perspectives. This approach requires having a greater number of data points prior to development of a coding structure. In a more inductive approach to coding and analysis, a researcher or team of researchers will read multiple transcripts and begin to create lists of themes, and memos about these themes, that will then coalesce into a coding structure. This, too, will be iteratively revised as coding progresses. Inductive analytic approaches are generally more difficult and time-consuming than deductive approaches, but are extremely valuable to the field.
Author Manuscript
A strong coding structure will inform the researcher’s analysis but is also, necessarily, informed by the analytic approach. For instance, researchers who are using a deductive approach will decide on many codes prior to reading their transcripts, whereas those using an inductive approach will read a number of transcripts prior to identifying any potential codes. Similarly, qualitative studies that reflect a certain theoretical background should be sure to include codes for all of the elements of this theory; a study that is informed by the Transtheoretical Model, for instance, should include codes for specific “stages of change,” “decisional balance,” “self-efficacy,” etc. The analytic approach should therefore be defined a priori (see Analysis, below for further details about the analytic approach). Many researchers find that their earlier codes are more descriptive (who, what, where), whereas codes developed later in the process are more analytic (how this is important, how it relates to theory).
Author Manuscript
Regardless of the methods used to create the initial codebook, qualitative researchers should maintain flexibility to revise the codebook; adding, subtracting, modifying, and clarifying coding categories as the work progresses. This process involves a “constant comparison” of new data in the context of existing coding categories. It is important for the credibility of the work to keep an “audit trail” of changes in the codebook as the coding and analysis progresses. The Coding Process. As the codebook is created, the researchers will apply it to their data. The process of applying codes to the content of the transcripts is often stylistic; some researchers prefer to “chunk” larger passages with numerous codes, while others prefer to “split” passages into small sections, each of which is assigned only a few, or single codes. “Chunking” saves time during the coding process and maintains more context, while “splitting” allows for easier identification of coded statements without having to sort through large sections of text and may save time during the analysis phase. See Data Supplement 1 to see an example of how codes have been applied to a transcript.
Author Manuscript
Much of the choice about coding style depends on the research questions and analytic goals. For instance, a researcher may choose to “chunk” broadly when using framework analysis to improve intervention development; or may “split,” coding line-by-line or even word-byword, when using grounded theory to develop a theoretical explanation for why some caregivers have negative experiences during their loved ones’ ED visit. Many studies use both styles.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 10
Author Manuscript
Most projects increase rigor by using double-coding, with more than one researcher independently assigning pre-specified codes to the data. Once the coding structure becomes well defined, some research projects will proceed using a single coder for remaining transcripts, being sure to check in to avoid coder drift throughout the coding process. This saves time. Other studies will double-code ALL transcripts. The former approach is more appropriate for large datasets or for coding structures that are minimally interpretative. The latter approach may be more appropriate for more inductive work, and/or small data sets. Some studies also use advanced data management software to apply preliminary or structural codes to data, for instance based on word counts or question order.
Author Manuscript
Some journals require tracking of concordance between the multiple coders, although this concept is controversial with qualitative researchers.22 Some journals will also request a report of what strategies were applied when discordance occurred to preserve the integrity of the analysis. Concordance is generally calculated as a kappa or inter-rater reliability scale, in which the number of agreed-upon codes is divided by the total number of codes. It is possible to calculate concordance by hand, or using software (although we urge researchers to be aware of the limitations of their data management software, if so doing). In summary, during coding, the investigators will be engaged in three tasks: 1) applying the pre-determined codes; 2) iteratively expanding on, and refining, the coding structure; and 3) taking notes about the data during the coding process. The notes can be attached to pieces of data (such as a quotation), to explain what conclusions the researchers drew, or they may be written as a reflection on an entire code. These notes should be saved and indexed, to inform the broader analysis. Data Analysis
Author Manuscript Author Manuscript
Software—There is a variety of data management software packages for qualitative analysis. It is important to note that these programs facilitate data analysis, but do not perform any analytical techniques themselves. Commonly used software packages include NVivo (formerly NUDIST), Atlas.ti, Microsoft Excel (for simpler datasets), Anthropac, and others. Advantages to using data management software include the ability to quickly identify and categorize data into major categories and themes, and the ability to assign demographic and clinical characteristics to the study participants. Advanced software programs allow for integration of data across categories, can provide a basis for developing research frameworks for a particular topic (see “Coding” section above), and can even auto-code some information (e.g. identity of speaker; sections of an interview) if a transcript is formatted correctly prior to upload.23 Excellent written and online training guides exist for all software types; experienced qualitative researchers can also be a good resource for readers interested in further guidance. The Analytic Process—The analysis of qualitative research is designed to understand the data in context. As described above, it begins with debriefs or field notes, extends into the transcription process and development of coding, and culminates in a consensus-based development of themes.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 11
Author Manuscript
Theme development is the process of identifying common threads between the pieces of the data, which have been artificially divided and categorized by codes. Like the coding process, it is often driven by specific frameworks including inductive or deductive theories, as described above and elsewhere.24 The process of theme development usually occurs in the setting of team meetings during which investigators consider the codes, re-read portions of the transcripts, and describe, characterize, or interpret the themes present in the text. Just like codebook development, theme development represents the collective opinion of the qualitative researchers. Critical, theory-informed immersion in the coding categories is a necessary first step toward developing themes.
Author Manuscript
By comparing data between coding categories, looking for differences in opinions between various types of participants, applying the researchers’ own lived experiences and theoretical perspectives, and collating and refining themes in an iterative manner, a valid and reliable analysis can be achieved. Some researchers explicitly operationalize this process by: 1) starting with a review of the text within a coding category; 2) using data management software to compare the codes for different types of participants (for instance, using Boolean search terms, matrices of participant characteristics, or network diagramming); 3) developing iterative, evolving lists of emerging themes, and revising the codes accordingly; and 4) collaborating with other researchers to compare and contrast emerging themes, finally achieving consensus regarding overarching theoretical constructs. Final themes or theories can be connected to pre-existing hypotheses, theory, or some other external framework, depending on the analytic paradigm used (e.g., framework analysis).
Author Manuscript
A number of well-described techniques exist to ensure the trustworthiness of analysis, such as peer review, member checking, triangulation, and explicit consideration of researcher bias. A good overview of these techniques can be found in all of the textbooks referenced, as well as in Malterud.18 These techniques should be described in the research manuscript.
Author Manuscript
Example: Researchers were interested in how emergency physicians (EPs) defined, used, and perceived asthma management guidelines. They interviewed EPs from a variety of practice and geographic settings, recorded, transcribed, and coded the narrative data. Using inductive techniques, they categorized the main themes and mapped them back to the central domains they were investigating. For definition/ awareness, they identified the themes of engagement and relationships between local and national recommendations. For use, they identified communication and expectation management. For attitudes, they identified physician autonomy, standardization, and unintended consequences. After identifying the themes, they then presented the themes back to a group of EPs to ensure that they were adequately representing their beliefs. Unlike quantitative analysis, qualitative analysis occurs contemporaneously with data collection. The two forms of analysis also contrast in their explicit acknowledgement of bias. Quantitative methods approach factors external to the hypothesis being tested as “noise” that needs to be “controlled” for, using various methodologic and statistical approaches. In contrast, qualitative methods embrace these external factors without attempting to eliminate them, and the analysis acknowledges these contextual factors and their effect on the topic being studied. While quantitative analysis often focuses on Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 12
Author Manuscript
correlations and sorting of data independent of context, qualitative analysis identifies connections between participant statements and the observed events in order to describe a coherent and whole phenomenon. Qualitative analyses may produce ideas that can lead to testable hypotheses, but rarely test specific hypotheses on their own. Presenting the Data
Author Manuscript
There are many ways to present qualitative data, including visual, audio, or written descriptions. Many qualitative studies will divide the results section of a paper according to the themes that emerged from the coding analysis. Specific quotations, representing archetypical or particularly rich descriptions of the themes that have emerged, should be provided. Tables can be used to present quotations or higher-level themes and ideas. Some studies will also provide figures showing the network of their constructs and themes. COREQ guidelines encourage manuscripts to include quotations from a wide variety of study participants, to allow independent verification of the credibility of the data.1 It is generally not appropriate to present qualitative findings using classic descriptive statistics, as this implies psychometric and statistical validity that cannot be claimed by qualitative methods.25 Instead, the description of the data should highlight where there is consensus among study participants (“many,” “most”) and where there are important outlier opinions or concepts (“with the exception,” “a few,” “one participant stated”). Other forms of data presentation include word trees, tree maps, and other rapidly emerging strategies. A classic overview of qualitative data visualization methods is Slone26; the various types of data management software are also frequently updating their abilities to provide new data visualization techniques.
Author Manuscript
CHALLENGES AND BARRIERS Potential Threats to Validity of a Qualitative Study
Author Manuscript
Just as with quantitative research, there are expected threats to the validity of qualitative research which can occur at any stage of the study.18 During study design, if researchers create an overly focused interview guide, they may unwittingly limit the data to meet their expectations, and obviate the possibility of considering alternative concepts. This is why some researchers choose to use an inductive method to create their interview guide, starting with open-ended questions to allow the data to guide development of topics and questions. During data collection itself, interviewers sometimes use leading questions, insert their own opinions into the discussion, explicitly affirm or negate participants’ statements, or otherwise influence what participants disclose; these errors may lead to answers that are one-sided, socially desirable, or otherwise “thin.” During data analysis, anecdotalism represents a threat to validity: this occurs when researchers may erroneously attribute increased importance to interesting but potentially non-representative statements. Methods to avoid anecdotalism include using multiple coders, “triangulating” data (when one’s own data or theory is compared with other data sources/theories to corroborate findings), and verifying data by seeking out statements that might contradict or disprove the findings.5 Qualitative studies may not be generalizable to other settings, depending on the researchers’ sampling and analytic methods. Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 13
Author Manuscript
Finally, the sheer amount of data available from a qualitative study (“data overload”) may preclude adequate analysis. Rigorous, credible qualitative analysis requires that researchers fully immerse themselves in the data. Cutting short the analytic process can result in superficial or inaccurate conclusions. Human Subjects Concerns
Author Manuscript
Often the greatest risk for subjects in a qualitative research study is confidentiality.27 A participant’s voice or face on a recording is considered protected health information. Maintaining confidentiality of qualitative recordings is therefore critical. Additionally, during data collection, participants often reveal personal, identifiable information (for instance, regarding their school, their work, or their treatment history), which must be carefully redacted from the transcript. It is particularly important to be aware of potential violations of “third party” confidentiality (the confidentiality of a participant’s friends and family) whose names and stories may be mentioned without their consent. Focus groups offer an additional threat to confidentiality in that study participants may have access to other participants’ data (because it is shared in a group setting). Therefore, focus group leaders must remind all participants that information discussed in the group must remain confidential. One can also caution focus group participants to not disclose particularly private information if they are concerned about confidentiality, as other group members are not strictly bound to confidentiality rules. It is helpful to provide, in the protocol, an avenue for focus group participants to provide such data privately in the course of their research participation.
Author Manuscript
Institutional review board (IRB) approval is always needed for original qualitative research, even for research with peers (e.g., with other physicians).28 Some forms of secondary data analysis (e.g., analysis of pre-existing social media) are often exempt, but researchers should confirm with their local review board. Most IRBs are comfortable with the fluidity of qualitative interview guides, recognizing that interview questions will evolve over the course of a study. Some researchers include photo and voice release forms in their consent documents, depending on the eventual purpose of the data and based on individual institutional and state policies and laws. If researchers do wish to de-identify a recording, to use it in future presentations or products, then video recordings should be pixilated and audio software should be used to alter voices. Regardless, all researchers should specify when the data will be destroyed.
CONCLUSIONS Author Manuscript
Qualitative research offers opportunities for emergency care researchers to explore novel topics, define new survey or assessment methods, generate hypotheses, and develop interventions. As with all forms of research, it must be conducted with rigor at every stage of the process, from initial interview development to participant sampling, creation of a coding structure, applying codes to transcripts, and identifying key themes. With appropriate training and practice, emergency physicians can conduct high-quality qualitative studies and become critical evaluators of and contributors to the emerging body of qualitative emergency care literature.
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 14
Author Manuscript
Supplementary Material Refer to Web version on PubMed Central for supplementary material.
References
Author Manuscript Author Manuscript Author Manuscript
1. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007; 19(6):349–57. [PubMed: 17872937] 2. [Accessed Jun 28, 2015] Critical Appraisal Skills Programme. 10 Questions to Help You Make Sense of Qualitative Research. Available at: http://www.caspinternational.org/mod_product/ uploads/CASPQualitative_Research_Checklist_31.05.13.pdf 3. Mello MJ, Merchant RC, Clark MA. Surveying emergency medicine. Acad Emerg Med. 2013; 20:409–12. [PubMed: 23701351] 4. Denzin, N.; Lincoln, Y. Handbook of Qualitative Research. 2. Thousand Oaks, CA: Sage Publications, Inc; 2000. 5. Crabtree, BF.; Miller, WL. Doing Qualitative Research. 2. Thousand Oaks, CA: Sage Publications, Inc; 1999. 6. Lincoln, YS.; Guba, EG. Naturalistic Inquiry. Thousand Oaks, CA: Sage Publications, Inc; 1985. 7. Schensul, SL.; Schensul, JJ.; LeCompte, MD. Essential ethnographic methods: observations, interviews, and questionnaires (Book 2 in Ethnographer’s Toolkit). Walnut Creek, CA: AltaMira Press; 1999. 8. Kawulich BB. Participant observation as a data collection method. Forum Quar Social Res. 2005; 6(2):article 43. 9. Roulston K, deMarrais K, Lewis JB. Learning to interview in the social sciences. Qual Inquiry. 2003; 9(4):643–68. 10. Miles, MB.; Huberman, AM. Qualitative Data Analysis: An Expanded Sourcebook. 2. Thousand Oaks, CA: Sage Publications, Inc; 1994. 11. Richards H, Emslie C. The ‘doctor’ or the ‘girl from the University’? Considering the influence of professional roles on qualitative interviewing. Fam Pract. 2000; 17(1):71–5. [PubMed: 10673494] 12. Ulin, PR.; Robinson, ET.; Tolley, EE. Qualitative Methods in Public Health: A Field Guide for Applied Research. San Francisco, CA: Jossey-Bass; 2005. 13. Curtis S, Gesler W, Smith G, Washburn S. Approaches to sampling and case selection in qualitative research: examples in the geography of health. Soc Sci Med. 2000; 50(7–8):1001–14. [PubMed: 10714922] 14. Morse, JM. Strategies for sampling. In: Morse, J., editor. Qualitative Nursing Research: A Contemporary Dialogue. Thousand Oaks, CA: Sage; 1991. p. 127-31.Rev. Ed 15. Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006; 18(1):59–82. 16. Maloney RS, Paolisso M. What can digital audio data do for you? Field Methods. 2001; 13(1):88– 96. 17. McLellan E, MacQueen KM, Neidig JL. Beyond the qualitative interview: data preparation and transcription. Field Methods. 2003; 15(1):63–84. 18. Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet. 2001; 358(9280): 483–8. [PubMed: 11513933] 19. Carey JW, Morgan M, Oxtoby MJ. Intercoder agreement in analysis of responses to open-ended interview questions: examples from tuberculosis research. Field Methods. 1996; 8(3):1–5. 20. Krippendorff, K. Content Analysis: An Introduction to Its Methodology. 3. Thousand Oaks, CA: Sage Publications; 2012. 21. Starks H, Brown Trinidad S. Choose your method: a comparison of phenomenology, discourse analysis, and grounded theory. Qual Health Res. 2007; 17(10):1372–80. [PubMed: 18000076] 22. Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ. 2001; 322(7294):1115–7. [PubMed: 11337448]
Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 15
Author Manuscript
23. Fielding, N.; Lee, R. Computer Analysis and Qualitative Research. Thousand Oaks, CA: Sage Publications; 1998. 24. Sandelowski M. Theory unmasked: the uses and guises of theory in qualitative research. Res Nurs Health. 1993; 16(3):213–8. [PubMed: 8497673] 25. Sandelowski M, Barroso J. Classifying the findings in qualitative studies. Qual Health Res. 2003; 13(7):905–23. [PubMed: 14502957] 26. Sloane DJ. Visualizing qualitative information. Qualitative Rep. 2009; 14(3):488–97. 27. Richards HM, Schwartz LJ. Ethics of qualitative research: are there special issues for health services research? Fam Pract. 2002; 19(2):135–9. [PubMed: 11906977] 28. Lincoln YS, Tierney WG. Qualitative research and institutional review boards. Qual Inquiry. 2004; 10(2):219–34.
Author Manuscript Author Manuscript Author Manuscript Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 16
Table 1
Author Manuscript
Outline of an Interview Guide for Focus Groups or Semi-Structured Interviews Section
Purpose
Introduction
Introduce yourself, establish the ground rules for the discussion (no interruptions, respect each other), reiterate privacy and confidentiality, allow questions.
Main Topics Opening questions
Simple, straightforward, non-sensitive question to get participants talking, overcome awkwardness, and to help them adjust to the “narrative” (vs survey) style of inquiry
Main (“grand tour”) questions
Introduce the major topics of the research project. Often more controversial topics come later so as not to induce early participant discomfort
Follow up questions (“prompts”)
Moves conversation from the main question to a deeper level, asking for more detail, usually within preplanned areas of inquiry.
Probes
Conversational tools that invite further discussion or elaboration. Probes may steer participants to specific areas, ask for more depth or context, or clarify earlier statements.
Author Manuscript
Summary
Closing statement summarizing some of the key points and allowing an opportunity for participants to clarify these key points or add additional pertinent data. Summaries can occur at the end of brief interviews, or after each major topic.
Debrief
Research staff records impressions of session, clarifies ambiguous data, and records data not captured in notes/recordings
Author Manuscript Author Manuscript Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Ranney et al.
Page 17
Table 2
Author Manuscript
Qualitative Questions
Author Manuscript
Goal
“Bad” Question
Explanation
“Better” Question
To elicit participants’ conception of prescribed medicines
“Let’s talk about how your child’s medications have helped her asthma.”
The question is leading, and is likely to result in the participant providing a biased and uninformative response. As qualitative data is made up of a participant’s spontaneous answers, this question will likely lead to uncodeable, and therefore unusable, data.
“Tell me about your child’s asthma medications?” [[[pause for response, then probe:]]] “What, if anything, has changed since she started the medications?” [[[Pause for response, then:]]] OK. I hear you saying that even though she has fewer symptoms now, you worry that your kid will get dependent on her meds if she takes them every day? Is this right? Tell me more!” [[[[Pause]]]
•
Leading question.
•
Too general; fails to obtain useable data to inform intervention design.
“OK, here are some screenshots of the intervention. Let’s start with the first one: tell me what this means to you?”
To hear a variety of opinions about the design of an intervention
Author Manuscript
To elicit participants’ perceived barriers to medication use
“Why do you think this program is a good idea?”
“You’re not taking your meds, huh? Are they too expensive?”
•
This is a leading question.
•
The question is judgment-laden.
•
This question is closed-ended, and invites a “yes/no” answer.
“Let’s talk about the last day when you didn’t take your medicines. Can you tell me about what was going on that day?”
Author Manuscript Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
Explanation •
These questions are open-ended and do not impose the interviewer’s viewpoint on the participant.
•
They also use “reflective statements,” in which the interviewer paraphrases the participant’s statements to ensure that they are understood correctly, and to prompt further data collection.
•
Elicits specific feedback about intervention design components.
•
Allows participants to describe their thought process about the design in their own words.
•
Allows interviewers to observe the logic of participants’ use of a resource, as well as their verbal responses.
•
Allows participant to describe barriers in their own words, even when unaware that something is a barrier.
•
Non-judgmental.
Ranney et al.
Page 18
Table 3
Author Manuscript
Interview/Focus Groups Logistics Check-List •
Use at least two recording devices
•
Check the room ahead of time (noise, temperature, set-up)
•
Bring facial tissues
•
Refreshments (nothing with a noisy wrapper, though!)
•
Paper and pencils for participants to write down thoughts (as applicable)
•
List of resources (as applicable, according to the interview topic)
•
Assure privacy of the room; post a sign on the interview door, or consider using a white noise machine to obscure the discussion
Author Manuscript Author Manuscript Author Manuscript Acad Emerg Med. Author manuscript; available in PMC 2016 September 01.
