JSLHR

Research Article

Coordination of Gaze and Speech in Communication Between Children With Hearing Impairment and Normal-Hearing Peers Olof Sandgren,a Richard Andersson,a Joost van de Weijer,a Kristina Hansson,a and Birgitta Sahléna

Purpose: To investigate gaze behavior during communication between children with hearing impairment (HI) and normal-hearing (NH) peers. Method: Ten HI–NH and 10 NH–NH dyads performed a referential communication task requiring description of faces. During task performance, eye movements and speech were tracked. Using verbal event (questions, statements, back channeling, and silence) as the predictor variable, group characteristics in gaze behavior were expressed with Kaplan-Meier survival functions (estimating time to gaze-topartner) and odds ratios (comparing number of verbal events with and without gaze-to-partner). Analyses compared the listeners in each dyad (HI: n = 10, mean age = 12;6 years, mean better ear pure-tone average = 33.0 dB HL; NH: n = 10, mean age = 13;7 years).

Results: Log-rank tests revealed significant group differences in survival distributions for all verbal events, reflecting a higher probability of gaze to the partner’s face for participants with HI. Expressed as odds ratios (OR), participants with HI displayed greater odds for gaze-to-partner (ORs ranging between 1.2 and 2.1) during all verbal events. Conclusions: The results show an increased probability for listeners with HI to gaze at the speaker’s face in association with verbal events. Several explanations for the finding are possible, and implications for further research are discussed.

F

examining gaze-to-partner during different verbal events (questions, statements, back channeling, and silence) in a referential communication task.

ace-to-face interaction consists of more than the verbal exchanges that make up the sound stream. The interaction is replete with gazes to the conversational partner. In a previous study, Sandgren, Andersson, van de Weijer, Hansson, and Sahlén (2012) demonstrated that children with normal hearing displayed increased probability of gaze-to-partner when asking questions than when making statements, results indicating that the linguistic and pragmatic content of the conversation influences gaze behavior. But does a hearing impairment affect the use of gaze in conversation additionally? In the present study, we test the hypothesis that children with hearing impairment gaze more to the conversational partner than normal-hearing peers by

Key Words: referential communication, eye tracking, child hearing impairment, gaze behavior, survival analysis

Gaze Behavior in Typical Populations

Correspondence to Olof Sandgren: [email protected] Editor: Rhea Paul Associate Editor: Elizabeth Crais

Gaze behavior during conversation has been the topic of a number of studies, and many provide consistent results (Bavelas, Coates, & Johnson, 2002; Kendon, 1967; Mirenda, Donnellan, & Yoder, 1983; Turkstra, Ciccia, & Seaton, 2003). In free and unscripted dialogue, listeners look more at the speaker than vice versa (Kendon, 1967; Turkstra et al., 2003). Turkstra et al. (2003), for example, in a study of interactive behaviors in adolescents, reported that participants look at their partners 65% of the time when listening compared with 40% of the time when speaking. Kendon (1967), reporting ranges and providing few details on participant characteristics, found that adults looked at their partners 30%–80% of the time when listening versus 20%–65% when speaking. Differences between the studies regarding data collection and

Received October 23, 2012 Revision received April 19, 2013 Accepted September 24, 2013 DOI: 10.1044/2013_JSLHR-L-12-0333

Disclosure: The authors have declared that no competing interests existed at the time of publication.

a

Lund University, Lund, Sweden

942

Journal of Speech, Language, and Hearing Research • Vol. 57 • 942–951 • June 2014 • A American Speech-Language-Hearing Association

analysis should, however, be considered when comparing results. Whereas both studies investigated gaze exchanges between unacquainted interlocutors, Turkstra et al. (2003) used video-based analysis of an adolescent population, and Kendon (1967) studied adults using technology with limited temporal resolution, likely increasing the variability of gaze-to-partner across individuals and obscuring rapid gaze exchanges between participants. The present study uses eye-tracking technology and a semistructured task to address the need for temporal resolution and ecological validity. Several attempts have been made to define the communicative and social role played by gaze to the partner in conversation. Kendon (1967), who used a conversation analytic approach, found a lower rate of gaze-to-partner at the beginning of utterances and a higher rate at the end, causing him to suggest a relation between gaze behavior and turn taking in conversation. This turn-regulating function has also been described by Cummins (2012) and Bavelas et al. (2002), who, in addition, went on to propose that gaze not only regulates turn exchanges but also creates short timewindows for back channeling responses (for example, “Yeah,” “Uh-huh,” “Mhm”) within an ongoing speaking turn. Previous studies, reviewed and summarized by Mirenda et al. (1983), have proposed additional, pragmatically related functions of gaze, including declaring an interest in the topic, expressing the degree of intimacy with the conversational partner, providing the speaker with information on misunderstandings and communicative breakdowns, and expressing emotions. By analogy, Turkstra (2005) concluded that an atypical gaze behavior can cause “negative outcomes that range from missing a cue to end a conversation to misperceiving the emotional communication of one’s partner” (Turkstra, 2005, p. 1430).

Gaze Behavior in Atypical Populations Gaze behavior is known to be affected by various clinical disorders, including autism spectrum disorders (ASDs), which are associated with a lower rate of gaze-to-partner (Corden, Chilvers, & Skuse, 2008) and/or a distorted timing of gazes (Willemsen-Swinkels, Buitelaar, Weijnen, & van Engeland, 1998), and Williams syndrome, for which atypically high levels of gaze-to-partner are reported (DohertySneddon, Riby, & Whittle, 2012). Norbury et al. (2009), in a study investigating the focus of gaze when viewing video clips of emotionally engaging dialogue, found normal rates of gaze to the characters’ eyes in teenagers with ASD but, interestingly, no relation between the gaze behavior and the level of social competence. Instead, gaze behavior was related to communicative competence. The authors concluded that the role of gaze behavior, and in particular gaze to the partner’s eyes, in the social competence of the teenagers with ASD may have been exaggerated and that linguistic ability, attention, and rigidity of behavior better predict social competence outcome. It has also been proposed that gaze-to-partner in ASD may reflect more of an orienting function and less of a reciprocal social exchange (Nadig, Lee, Singh, Bosshart, &

Ozonoff, 2010). This stems from an experiment in which children with high-functioning autism, speaking about personal interests as opposed to generic topics, exhibited more atypical verbal production but more typical gaze behavior. Nadig et al. (2010) concluded that speaking about a personal interest made participants with ASD more stereotypical and monologue-like. Higher, more typical rates of gaze-to-partner were only made possible because talking about a highly practiced topic made cognitive resources available to be directed at, for example, the partner. Nadig et al.’s (2010) conclusion, which implies that an appropriate gaze behavior requires sufficient allocation of cognitive resources, finds support in the cognitive load hypothesis proposed by Glenberg, Schroeder, and Robertson (1998). The cognitive load hypothesis has been investigated in association with gaze aversion, that is, the deliberate avoidance of gaze-to-partner, in children with both typical and atypical development. Doherty-Sneddon and Phelps (2005) investigated why the recipient of a question looks away from the partner and tested whether this behavior serves to reduce the cognitive load or to alleviate the social stress associated with the risk of giving the wrong answer. Using a design comparing face-to-face and video-linked questioning, the authors found that the largest impact on gaze aversion was the degree of difficulty of the questions, indicating that gaze aversion serves more to manage cognitive load than social stress. It is possible that the same mechanisms can help explain Kendon’s (1967) finding of lower rates of gaze-topartner at the beginning of utterances. The linguistic planning required to form an utterance is more easily performed when blocking out unnecessary visual stimuli. On the other hand, turn-regulation mechanisms could also explain gaze aversion in the recipient of a question. By avoiding gaze-to-partner, the recipient claims the speaking turn and signals that a response is imminent.

Present Study From previous research, it can be concluded that gaze-to-partner is actively used in face-to-face conversation. Furthermore, gaze behavior is affected by linguistic and/or social deficits, and results have indicated a possible link between linguistic proficiency and gaze-to-partner. The present study addresses the paucity of research on gaze-topartner in participants with hearing impairment, a population reported to use visual cues more than normal-hearing peers (Skelt, 2006) and more often exhibiting language delay (Gilbertson & Kamhi, 1995; Hansson, Forsberg, Löfqvist, Mäki-Torkko, & Sahlén, 2004; Yoshinaga-Itano & Sedey, 1998). An increased use of visual cues has been suggested as a compensatory strategy in children with hearing impairment, aiding language processing and comprehension and compensating for the degraded auditory input and restricted ability to use incidental hearing for learning (Blamey et al., 2001). This suggestion is further supported by findings of improved speech perception for audiovisual speech over speech presented through the auditory modality only (Bergeson, Pisoni, & Davis, 2003; Garcia & Dagenais, 1998; Most,

Sandgren et al.: Coordination of Gaze and Speech

943

Rothem, & Luntz, 2009; Woodhouse, Hickson, & Dodd, 2009). Furthermore, visual cues are used in the perspective-taking crucial for efficient communication, as shown by children taking into account the partner’s field of view when interpreting instructions (Nadig & Sedivy, 2002; Nilsen & Graham, 2009). Together, the results illustrate the necessity of instant integration of visual information in the perception and processing of speech and in the pragmatic understanding of communication. In a detailed analysis of the gaze behavior of adults with severe-to-profound hearing impairment while speaking to either their audiologist or a family member, Skelt (2006) found the participants with hearing impairment behaved qualitatively similarly to the participants with normal hearing in Kendon’s (1967) study, however with higher rates of gazeto-partner when listening and lower rates when speaking. Skelt (2006) described how the participants with hearing impairment through use of gaze initiations and gaze withdrawals controlled the turn exchanges in the conversations. Although gaze is only one of several tools used in turn regulation, Skelt (2006) emphasized its role in displaying readiness to accept or reject the speaking turn. Participants with hearing impairment are clinically relevant to study since this population in many countries is educated in inclusive settings (Hyde & Power, 2003; Stinson & Antia, 1999). Modern teaching often involves classroom tasks requiring collaboration between individual students or work in small groups (Toe & Paatsch, 2010), for which questioning and responding is necessary. Consequently, the school setting demands well-functioning interaction, verbal as well as nonverbal. Previous studies have found students with hearing impairment to be less prone to request clarifying information despite misunderstanding (Marschark et al., 2007) and to exhibit difficulties with turn taking, at least when normal-hearing turn-taking behavior is considered the norm (Duchan, 1988). To address these issues, the present study reports data on participants in middle childhood, a period of increasing demands on independence in school work and peer interaction, and investigates gaze behavior for its role in the buildup and management of social interaction. The present study used a referential communication task requiring collaboration between participants for successful completion. In order to ensure ecological validity, a semistructured, unscripted paradigm was used, defining the procedure without restricting the participants’ production or choice of conversational partner. In a previous study of children with normal hearing, Sandgren et al. (2012) have shown the verbal production of questions, statements, back channeling, and silence to influence the probability of gazeto-partner. In the present study, we investigate the influence of these verbal events on the probability of gaze-to-partner in children with hearing impairment. While expecting the verbal event to influence the probability of gaze-to-partner for all participants, hearing impaired or not, we hypothesized that the participants with hearing impairment would consistently exhibit higher probability of gaze-to-partner than the participants with normal hearing, possibly using gaze-topartner as a compensatory strategy.

Method Participants HI–NH dyads. Twenty children and adolescents (for the sake of brevity, henceforth labeled children), seven girls and 13 boys, ranging in age between 9;8 (years;months) and 15;10 (M = 12;4, SD = 1;9) were recruited to form conversational pairs. Out of these, 10 participants, three girls and seven boys, mean age = 12;6 years, SD = 2;0, labeled HI), had documented bilateral mild-to-moderate sensorineural hearing impairment, that is, pure-tone average (PTA) airconduction hearing thresholds for octave frequencies from 0.5 to 4 kHz (ISO 8253–1, 2010) between 26 and 55 dB HL (Clark, 1981), and had received bilateral hearing aids. In the collected sample, better ear PTA ranged between 20 and 43 dB HL (M = 33.0, SD = 7.8). All impairments were symmetrical (mean difference = 7.1 dB, SD = 6.1). According to medical records, mean age at identification of the hearing impairment was 3;7 years (SD = 1;1) and mean age at amplification was 5;2 years (SD = 2;7). An outlier, with 25 dB HL, identified at 5;0 years and receiving amplification at 10;0 years, was found not to differ from the other participants on the measures obtained in this study and was included in the analyses. All participants with hearing impairment were raised in oral speaking families and were educated in oral settings, exhibited no speech impairments, and were given no formal training in sign language, visually aided communication, or speech reading. The remaining 10 participants, four girls and six boys (mean age = 12;3, SD = 1;7) were normal-hearing same-age peers invited by the participant with hearing impairment to take part in the study as conversational partners. All participants with hearing impairment chose to bring a classmate, thus, a partner familiar with their hearing loss, differing maximally 1 year in age. All except three HI–NH dyads consisted of same-sex participants. NH–NH dyads. Twenty children and adolescents, 10 girls and 10 boys, ranging in age between 10;2 and 15;4 (M = 13;6, SD = 1;11), were recruited to form normal-hearing control dyads. Half of the participants in the control dyads, five girls and five boys (mean age = 13;7, SD = 1;11, labeled NH) composed a control group, matched to the ages of the HI group. The other half, five girls and five boys (mean age = 13;5, SD = 2;0), were classmates invited by their NH peers to participate as conversational partners. All NH–NH dyads consisted of same-sex participants. Group descriptives are summarized in Table 1. The HI and NH groups did not differ significantly on age, t(18) = 1.281, p = .22, or receptive grammar, t(18) = 1.469, p = .159, as measured by standardized assessment with the Test for Reception of Grammar, Version 2 (TROG-2; Bishop, 2009). All participants had Swedish as their first language and all had nonverbal IQs within normal limits as measured by Raven’s Standard Progressive Matrices (Raven, Raven, & Court, 2004). All participants had normal or corrected to normal vision, and all normal-hearing participants passed a 20 dB pure-tone hearing screening at 0.5, 1, 2, 4, and 6 kHz before data collection. Ethical approval for the

944 Journal of Speech, Language, and Hearing Research • Vol. 57 • 942–951 • June 2014

Table 1. Group descriptives. Study groups and conversation partners (n; sex)

Mean age (SD)

Mean BEPTA (SD)

Mean age at identification (SD)

Mean age at amplification (SD)

Target HI (n = 10; 3 f, 7 m) HI partner (n = 10; 4 f, 6 m) Control NH (n = 10; 5 f, 5 m) NH partner (n = 10; 5 f, 5 m)

12;6 (2;0) 12;3 (1;7) 13;7 (1;11) 13;5 (2;0)

33.0 (7.8)

3;7 (1;1)

5;2 (2;7)

Note. BEPTA = better ear pure-tone average; HI = participants with hearing impairment; HI partner = normal-hearing conversational partners of HI; NH = normal-hearing control group; NH partner = normal-hearing conversational partners of NH; f = female participants; m = male participants.

study was granted by the Regional Ethics Review Board for southern Sweden, approval number 2009/383.

Materials and Procedure Experimental task. An unscripted referential communication task was used, in which the target children with hearing impairment and normal-hearing controls acted as the listeners, and the conversational partners acted as the speakers. This study reports data on the listener. The task has previously been used in several studies of conversational strategies and interaction in children with language and/or hearing impairment (Ibertsson, Hansson, Mäki-Torkko, Willstedt-Svensson, & Sahlén, 2009; Sandgren, Ibertsson, Andersson, Hansson, & Sahlén, 2011) and in a study of gaze behavior during linguistic problem solving between children with normal hearing (Sandgren et al., 2012). A screen displaying 16 pictures of faces, visible only to the speaker, was placed between the participants. The listener was provided with 24 pictures of faces. The instructions given were for the speaker to describe each picture and its position with as much detail for the listener to be able to identify the correct picture and place it in the correct position. The pictures of faces differed only in details, and the listener was forced to request further information when confronted with an insufficiently detailed description. The analysis focused on the process of task resolution, not the end result, and a ceiling effect was expected. Equipment and data collection. During the referential communication task, the participants wore identical SMI iView X HED head-mounted video-based pupil and corneal reflex eye-tracking systems, calibrated with a 9-point calibration procedure. The data from each eye-tracking system were merged with the video of a forward-facing camera, creating an output video showing the participant’s field of view with a moving cursor indicating gaze position. The video was filmed at 25 frames per second, creating an effective sampling frequency of 25 Hz. The participants were seated approximately 120 cm from each other, separated by the 30-cm tall picture screen. The height of the screen created real-life-like conversational conditions by allowing eye contact and visual cues. The dialogues were video recorded using a fixed digital video camera capturing both participants from a side view. For audio recording, the camera’s built-in

microphone was used. Recordings were made in a quiet laboratory setting in the Humanities Laboratory at Lund University. The dialogues were transcribed orthographically by the first author and transcriptions were exported to ELAN (Wittenburg, Brugman, Russel, Klassmann, & Sloetjes, 2006), an open-source audio and video annotation software, where each listener’s speech was categorized into four types of verbal events: requests, nonrequests, back channeling, and silence. Requests included requests for confirmation of new information (“Has she got blue eyes?”), requests for confirmation of old information (“Did you say she had blue eyes?”), and requests for elaboration (“What color are her eyes?”). In a previous study (Sandgren et al., 2011), these types of requests have been found to account for over 90% of requests. Back channeling included verbal signals of comprehension and interest (for example, “Yeah,” “Uh-huh,” “Mhm”). The remaining speech, including, for example, statements from the listener not directly related to the task resolution, such as “He looks a bit like your dad,” was categorized as nonrequests in order to provide a baseline condition for comparison in the analysis of gaze-to-partner during requests. Similarly, periods of silence, often constituting the partner speaking, were categorized to provide a baseline condition in the analysis of gaze-to-partner during back channeling. The fourth author independently coded the verbal events in 25% of the dialogues. The interrater reliability as estimated with Cohen’s kappa was .941. Table 2 provides examples and group data on verbal event types. Annotation of eye movements was made by the first author using ELAN (Wittenburg et al., 2006). The output videos of the eye-tracking systems were merged and synchronized with the orthographic transcription, creating an annotation file containing all verbal and gaze annotations. Three areas of interest regarding gaze focus were specified: task (the pictures of faces), face (the partner’s face), and off (gaze focused elsewhere). All instances of gaze within the specified areas of interest were recorded, providing information on the participants’ gaze focus for the duration of the conversation. The second author independently annotated the eye movements in 20% of the data. Reliability was measured using the overlap calculation in ELAN (Wittenburg et al., 2006), with a modification weighting the annotations on their duration in time, allowing annotations of greater

Sandgren et al.: Coordination of Gaze and Speech

945

Table 2. Verbal event types, descriptions, examples, and distribution. Verbal event type

Description

Requests

Questions

Nonrequests Back channeling Silence Total

Statements Feedback Partner speaking

Example

“Has she got blue eyes?”a “Did you say she had blue eyes?”b “What color are her eyes?”c “He looks a bit like your dad.” “Uh-huh.” “Mhm.”

n (HI)

n (NH)

288 (194)a (54)b (40)c 176 269 745 1,478

254 (182)a (57)b (15)c 309 165 740 1,468

Note. n shows number of verbal events of each type. a

Request for confirmation of new information. bRequest for confirmation of old information. cRequest for elaboration.

duration to affect the reliability score more than shorter annotations. The interrater reliability was 88.5%.

Data Analysis Gaze and verbal annotation data were extracted from ELAN for analysis. In all, 2,946 cases of verbal events were identified and used in the analyses. The dependent variable (gaze to the speaker’s face) was scored binarily, on 10-ms intervals, over a 3,000-ms time window centered at the onset of the predictor event (the verbal events). Thus, for each case of a verbal event, 300 measurements of the occurrence of gaze-to-partner (1/0) were made, covering the time span between 1,500 ms preceding and 1,500 ms following the verbal event onset. The raw data were plotted in SPSS to provide probability plots of gaze to the partner’s face for the different verbal event types. Figure 1 shows a schematic illustration of the verbal events and the gaze analysis window. In order to analyze not only if but also when gaze to the speaker’s face occurred, data were fitted to a survival function estimating the probability of the target event to

occur. The survival function estimates the event time (the time from the beginning of measurements, in this case 1,500 ms preceding the verbal event onset) to the target event, while statistically accommodating the influence of censored cases (verbal events performed without gaze-to-partner within the time window) and displaying the probability of target event occurrence as a cumulative survival. For group comparisons of the probability of gaze-to-partner during the different verbal event types, Kaplan-Meier survival analysis with Mantel-Cox log-rank tests were performed in SPSS. Three analyses were performed, comparing (a) the probability of gaze-to-partner during requests to the probability during a baseline of nonrequests; (b) the probability of gaze-to-partner during back channeling to the probability during a baseline of silence; and (c) the probability of gaze-to-partner during the two main types of requests, that is, requests for confirmation of new information and requests for confirmation of old information. The raw data (883,800 rows) used for the probability plots were aggregated in two steps to (a) display whether the target event (gaze-to-partner) occurred within the 3,000-ms time window and (b) establish the point in time at the target event. The

Figure 1. Schematic illustration of verbal and gaze data. aRequest for confirmation of new information. bRequest for confirmation of old information. cRequest for elaboration. dNonrequests. eBack channeling. fSilence. Gaze analysis window showing 3,000-ms time frame for study of listeners’ gaze-to-partner, centered at verbal event onset.

946 Journal of Speech, Language, and Hearing Research • Vol. 57 • 942–951 • June 2014

Kaplan-Meier survival analysis eliminates cases as they experience the target event, expressed graphically as a declining slope ultimately showing the “survivors,” or censored cases (that is, the cases not experiencing the target event). Quantification of group differences in survival time was expressed as odds ratios obtained by dividing the ratio of cases experiencing the target event in the two groups with the ratio of censored cases from both groups. Odds ratios with confidence intervals not encompassing 1 indicate significant differences between the groups.

Results Raw Data Examination In an initial analysis, raw data were examined to explore the probability of gaze-to-partner during the various verbal events. Figure 2 provides an example of the raw data, showing probability plots for gaze-to-partner during requests (including requests for confirmation of new information, requests for confirmation of old information, and requests for elaboration) compared with a nonrequest baseline in a 3,000-ms analysis window centered at the request/nonrequest onset. Raw data exploration indicates a higher probability of gaze-to-partner after request onset for HI than for NH. Analysis of the corresponding raw data graphs generated for gaze-to-partner during back channeling compared

with a baseline of silence indicates that back channeling onset entailed a decreased probability of gaze-to-partner, still, however, with higher probability in the HI group. While indicating a higher overall probability of gaze-topartner for participants with HI than for participants with NH, raw data graphs displaying gaze-to-partner during the two types of requests for confirmation (of new and old information, respectively) do not demonstrate clear differences between request types.

Survival Function Data were fitted to a Kaplan-Meier survival analysis estimating the event time from beginning of measurements to occurrence of gaze-to-partner for the verbal events of the participant groups and displaying the estimate as a cumulative survival. Figure 3 presents survival estimates for request and nonrequest, with data labels displaying censored data (that is, verbal events produced without gaze-to-partner). Mantel-Cox log-rank tests of equality of survival distributions revealed significantly lower survival rates for the HI group for requests (c2 (1, N = 542) = 4.826, p = .028) as well as for nonrequests (c2 (1, N = 485) = 6.354, p = .012). Survival estimates for back channeling and a baseline of silence are presented in Figure 4. Again, Mantel-Cox log-rank statistics revealed significantly lower survival rates

Figure 2. Mean probability of gaze-to-partner during requests (solid line) and nonrequests (dotted line) for participants with NH (upper panel, n = 10) and participants with HI (lower panel, n = 10). The horizontal axis shows time, with 0 ms marking the onset of the verbal event.

Sandgren et al.: Coordination of Gaze and Speech

947

Figure 3. Kaplan-Meier estimates of survival probabilities during requests (left panel) and nonrequests (right panel) for participants with HI (solid line) and participants with NH (dotted line). The horizontal axis shows time, with 0 ms marking the onset of the verbal event. Data labels present censored data.

Figure 4. Kaplan-Meier estimates of survival probabilities during back channeling (left panel) and silence (right panel) for participants with HI (solid line) and participants with NH (dotted line). The horizontal axis shows time, with 0 ms marking the onset of the verbal event. Data labels present censored data.

948 Journal of Speech, Language, and Hearing Research • Vol. 57 • 942–951 • June 2014

distributions and odds ratios were found for the different types of requests for confirmation.

for participants with HI than for participants with NH, regarding both back channeling (c2 (1, N = 434) = 11.801, p = .001) and silence (c2 (1, N = 1485) = 26.881, p < .001). Mantel-Cox log-rank tests of equality of survival distributions for the two types of requests for confirmation (that is, of new and old information) revealed no significant group differences for either request type (requests to confirm new information: c2 (1, N = 376) = 0.879, p = .348; requests to confirm old information: c2 (1, N = 111) = 2.024, p = .155).

Discussion and Conclusions The results presented in this study support the hypothesis that participants with hearing impairment do, indeed, gaze more to their conversational partner during verbal events than do normal-hearing peers (matched in age, nonverbal reasoning, and receptive language skills). The findings— gathered from a task posing demands similar to many school tasks—provide evidence that school-age children and adolescents with HI display significantly lower survival distributions and increased odds for gaze-to-partner when asking questions, when making statements, when providing the speaker with back-channeling responses, and during silence. This discussion proposes directions for the next steps of research, using these basic gaze data to delve into the reasons behind the increased use of gaze. First, task-dependent characteristics may have influenced the gaze behavior. In this referential communication task, the participants were seated face-to-face in a laboratory setting, with competing auditory and visual stimuli kept to a minimum. This may have made the participants with HI more prone to gaze at the speaker than would otherwise have been the case. It is, however, also easy to envision how the laboratory setting could have resulted in the opposite behavior, that is, that the favorable acoustic conditions would have made gaze-to-partner less necessary. Adequate examination of the influence of the task on the participants’ gaze behavior requires the experiment to be replicated in a variety of settings, the most naturalistic being a classroom environment. Future studies should also evaluate the effect of the conversational partner, comparing gaze with known and unknown partners. Furthermore, the referential communication task used in this study should be scrutinized and compared with other tasks encouraging more gaze exchanges between

Odds Ratios Odds ratios were estimated by dividing the ratio of cases experiencing the event with the ratio of censored cases. Table 3 summarizes the data used for the analyses with significant group differences in odds ratios and survival distributions highlighted. A higher probability of gaze-to-partner in association with requests, nonrequests, back channeling, and silence is shown for the participants with hearing impairment, as indicated by significant differences in MantelCox log-rank tests of equality of survival distributions and significant differences in odds ratios, compared with the participants with normal hearing.

Summary of Results To summarize the results, an increased probability of concurrent gaze-to-partner during verbal events was shown for participants with hearing impairment compared with participants with normal hearing. Kaplan-Meier survival functions showed significantly reduced survival rates for participants with HI, reflecting a higher propensity to gaze in association with requests, nonrequests, back channeling, and silence. Odds ratios express this finding by showing an increase in the odds to look at the partner of 1.5 to 2.1 for participants with HI during the same verbal events. While displaying similar overall patterns, nonsignificant survival

Table 3. Data and result summary.

Verbal event

Group

Request

HI NH HI NH HI NH HI NH HI NH HI NH

Nonrequest Back channeling Silence Requests for confirmation of new information Requests for confirmation of old information

Cases with eventa 136 95 71 89 107 39 342 242 88 73 23 17

Censored casesb 152. 159. 105. 220. 162. 126. 403. 498. 106. 109. 31. 40.

Odds ratio [95% CI]

c2

Log rankc p

1.5 [1.1, 2.1]

4.826

.028

1.7 [1.1, 2.5]

6.354

.012

2.1 [1.4, 3.3]

11.801

.001

1.7 [1.4, 2.]

26.881

.000

1.2 [0.8, 1.9]

0.879

.348

1.7 [0.8, 3.8]

2.024

.155

Note. Significant group differences are in bold. a Number of cases experiencing gaze-to-partner. bNumber of cases not experiencing gaze-to-partner. cMantel-Cox p value for test of group difference in survival distribution between HI and NH groups.

Sandgren et al.: Coordination of Gaze and Speech

949

the participants. However, gaze-to-partner during a task requiring visual attention to be directed away from the partner, performed with a friend with whom the child functions well, strengthens, rather than weakens, the argument of gaze to the conversational partner serving an important communicative role. Whereas the present investigation provides necessary groundwork on the gaze behavior in a semistructured task without restrictions on verbal productions, future studies should also use scripted utterances of predetermined duration. By expanding the gaze analysis window, this would provide an experimental control condition to more naturalistic tasks, adding details on the probability of gaze-to-partner over the course of an entire utterance. Second, the higher probability of gaze-to-partner in participants with HI could be interpreted as a way of compensating for the degraded auditory signal. This should be further explored in different ways. Systematic variation of the participants’ access to visual cues would provide information on the possible benefit of visual cues. Previous studies have showed improved speech perception for audiovisual speech compared with speech presented through only one modality, in participants with hearing impairment (see, e.g., Woodhouse et al., 2009), a finding deeply rooted in common knowledge and practice. Future studies should further investigate individual characteristics of the participants as a factor influencing gaze-to-partner. If future studies confirm a compensatory role of gaze-to-partner, attempts must be made to tease apart underlying causes that may interact with the hearing impairment such as cognitive and linguistic ability. Participants from other age groups, and other degrees and etiologies of hearing impairment, should be studied. Although inflated by an outlier who received auditory amplification unusually late, the participants in this study received hearing aids at a mean age of approximately 5 years. Today, newborn infants in Sweden undergo otoacoustic emission screening for hearing impairment, enabling earlier identification and amplification. Thus, an investigation of whether earlier amplification reduces the need for gaze-to-partner is warranted, as is a replication of the study in participants with more severe, and also unilateral, hearing impairments. Furthermore, a possible increased cognitive load on individuals with hearing impairment could be investigated using measures of gaze aversion. Without ruling out the possibility of the intraindividual benefits described above, a third alternative, encompassing interindividual benefits of gaze-to-partner during conversation, should be investigated. Instead of just a monitoring role for gaze, the combination of gaze and speech can be used to adjust the communicative content to reach conversational objectives. This is similar to how, for example, number and length of speaking turns and the use of requests can be used by individuals with hearing impairment to control conversation (Caissie, Dawe, Donovan, Brooks, & MacDonald, 1998). Skelt (2006) described how participants with hearing impairment, by maintaining or withholding gaze, exert control over the turn exchanges. The conversational partners in her study adhered to the gaze cue, allowing it to “overrule” syntactic and prosodic cues for turn exchange. In our data,

the overall similarities in gaze behavior between the participants with and without hearing impairment can be seen as an indication of gaze serving as a more generally applied turnregulating mechanism, similar for both groups. The group differences in the probability for gaze-to-partner could, therefore, express the extra need for visual cues accumulated by the hearing impairment. If future studies show gaze-to-partner to compensate for auditory deficits and play a role in turn regulation, it is evident that gaze cues, and the opportunity to use them, must be considered vital for the ability of individuals with hearing impairment to participate in interaction on equal terms with their normal-hearing peers. Clinical and educational implications could include increased awareness of the use of gaze during conversation in the child with hearing impairment, family members, teachers, and friends, as well as classroom modifications optimizing both visual and auditory aspects of communication. With well-informed interlocutors and well-adapted surroundings, individuals with hearing impairment are more likely to be able to show their full potential, using both verbal and gaze cues to participate fully in the interaction.

Acknowledgments We gratefully acknowledge the support of the Linnaeus Centre Thinking in Time: Cognition, Communication and Learning, financed by the Swedish Research Council (Grant 349-2007-8695). We would also like to thank Jonas Brännström for valuable comments in the preparation of the manuscript and express our sincerest gratitude to all participants.

References Bavelas, J. B., Coates, L., & Johnson, T. (2002). Listener responses as a collaborative process: The role of gaze. Journal of Communication, 52, 566–580. Bergeson, T. R., Pisoni, D. B., & Davis, R. A. (2003). A longitudinal study of audiovisual speech perception by children with hearing loss who have cochlear implants. The Volta Review, 103, 347–370. Bishop, D. V. M. (2009). Test for Reception of Grammar, Version 2 (M. Garsell, Trans.). Stockholm, Sweden: Pearson. Blamey, P. J., Sarant, J. Z., Paatsch, L. E., Barry, J. G., Bow, C. P., Wales, R. J., . . . Toocher, R. (2001). Relationships among speech perception, production, language, hearing loss, and age in children with impaired hearing. Journal of Speech, Language, and Hearing Research, 44, 264–285. Caissie, R., Dawe, A. L., Donovan, C., Brooks, H., & MacDonald, S. M. (1998). Conversational performance of adults with a hearing loss. Journal of the Academy of Rehabilitative Audiology, 31, 45–67. Clark, J. G. (1981). Uses and abuses of hearing loss classification. ASHA, 23, 493–500. Corden, B., Chilvers, R., & Skuse, D. (2008). Avoidance of emotionally arousing stimuli predicts social-perceptual impairment in Asperger’s syndrome. Neuropsychologia, 46, 137–147. Cummins, F. (2012). Gaze and blinking in dyadic conversation: A study in coordinated behaviour among individuals. Language and Cognitive Processes, 27, 1525–1549.

950 Journal of Speech, Language, and Hearing Research • Vol. 57 • 942–951 • June 2014

Doherty-Sneddon, G., & Phelps, F. (2005). Gaze aversion: A response to cognitive or social difficulty? Memory & Cognition, 33, 727–733. Doherty-Sneddon, G., Riby, D. M., & Whittle, L. (2012). Gaze aversion as a cognitive load management strategy in autism spectrum disorder and Williams syndrome. Journal of Child Psychology and Psychiatry, 53, 420–430. Duchan, J. F. (1988). Assessing communication of hearing-impaired children: Influences from pragmatics. Journal of the Academy of Rehabilitative Audiology, 21(Mono. Suppl.), 19–40. Garcia, J. M., & Dagenais, P. A. (1998). Dysarthric sentence intelligibility: Contribution of iconic gestures and message predictiveness. Journal of Speech, Language, and Hearing Research, 41, 1282–1293. Gilbertson, M., & Kamhi, A. G. (1995). Novel word learning in children with hearing impairment. Journal of Speech and Hearing Research, 38, 630–642. Glenberg, A., Schroeder, J., & Robertson, D. (1998). Averting the gaze disengages the environment and facilitates remembering. Memory & Cognition, 26, 651–658. Hansson, K., Forsberg, J., Löfqvist, A., Mäki-Torkko, E., & Sahlén, B. (2004). Working memory and novel word learning in children with hearing impairment and children with specific language impairment. International Journal of Language & Communication Disorders, 39, 401–422. Hyde, M., & Power, D. (2003). Characteristics of deaf and hardof-hearing students in Australian regular schools: Hearing level comparisons. Deafness and Education International, 5, 133–143. Ibertsson, T., Hansson, K., Mäki-Torkko, E., Willstedt-Svensson, U., & Sahlén, B. (2009). Deaf teenagers with cochlear implants in conversation with hearing peers. International Journal of Language & Communication Disorders, 44, 319–337. International Organization for Standardization. (2010). ISO 8253–1. Acoustics—Audiometric test methods—Part 1: Pure-tone air and bone conduction audiometry. Geneva, Switzerland: Author. Kendon, A. (1967). Some functions of gaze-direction in social interaction. Acta Psychologica, 26, 22–63. Marschark, M., Convertino, C. M., Macias, G., Monikowski, C. M., Sapere, P., & Seewagen, R. (2007). Understanding communication among deaf students who sign and speak: A trivial pursuit? American Annals of the Deaf, 152, 415–424. Mirenda, P. L., Donnellan, A. M., & Yoder, D. E. (1983). Gaze behavior: A new look at an old problem. Journal of Autism and Developmental Disorders, 13, 397–409. Most, T., Rothem, H., & Luntz, M. (2009). Auditory, visual, and auditory-visual speech perception by individuals with cochlear implants versus individuals with hearing aids. American Annals of the Deaf, 154, 284–292. Nadig, A., Lee, I., Singh, L., Bosshart, K., & Ozonoff, S. (2010). How does the topic of conversation affect verbal exchange and eye gaze? A comparison between typical development and high-functioning autism. Neuropsychologia, 48, 2730–2739. Nadig, A. S., & Sedivy, J. C. (2002). Evidence of perspective-taking constraints in children’s on-line reference resolution. Psychological Science, 13, 329–336.

Nilsen, E. S., & Graham, S. A. (2009). The relations between children’s communicative perspective-taking and executive functioning. Cognitive Psychology, 58, 220–249. Norbury, C. F., Brock, J., Cragg, L., Einav, S., Griffiths, H., & Nation, K. (2009). Eye-movement patterns are associated with communicative competence in autistic spectrum disorders. Journal of Child Psychology and Psychiatry, 50, 834–842. Raven, J., Raven, J. C., & Court, J. H. (2004). Manual for Raven’s Progressive Matrices and Vocabulary Scales. Section 3: Standard Progressive Matrices: 2000 edition, updated 2004. San Antonio, TX: Pearson. Sandgren, O., Andersson, R., van de Weijer, J., Hansson, K., & Sahlén, B. (2012). Timing of gazes in child dialogues: A timecourse analysis of requests and back channelling in referential communication. International Journal of Language & Communication Disorders, 47, 373–383. Sandgren, O., Ibertsson, T., Andersson, R., Hansson, K., & Sahlén, B. (2011). “You sometimes get more than you ask for”: Responses in referential communication between children and adolescents with cochlear implant and hearing peers. International Journal of Language & Communication Disorders, 46, 375–385. Skelt, L. (2006). See what I mean: Hearing loss, gaze and repair in conversation (Unpublished doctoral dissertation). Australian National University, Canberra. Stinson, M., & Antia, S. (1999). Considerations in educating deaf and hard-of-hearing students in inclusive settings. Journal of Deaf Studies and Deaf Education, 4, 163–175. Toe, D. M., & Paatsch, L. E. (2010). The communication skills used by deaf children and their hearing peers in a question-andanswer game context. Journal of Deaf Studies and Deaf Education, 15, 228–241. Turkstra, L. S. (2005). Looking while listening and speaking: Eye-to-face gaze in adolescents with and without traumatic brain injury. Journal of Speech, Language, and Hearing Research, 48, 1429–1441. Turkstra, L. S., Ciccia, A., & Seaton, C. (2003). Interactive behaviors in adolescent conversation dyads. Language, Speech, and Hearing Services in Schools, 34, 117–127. Willemsen-Swinkels, S. H. N., Buitelaar, J. K., Weijnen, F. G., & van Engeland, H. (1998). Timing of social gaze behavior in children with a pervasive developmental disorder. Journal of Autism and Developmental Disorders, 28, 199–210. Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. (2006, May). ELAN: A professional framework for multimodality research. Paper presented at the Fifth International Conference on Language Resources and Evaluation, Genoa, Italy. Woodhouse, L., Hickson, L., & Dodd, B. (2009). Review of visual speech perception by hearing and hearing-impaired people: Clinical implications. International Journal of Language & Communication Disorders, 44, 253–270. Yoshinaga-Itano, C., & Sedey, A. L. (1998). Language of earlyand later-identified children with hearing loss. Pediatrics, 102, 1161–1171.

Sandgren et al.: Coordination of Gaze and Speech

951

Copyright of Journal of Speech, Language & Hearing Research is the property of American Speech-Language-Hearing Association and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.

Coordination of gaze and speech in communication between children with hearing impairment and normal-hearing peers.

To investigate gaze behavior during communication between children with hearing impairment (HI) and normal-hearing (NH) peers...
298KB Sizes 0 Downloads 0 Views