Psychol. Res. 37, 211--227 (1975) © by Springer-Verlag 1975

The Influence of Visual and Auditory Images on the Recall of Items of Visual and Auditory Presentation Mode tL-G. Bosshardt Psyehologisehes Institut der Ruhr-Universit~t Bochum Received August 12, 1974

Summary. Acoustical and visual images were introduced as a distractor task during the retention of words which have been presented in the acoustical and visual mode. The memory material consisted of three classes of nouns which differed as to the ease with which their referent could be imagined in the visual and auditory mode. Lists of eight nouns were presented which had to be recalled l i sec after presentation. Presentation mode and sensory mode of the distraetor imagery were varied factorially in independent groups (n = 18). Although the recall in the present experiment was delayed for 1t sec the modality-effeet, well-known from experiments with immediate recall, was obtained. Moreover, it was found ~hat the items presented acoustically were more "resistant" to distracting visual and auditory images than were the items presented visually. These differences between the presentation modes, were interpreted as being the consequence of encoding the items with different redundancy in STS. I t was assumed that the slowly decaying auditory registers allow a more redundant encoding in the STS; retention of these items requires less processing capacity and is more resistant to distracting imagery. I t was found that the encoding of images of different sensory modality is determined not alone by the meaning of the words to be retained but also by their presentation mode. This result was to be expected on the assumption that imagery and perception interfere with one another more if their sensory mod~lity is identical than if it is different. The theoretical background of this thesis was discussed. To p r e d i c t t h e influence of images on certain m e m o r y t a s k s t h e o r e t i c a l assumptions concerning t h e i m a g e r y processes, t h e m e m o r y system, a n d t h e i n t e r a c t i o n b e t w e e n i m a g e r y a n d m e m o r y processes are required. The t h e o r y o f i m a g e r y processes which h a s been d e v e l o p e d elsewhere (Bosshardt, 1973) can be s u m m a rized as follows: Two levels of i n f o r m a t i o n processing are to be d i s t i n g u i s h e d (compare e.g. B r o a d b e n t , 1971; Moray, t 970). Lower-level processing is c o m p l e t e l y stimuluscontrolled and, therefore, c a n n o t p r o d u c e s t i m u l u s - i n d e p e n d e n t i m a g e r y . I t h a d been a s s u m e d t h a t a c e r t a i n o b j e c t is i m a g i n e d w h e n e v e r t h e higher-level a c t i v i t y is similar t o t h e p e r c e p t u a l a c t i v i t y during t h e p e r c e p t i o n o f this s a m e o b j e c t (Hebb, 1968; Neisser, 1967). As shown b y B o s s h a r d t (1973), t h e r e is h o w e v e r one f u n d a m e n t a l difference b e t w e e n t h e higher-level processes during p e r c e p t i o n a n d i m a g e r y of a certain o b j e c t : T h e continuous i n t e r c h a n g e o f i n f o r m a t i o n s b e t w e e n t h e two processing levels t a k i n g p l a c e while perceiving a n o b j e c t has necessarily to be b l o c k e d while forming a m e n t a l i m a g e o f this object. I t h a s been a r g u e d t h a t t h e lower-level s y s t e m is a t t e n u a t e d in a m o d a l i t y - s p e c i f i c way, i.e. t h e lower-level The author gratefully acknowledges the assistance of Mr. Anyanwu and all the members of the unit "Sprachpsychologie und Informationsverarbeitung".

212

H.-G. Bosshardt

visual system is attenuated by visual images and the lower.level auditory system by auditory images. This hypothesis is the starting point for some straightforward predictions concerning the effect of images on memorizing tasks. It is the aim of the present work to test these predictions empirically. Information processing without storage is impossible. Therefore, in modern theories of memory the distinction between levels of processing is usuMly completed by a similar distinction between two kinds of stores (disregarding the longterm store for the moment - - Atkinson and Shiffrin, i968; Crowder and Morton, 1969; Murray, 1970; Morton, i970; ~Torman and l~umelhart, i970): the sensory register correspond to the subordinate processing level (Atkinson and Shiffrin, i968) whereas the higher-level processing and the short-term store belong somehow together. If taken together with the theory of imagery mentioned above, such a relation between the processing stages and the kind of store renders it possible to predict the effect o£ images on the retrieval of informations from the sensory registers: The modality-specific attenuation by images should make inaccessible the information in a certain sensory register because images of a certain sensory modality attenuate the lower-level informations of this same sensory moda]ity. It is necessary first to discuss the effect of sensory registers on the recall performance so that the effect of their attenuation by images can be predicted precisely. The effect of presentation mode is investigated best by presenting verbal material in the visual and auditory mode, the presentation conditions being as comparable as possible in all other respects. Under these conditions - - as shown by Murdock (1966, 1967 a, i968, 1969 ; Murdock and Walker, i 9 6 9 ) - the last (about) four items presented acoustically have a higher recall probability than the visual items of the same serial position. The performance for the items presented first is usually not different in the two presentation modes. This interaction between presentation mode and serial position is called the modality effect; it is interpreted usually by two assumptions (not mutually exclusive) about the effect of sensory registers : a) In case of acoustical presentation recall of the last few items is superior to the visual presentation because the traces in the auditory register have a longer persistence. Crowder and Morton (i969) assume that some extra information is supplied by the auditory register (or BAS in the terminology of these authors) : "Auditory presentation.., supplies S with extra information as opposed to visual presentation due to the persistence characteristics of PAS. The extra information leading to the superiority of auditory presentation is restricted to the last few scrim positions . . . " (p. 366L). With acoustical presentation the items presented last can be retrieved from both the auditory registers and the short-term memory whereas the recall of items presented visually is accomplished by the short-term memory alone (Crowder and Morton, i969; Morton, i970; Morton and Holloway, 1970). b) Different decay rates of the sensory registers affect the recall as much as they change the encoding and storage in the short-term memory. It has been assumed by Posner (i967) that items presented acoustically can be encoded more easily and quickly in the verbal-auditory code. Even i£ this assumption explains quite well the interaction between presentation mode and speed of presentation (Conrad and ttille, 1958 ; Dornbush, 1968a ; Mackworth, i965; Murray and l~oberts i968) it has

The Influenceof Visual and Auditory Images

213

nevertheless been disproved rather convincingly by Dornbush (1968b), and Murdoek and Walker (i969). Corballis (1966; Corballis and Loveless, 1967) assumed that the slowly decaying auditory trace allows a quite flexible kind of rehearsal and encoding of the items in short-term memory. The rapid decay in the visual registers requires a so-called cumulative rehearsal. Taking into consideration the fact that rehearsal means not only silent repetition but recoding, (compare Sanders, i961; Wickelgren, t964), the assumptions of Corballis point to the possibility of qualitative encoding differences in short-term memory as a function of modality of presentation. Peterson and Johnson (~97i) recently came along to essentially the same conclusion: Depending on the mode of presentation the possibilities for encoding and recoding the material in short-term memory are used in somewhat different ways. There are enough cues to postulate some effects of sensory registers on the processes in short-term memory but it is not yet possible to specify these effects in a way that enables the explanation of the modality-effect in any satisfying way. Subjects can be instructed to produce images in the acoustical and the visual mode (see Bosshardt, 1973). Images of this kind can be used as distractors (in the sense of Murdock, 1967 b) during the retention interval of a memory task. Visual and auditory images as distractors in a memory task influence the recall performance presumably in two different ways : a) Short-term memory requires continuous processing to prevent forgetting (Broadbent, 1963; Posner and Konick, 1966; Posner and Rossman, t965). Both rehearsal of memory items and imagery are performed by the higher-level system of limited processing capacity (i.e. the "central mechanism of limited capacity" in the sense of Kerr, t973). Images as distractors during the retention interval reduce the processing space available for the memory task and reduce therefore the recall performance in comparison to control conditions without a distractor task. Because the higher-level system is presumably involved irrespective of the modality of images or of item presentation, no interaction between the sensory modality of the image and the presentation mode of the memory items can be expected. This influence of images is explained so far by the capacity necessary for producing visual or acoustical images. This kind of influence should manifest itself in a difference between the imagery and control conditions. b) As far as the modality-effect is a consequence of the differential availability of informations in the visual and auditory registers, images as distraetors during retention should reduce the superiority of the acousticalpresentation in a modalityspecific way. The well-established capacity assumption has to be completed by the hypothesis of modality-specific attenuation by images. After acoustical presentation of an item presumably some information is stored in the acoustical register. Visual images formed during the retention interval decrease the probability of encoding and rehearsing these items in the short-term memory by reducing the processing space available for the memory task. Auditory images however reduce not only the processing space but prevent the encoding of information from the auditory register by attenuating the lower-level auditory system. It is expected that the recall of the final items presented acoustically is reduced to a greater extent by auditory than by visual images; the items of the first serial positions should be influenced by auditory and visual images in accordance with the

214

H.-G. Bosshardt

capacity assumption, i.e. in a modality-nonspecific way. In case of visual presentation it is highly improbable that the visual register contains informations at the time images are reproduced; the results of this modality, therefore, are predicted on grounds of the capacity assumption alone. The hypothesis of modality-specific attenuation can be tested not only by introducing images as distractors but it has also some implications for the spontaneous usage of images during memory tasks. Images in this case are task-relevant encoding processes the frequency of which can be manipulated by properly selecting the to-be-memorized items : Paivio and his eoworkers have shown in a number of experiments that it is much more difficult to produce images related to abstract words like "justice" than it is to imagine the referent of a concrete word like "dog" (for summaries of this work see Paivio, 1969, t970, 197ta, b; Rohwer, i970). This difference in the "imagery value" of concrete nouns goes along with the better retention of concrete nouns in paired associate experiments (Paivio, 1965, 1966, t968), in recognition tasks (Gorman, 1961 ; Paivio and Csapo, 1969), in free recall (Paivio 1967; Paivio and Csapo, 1969; Paivio, Yuille and Rogers, t969), and in serial recall (Paivio and Csapo, 1969). Paivio (1969) presumed the existence of two codes - - the verbal and imagery codes. Abstract and concrete words do not differ in the extent to which they are encoded verbally, but they differ with respect to the availability of the imagery code. The superiority of concrete over abstract nouns in memory may be explained by assuming that for the concrete words more attributes (in the sense of Bower, 1967) are encoded; concrete nouns can be encoded b y using both the verbal and the imagery code whereas for encoding abstract nouns only the verbal code is available. What is meant b y the two codes can best be specified by reference to Kintsch (1970) who proposed that the encoding of words is in terms of semantic and image markers. The semantic marker concept is well-defined in psycholinguistics and the I-marker contain the instruction to perform the processes relevant for imaging a certain o]0ject or event. According to the hypothesis of modality-specific attenuation by images, the encoding of I-markers cannot only be varied b y properly selecting items with high or low imagery value, but b y variation of the presentation mode too. I f the sensory mode of the image to be formed and the presentation mode of the items are identical, attenuation of the lower-level system being a prerequisite for imagery is impossible. Because every item is perceived, i.e. processed at the superordinate level, the exchange of information between the processing levels takes place. l=fence, no imagery in the same sensory mode is possible. The tendency to encode I-markers for words which denote something that can be easily imagined visually (Wvis), should be significantly less in case of visual than in case of acoustical presentation. Similarily, in the case of acoustical presentation the tendency to encode I-markers should be less if the words denote something t h a t can be imagined auditorily (Waud). Abstract words are usually encoded without I-markers - - irrespective of the modality in which they are presented. Therefore it is to be expected that the recall performance for abstract words lies between the performance for the two other classes of words. An experiment was performed to test the two hypotheses which have been described above. Both hypotheses were derived from the assumption of modality-

The Influence of Visual and Auditory Images

215

specific a t t e n u a t i o n b y images. The first hypothesis specifies the effect of a u d i t o r y a n d visual images as distractors during the r e t e n t i o n of items t h a t have been presented visually or acoustically. B y a t t e n u a t i n g the lower-level a u d i t o r y system, a u d i t o r y images m a k e the i n f o r m a t i o n s stored i n t h e a u d i t o r y register u n a v a i l a b l e . Therefore it was expected t h a t with acoustical p r e s e n t a t i o n of items, a u d i t o r y images as distractors, reduce the recency p a r t of the serial position curve more t h a n visual images do ; the p r i m a c y part, however, was n o t expected to be affected differently b y the two i m a g e r y conditions. According to the second hypothesis which has been described above a n i n t e r a c t i o n between the p r e s e n t a t i o n mode a n d t h e m o d a ] i t y of images t h a t are used for encoding t h e words to be r e t a i n e d was expected. The t e n d e n c y to use visual or a u d i t o r y images for encoding the r e t e n t i o n m a t e r i a l can p r e s u m a b l y be varied b o t h b y selection of a p p r o p r i a t e word classes (Wvis, Wau~, Wabstr) a n d b y different modes of p r e s e n t a t i o n (visual or acoustical).

Method

Memory Task The influence of visual and auditory images on the free recall of disyllabic nouns was investigated in a distraction experiment. Series of eight nouns had to be recalled after one presentation. Presentation of items was followed by a retention interval of 10 see duration, during which the distracting images had to be formed. The retention material was presented visually or acoustically. The stimulus material consisted of nouns belonging to one of three word classes (Wvi~,Waua, W~bst,)which differ with respect to the ease and to the sensory mode in which their meaning can be imagined. There were 11 series of eight disyllabic nouns, the first two of which were used to practise the task and hence were dropped from the analysis. The eight words of each series were presented for 0,8 sec per word. Presentation was either visual or acoustical; the words were either projected in capital letters on a screen or were presented by a female voice on tape. The temporal course of one trial was as follows: The presentation of the retention material was preceded by a warning signal; in case of visual presentation the screen was illuminated for 0,5 see, in case of acoustical presentation a tone of equal duration was presented. After the eighth word of a series, another word not belonging to the retention material was exposed for l sec. This word had the function of inducing the distractor imagery; how this was achieved will be described in detail in the next paragraph. To distinguish this word from the retention material of the series it was underlined. The underlined word was succeeded by a time interval of 10 sec duration, the end of which was signalled by a short illumination of the projection screen in case of visual, and by a short tone in ease of acoustical presentation. The subjects were instructed to recall the words of the series thereafter. During the retention interval, the subjects of the imagery groups were instructed to form their images, whereas the subjects of the control groups were instructed to read the underlined word only. After reproduction of the word the subjects of the imagery groups rated the vividness of their imagery on a 6-point rating scale which will be described in the next paragraph. Every subject learned 11 series of words in all but only the results of 9 series will be described below. The first two series were used for practising purposes only. There were three series of words which denoted something that can be imagined visually (Wvu) another group of three series consisted of words which can be imagined auditorily (W~d) and the remaining three series consisted of abstract words (Wabstr). The three series of one word class were always presented one after the other, the succession being identical in all cases. Only the succession of different word classes was varied. The succession of words within every series was also held constant.

Distractor Imagery As already mentioned, an underlined word was exposed after presentation of the retention material. This underlined word was the sign to form a certain image. Each word refers to a certain object or situation which the subjects in the condition I ~ were instructed to imagine

216

H.-G. Bosshardt

visually. Subjects in the condition L=a were instructed to speak the underlined word silently without moving the lips. I t was thereby achieved, that visual or auditory images were produced with the stimulus situation remaining constant. The memory task was preceded by an introductory part in which the subjects learned to form the distracting images. I n the introductory part of the experiment, the word (which was underlined during the memory task) was paired with a photographical picture of its referent object or situation. This enabled the subjects on appearance of the underlined word during the memory task to imagine the picture previously presented together with that word. The pictures represented objects and situations which can be labeled unambiguously by one noun; there was for example a helicopter, a binocular, a high building and so on. After one exposure of 36 pictures, each one being followed by its label, subjects were examined whether they had learned the association between the pictures and their word labels; after that the pictures were presented for the second time, the subjects wrote down the world label and were given feedback as to the correctness of their response. In the introductory part moreover, the production of images was exercised. All subjects irrespective of their membership in one of the experimental groups, practised the production of visual and auditory images. After exposition of one label, subjects h~d to imagine the appropriate picture and gave a sign when the image was as vivid as possible. The vividness of imagery was rated on a six-point-scale reaching from zero (no visual image at all) to five (the image is so vivid as if the picture had been actually presented). This procedure was repeated with five different labels. I n the same way all subjects were trained to hear themselves speak by articulating in their normal rate of speaking the label silently without moving the lips. When the vividness of the auditory image had reached its maximum this vividness had to be rated on a six-point-scale reaching from zero (no auditory image at all) to five (the impression of hearing oneself speak is as clear as though one had actually spoken aloud).

Design The experimental design contained the following five factors: Presentation Mode. The items to be retained were presented visually (Pvis) and acoustically

(Pao). Distractor Imagery. The recall of words presented visually and acoustically was investigated under the influence of visual and auditory images as distractors during the retention interval. Three kinds of distraetor imagery were introduced: Visual imagery (Ivis), auditory imagery (Lu0), and control conditions (C) without the instruction to form images during the retention interval. Word Class. The "imagery value" of the nouns to be remembered was varied as a factor with repeated measurements, i.e. every subject had to remember words which denote something that can be easily imagined visually (WAs), that can be easily imagined in the auditory mode (W~ud), and abstract words (W~bst~) which cannot be imagined at all. Succession o] Word Classes. Variation of the world class as a factor "within subjects" made necessary the variation of the succession of the three word classes; complete permutation of the succession of three word classes gives six different sequences. The sequence of word classes was varied in independent groups of subjects. Serial Positlon. The task of the subjects was to reproduce a series of eight nouns. Serial position was introduced as the second within subjects factor. The factors Presentation mode, Distractor imagery, and Succession of word classes were varied in independent groups of subjects; with three subjects in every cell, there were 08 subjects necessary. The factors Word class and Serial, position were varied within subjects. This design was actually somewhat more complex than described above. The complication resulted from the combination of the memory experiment with another investigation in which the influence of images on perception was investigated (see Bosshardt, 1973). With 72 Ss the succession of the memory and the perceptual task was varied in independent groups. Because this variation had no effect on the results, it was dropped from the design for the last 36 Ss; this latter group of Ss performed the memory task first.

The Influence of Visual and Auditory Images

217

Apparatus Stimuli were presented visually by a projector (Kindermann TC 2000) which was placed in the neighbouring room. Pacing of the visual presentation of stimuli was controlled by a program played from a tape recorder (Revox G36) and a Sono-Dia-Apparatus (Grundig). Acoustical stimuli were played back from the tape and presented by earphones (Sennheiser HI) 44) mounted in an ear protector (GehSrmax D 48).

Retention Material 150 disyllabic nouns were classified preliminarilyby the investigator as belonging to one of the three word classes (W~, Waud, W~bstr). These nouns were then rated by 12 jugdss with respect to the relative difficulty with which their referents can be imagined ~isually, auditorily, and with respect to their concreteness-abstractness. The reliability of these ratings as determined by correlating the mean ratings of two equivalent groups of 6 judges was very high (for details see Bosshardt, 1973, p. t i t ) . Intercorrelations between the three rating scales were computed. Only concreteness and the visual imagery rating are correlated (Spearman) to a significant degree. The correlation is so large that the dimension concrete-abstract can be treated as almost identical with "high visual imagery value-low visual imagery value" (for details see Bosshardt, 1973, p. I l l ) . From this pool of nouns three classes were selected: W ~ = words with the highest visual imagery value and the smallest interindividual variability were selected. These words were concrete words and their referent is very difficult to imagine in the auditory mode. Warstr = These words were judged as being abstract with small interindividual variability. Their reference is very difficult to imagine in the visual or auditory mode. W~ua = Words with the highest auditory imagery value were selected. These words could not be concrete and at the same time have a low visual imagery value, because the two dimensions cannot be varied independently from one another (see above). The words were selected in such a way that they were much more concrete than Wabstralthough, their referent was very difficult to imagine in the visual mode. But, of course, the concreteness of this class of words was less than the concreteness of Wvis.

Results Score was the n u m b e r of items reproduced correctly. Rawscores varied b e t w e e n zero a n d three because t h e y were s u m m e d over t h e three series of a given word class. A r e p r o d u c t i o n was scored correct if the word produced h a d been presented a t a n y of the eight serial positions. A five-factorial analysis of v a r i a n c e (presentat i o n mode, k i n d of imagery, succession of word classes, word class, serial position) with repeated m e a s u r e m e n t s on two factors (word class a n d serial position) was performed. I n c o m p u t i n g the within-subjects variance, the factor "succession of word classes" was dropped from the design. This c o m p o n e n t of v a r i a n c e was e x t r a c t e d only for reasons of reducing the t o t a l error variance. The results of this analysis are shown i n T a b l e I. This analysis presupposes homogeneity of error variances and of covarianees. The homogeneity of the 36 error variances "subjects within groups" cannot be tested in any of the conventional ways because neither the Cochran-test nor the F ~ - t e s t of Hartley (Winer, 1962, p. 93ff.) can be used; the test of Bartlett (Winer, 1962, p. 95ff.) cannot be used because every variance has only two degrees of freedom. Therefore the 36 error terms were pooled over the six levels of the factor "succession of word classes" and the homogeneity of these 12 pooled error terms with 12 degrees of freedom each was tested with the Cochran-test; the null hypothesis cannot be rejected by this test (C (6; 12) = 0,29; p > .05). A second check on the homogeneity assumption was accomplished by pooling the error terms within each level of the factor "succession of word classes"; the

2:18

H.-G. Bosshardt Table t. Analysis of variance of the free recall performance

Source of variance Between Subjects

SS

d]

347,666

107

A (Fresentation mode) 78,125 B (Kind of imagery) 60,441 C' (Succession of word classes) 12,258 Ax B 15,127 A x C' 10,372 B x C' 26,256 A x B × C' 26,135 Subjects w.-gr. :118,952 Within Subjects 2158,:137 C (Word Class) 3,946 AxC 11,445 B ×C i,380 AxB ×C 0,427 C × Subjects w. gr 92,424 D (serial position) 330,525 AxD 106,8:13 B ×D 6,538 Ax B x D 14,347 D x Subjects w. gr. 609,532 CxD 87,770 A x Cx D 16,920 B x CxD 23,050 Ax B x Cx D 26,302 C x D x Subjects w. gr. 826,718

i 2 5 2 5 10 10 72 2484 2 2 4 4 204 7 7 14 :14 714 14 14 28 28 1428

M~

.F

P

78,125 30,221 2,452 7,564 2,074 2,626 2,613 1,652

47,29 18,29 :1,48 4,58 t,25 :1,59 t,58

< .005 < .005

1,973 5,723 0,345 0,t06 0,453 47,217 :15,259 0,467 1,024 0,853 6,269 1,208 0,823 0,939 0,578

4,35 :12,63 .05) and the Barlett-test was computed for the latter (2 = 7,22; d f = 5; p > .05). No test on the homogeneity of covariances was performed because according to Winer (1962, p. 322) the homogeneity of variances "C × D × Subjects within groups" is a partial check on the homogeneity of the covariance terms. Moreover the statistical decisions were identical (with one exception) if the conservative F-test (see Winer, 1962, p. 322) without the homogeneity assumption was carried out. Following the suggestion of Lindquist (1953, p. 78ff.) the normality of the distribution, was checked by inspection. The distributions do not differ in any remarkable extent from the general bell-shape. The assumption of normality therefore holds true.

1. Presentation Mode and K i n d o / I m a g e r y The m a i n effects of p r e s e n t a t i o n mode, k i n d of i m a g e r y a n d their i n t e r a c t i o n were significant. The i n t e r a c t i o n is of central i m p o r t a n c e for t h e theoretical t h o u g h t s presented above (see page - - 3 f . ) : this i n t e r a c t i o n is shown i n Fig. I. B o t h visual a n d a u d i t o r y images as distractors reduce t h e recall p e r f o r m a n c e conside r a b l y i n c o n d i t i o n Pvis. W i t h acoustical p r e s e n t a t i o n (Pae) o n l y a u d i t o r y images

219

The Influence of Visual and Auditory Images 0.7-

0.6

0.5 /

s

s

0A

0.3

[ laud

I Ivis

T C

Fig. 1. Interaction between Presentation mode and Kind of imagery (with probability of recall a~ the ordinate). ~qindicates significant comparisons

reduce the performance as compared to the control conditions (C). For each presentation mode the effect of visual and acoustical imagery was determined by comparing the two imagery groups with their respective controls. The statistical test of Dunnette is appropriate for evaluating the differences of several treatment means in comparison to one control group. The Dunnette4est gives the probability that all decisions about the relation between several treatments and one control group are correct (Edwards, 1968, p. 148). In the present context the critical t for this test is tc = 2,27 (d] = 60; K = 2 ; p < .025; one-tailed probability). Comparison of Pac, Iaud with Pac C gives an empirical t which is higher than tc (t (72) = 2,55) ; the comparison between PacIvis and Pac C is not significant because t (72) = 1,48. Therefore, we can conclude that with acoustical presentation of the items only auditory images reduce the recall performance in comparison to the controll group, whereas visual images don't influence the recall performance. The Fig. l shows that images influence the recall less in the case of acoustical, than in the case of visual presentation of the items. The component of interaction between presentation mode and Iaud and C was computed (see Edwards, 1968, p. 24iZ); this component of interaction is significant (F (l; 102)= 4,72; p < .05). The significance of the other component of interaction (between presentation mode and Iris and C) can be inferred from the fact that the difference between Iris and C is significant with visual presentation of the items but not with acoustical presentation. I t can be concluded that the recall of items presented acoustically was less reduced by images as distractors than the recall of items presented visually. The words presented acoustically were more "resistant" to distractor imagery of whatever kind than the words presented visually.

220

H.-G. Bosshardt 0.7 -

0.6

O.S ~-0- . . . . . . . .

- 0 ~vis

0.~VlS Word

ABSTR class

AUD

Fig. 2. Interaction between Word class and Presentation mode (with probability of recall at the ordinate). ~ indicates significant comparisons

2. The Factor Word Class and Its ]nteractlons The factor word class a n d its interaction with presentation mode reaches the level of significance. The interaction is shown in Fig. 2. The simple effects of the factor word class were c o m p u t e d for each of the two presentation modes separately; only in the case of acoustical presentation does the F-test allow a rejection of the null hypothesis (F (2 ; 204) = t5,6; p < .005) whereas with visual presentation the recall of the three word classes differs only b y chance (F (2; 2 0 4 ) = 1,3; p > .05). W i t h acoustical presentation Wvis was retained better t h a n Wand (t ( 2 0 4 ) = 5,61; p < .005) and t h a n Wabstr (t ( 2 0 4 ) = 3,48; /~ < .005); the a b s t r a c t words (Wabstr) were retained significantly better t h a n Waud (t ( 2 0 4 ) = 2,i3; p < .025). W i t h visual presentation none o f the differences between the word classes reaches t h e 5 % level o f significance; however the differences between Wvis and Wand and between Wvis and Wabstr just fail to be significant (t (204) = 1,36; t (204) = 1,49; p < A0 in b o t h cases). The factor word class has significant interactions with the serial position (C × D) and with the presentation mode and serial position (A × C × D). These interactions cannot be interpreted because there were only three different words of a given class at any of the serial positions. The interpretation of these effects would be highly questionable because, three words are probably not representative for the whole word class. The assumption that these interactions are the result of nonrepresentative sampling at different serial positions was submitted to a further test. To get adequate numbers of words in each part of the list, the lists were divided in two halves; thus, each half consists of t2 different words. The interactions between the factors word class and list halves, and between presentation mode, word class, and list halves are insignificant. Therefore, it is assumed that the significant interactions C × D and A × C × D are the result of inadequate sampling of word classes at different serial positions.

3. The Factor Serial Position and Its Interactions The interaction between serial position and presentation mode is shown in Fig. 3. This p a t t e r n of interaction corresponds exactly to the so-called modalityeffect, usually described in literature (Murdock, i966, 1967b; NIurdock and

The Influence of Visual and Auditory Images

221

1.0-

0.8 ×

0.6

0./.

0.2--

0

]

2

3 /+ 5 6 Seriat position

7

8

Fig. 3. Interaction between Serial position and Presentation mode (with probability of recall at the ordinate)

Walker, 1969): The recency effect is greater with acoustical than with visual presentation whereas no difference is found in the primacy part of the curve. Because the interaction between presentation mode, kind of imagery, and serial position was of central importance in our theoretical reasoning, its insignificance requires to be mentioned. I t has to be concluded that images as distractors, affect only the general level, but not the 8haTe of the position curve.

4. Summary o/the Results a) The recency effect was greater with acoustical than with visual presentation of items, whereas the primacy effect did not differ in the two presentation modes. b) In ease of acoustical presentation the retention was affected less b y images as distractors than it was in the case of visual presentation of words. c) Visual and auditory images as distraetors had no different effect on the retention of words presented visually. In ease of acoustical presentation only auditory images reduce the recall performance to a significant degree; with acoustical presentation no effect of visual imagery was present. d) The differences between the three classes of words was more pronomaeed if the presentation was acoustical than it was if the words were presented visually. With acoustical presentation, recall of Wand was poorest, and of Wvis was best with the abstract word intermediate between the two other classes. With visual presentation there was only a tendency to reproduce less words of the class Wvls.

222

H.-G. Bosshardt

Discussion The present experiment investigated the interaction between several modalitybound processes. The influence of images on the retention of words presented visually and acoustically was predicted on the grounds of certain assumptions concerning the memory effect of presentation mode and of the modality-specific attenuation by images. Therefore it is necessary to discuss both assumptions in the light o£ the results obtained here.

1. Theory o[ the Modality-EBect An interaction between serial position and presentation mode was obtained which is quite similar to the modality-effeet usually found (see e.g. Murdock and Walker, t969) although, in the present experiment, recall had been delayed i i sec after presentation o£ items. Because this interaction was found here with and without distracting imagery this result is at variance with every interpretation o£ the modality-effeet as being the consequence of the different availability of information in the auditory and visual registers at the time of recall. The complete absence of any differences in the shape of the serial position curves under the control and imagery conditions, indicates two very important points: a) At the time of the onset of imagery (about 2 sec after presentation of the last item) there was no more information available in the auditory register. Otherwise it would be expected that under control conditions this information was stored in some more permanent form during the retention interval; the result would be a higher recency part in the conditions PaoC than in any of the other two conditions (Pac Iaud, Pac Iris). That is not the case and therefore it must be concluded that at the time of onset o£ imagery no information is stored in the auditory register, b) The different susceptibility of different parts of the serial position curve to distractor activities has recently been interpreted by Glanzer (1972) by identifying the recency part with the performance of the STS and the primacy part with LTS. In the present experiment both kinds of imagery distractors reduce homogeneously the general level of the serial position curve. Therefore it is assumed that all parts of the serial position curves obtained here are the product of the same memory system. Retention in the present experiment is assumed to be primarily a product of STS because forgetting generally was produced by the distractor task. The modality-effect has to be explained as being the result of processes in the STS which differ, as a consequence of different presentation modes. Two effects of the presentation mode are to be explained: a) Comparing imagery with control groups, we find that distractor imagery o£ whatever kind reduced the recall of all items presented visually more than the recall of words presented acoustically. With acoustical presentation all items from all serial positions are more resistent to distracting imagery than are the items presented visually. b) Comparing visual with acoustical presentation, we find that the superiority of the acoustical presentation is confined to the last three serial positions. The first result was interpreted by differences in the rate of forgetting caused b y encoding the items presented visually and acoustically with different redundancy

The Influence of Visual and Auditory Images

223

in the STS. A certain number of cues is minimally necessary for retrieval of every item. Items for which more cues are encoded are forgotten more slowly even if the decay rate for every single cue is constant (compare Bower, 1967). The rapidly decaying visual register is supposed to effect a nonredundant encoding of the items in the STS. This nonredundant encoding of items in the STS requires continuous processing of the material, in very short cycles in order to keep available the cues minimally necessary for retrieval. With acoustical presentation on the other side the items are encoded in a relatively redundant way in the STS. This information can therefore be recycled in relative long and variable intervals of time - - it requires less processing to be held available as compared to the items which have been presented visually. Therefore a distractor activity reducing the processing capacity available for storage, affects the retention of words more, if they have been presented visually than if they have been presented acoustically. Turning now to the second result (b) mentioned above (p. 322) interpretation of the interaction between serial position and presentation mode can be put forward by unfolding the consequences of this modality-dependent differences in encoding. During presentation of a list of words there is less time available for encoding the items later in the list because the words presented earlier require some time for their rehearsal. This reduction of the time for encoding the items later in the list results in a less redundant encoding of the later items as compared to the first items of the list. There is always a gradual decrease in the redundancy of encoding from the beginning to the end of the list. Taking this together with the assumption that there is an over-all difference in the redundancy of encoding the words presented visually and acoustically the interaction between serial position and presentation is easily interpreted. At any of the later serial positions there is comparatively more time necessary for processing the items earlier in the list if the presentation is visual. There is a gradual reduction of the encoding redundancy from the beginning to the end of the list; this gradient, however, is steeper in case of visual than in case of acoustical presentation. With visual presentation the last items of the list have, therefore, a greater probability of being forgotten before a new rehearsal has taken place. The results which have been presented above led to an interpretation of the effect of presentation mode on the recall performance which differs in one import a n t respect from the interpretation conventionally put forward: The modalityeffect is interpreted as a consequence of encoding differences in the STS and not as a consequence of the different availability ofinformations in the sensory registers at the time of recall. 2. The E#evt o/Images as Distractors I t has been shown in the last paragraph that the information in the auditory as well as in the visual register is completely decayed at the time when the imagery begins. Therefore the theoretical basis for interpreting the effect of distractor imagery on the recall has changed considerably as compared to expectations before completion of the experiment. The effect of images on the recall performance cannot be interpreted b y the assumption of modality-specific attenuation b y images. I t is impossible to test this assumption if there is no information stored in the sensory registers at the time of imagery.

224

H.-G. Bosshard~

The modality-specific effect of images on tim recall performance cannot be accounted for by the reduction of processing capacity by images. Auditory images reduced the recall performance of items presented acoustically but the visual images did not; with visual presentation the two kinds of images did not have a different effect. This result might be interpreted by assuming qualitatively different encoding of items presented visually and acoustically. Some experiments of Murray (i967, 1968) and Peterson and Johnson (1971) show that the encoding of acoustical retrieval cues in the STS, is especially important in the case of acoustical presentation. Therefore, it has to be assumed that independent of the quantitative differences in the number of retrieval cues encoded, the proportion of auditory cues is greater in case of acoustical presentation than it is in case of visual presentation. Auditory images were produced in this experiment by silently articulating the to-be-imaged word label. Because of the close interrelationship between the verbal-auditory and the verbal-articulatory processes, it can be supposed that this activity interferes particularly with the rehearsal of auditory cues for the memory items. This group of cues however is of special importance for the retention of items that have been presented auditorily.

3. The Word Classes The memory material consisted of nouns from three distinct classes : the first class of nouns denoted something that can be easily imagined in the visual mode (Wvis) ; the referent of the second class of words could be easily imagined in the auditory mode (Wau~) and the third class consisted of abstract nouns (Wabstr) the referent of which was very difficult to imagine. The ease with which the referent of a word can be imagined was assumed to determine the encoding of this word. The word classes differ presumably in the extent and kind of I-markers that are encoded. However, the encoding of visual or auditory I-markers, cannot de determined by selection of words alone. On grounds of the assumption of modalityspecific attenuation by images, an interaction between the factors "word class" and "presentation mode", was expected. With acoustical presentation, the empirical results were in complete agreement with the theoretical expectations. With acoustical presentation the encoding of visual but not of auditory I-markerS is possible. Accordingly, the words Wvis were retained better than Wabstr whereas the retention of Waud was less than that of Wabstr. Comparing the two presentation modes, the differences between the three word classes were significantly smaller when the presentation was visual. This result agrees quite well with the assumption of a less redundant encoding of items which have been presented visually. With visual presentation there is less time for encoding. The encoding of semantical informations and I-markers, however, requires considerable time (compare Moore, i9i5; Shnlman, 197i, i972 et al.). Encoding of items presented visually therefore, lacks particularly semantic and I-markers. In the terms of Craik and Lockart (1972) one would say that the "depth of encoding" is less with visual than with acoustical presentation. The results show that the use of I-markers is a joint effect of the kind of word to be retained and of its presentation mode. This holds true especially in case of

The Influence of Visual and Auditory Images

225

a c o u s t i c a l p r e s e n t a t i o n . B u t t h e r e are some hints as to t h e correctness o f this s t a t e m e n t in case o f visual p r e s e n t a t i o n t o o ; t h o u g h insignificant t h e r e is nevertheless, a t e n d e n c y to recall less words Wvis t h a n Wabstr i f t h e p r e s e n t a t i o n is visual. This agrees g e n e r a l l y w i t h t h e a s s u m p t i o n t h a t i m a g e s a t t e n u a t e t h e lower.level processing u n i t s ; i f t h e sensory m o d e of t h e i m a g e a n d o f t h e i t e m p r e s e n t a t i o n is identical, this a t t e n u a t i o n is p r e v e n t e d and, hence, no images can be p r o d u c e d .

References Atkinson, R. L., Shiffrin, R. ~ . : Human memory: A proposed system. In: The psychology of learning and motivation, vol. I I (eds. K. W. Spence, J. T. Spenee), pp. 89--t95. New York, London: Academic Press 1968 Bosshardt, H.-G.: Der EinfluB yon Vorstellungen auf Wahrnehmen und Behalten. Unver6ffentl. I)iss. Bochum 1973 Bower, G. H.: A multieomponent theory of the memory trace. In: The psychology of learning and motivation, vol. I (eds. K. W. Spenee, J. T. Spence), pp. 230--327. New York, London: Academic Press 1967 Broadbent, I). E.: Flow of information within the organism. J. verb. learn, verb. behav. 2, 34--39 (1963) Broadbent, I). E.: Decision and stress. London: Academic Press 197t CorbaIlis, M. C.: l~ehearsal and decay in immediate recall of visually and aurally presented items. Canadian J. Psychol. 20, 43--51 (1966) Corballis, M. C., Loveless, T. : The effect of input modality on short-term serial recall. Psyehon. Sci. 7, 275--276 (t967) Conrad, 1~., Hille, B. A.: The decay theory of immediate memory and paced recall. Canadian J. Psyehol. 12, 1--6 (1958) Craik, :F. I. ]~¢[.,Loekart, 1~. S.: Levels of processing: A framework for memory research. J. verb. learn, verb. behav. 11, 67t--684 (i972) Crowder, 1~. G., ~lorton, J.: Preeategorieal acoustic storage (PAS). Pere. Psyehophysies 5, 365--373 (1969) I)ornbush, R. L. : Input variables in bisensory memory. Pere. Psyehophysies 4, 41 --44 (t968a) I)ornbush, 1~. L.: Shadowing in bisensory memory. Quart. J. exp. Psyehol. 20, 225--231 (1968b) Edwards, A. L.: Experimental design in psychological research. New York: Hult, l~inehar~ & Winston 1968 Glanzer, M. : Storage mechanisms in recall. In: The psychology of learning and motivation, vol. 5 (ed. G. H. Bower), pp. 129--t94. New York, London: Academic Press 1972 German, A. ~ . : l~ecognition memory for nouns as a function of abstraetness and frequency. J. exp. Psychol. 61, 23--29 (i961) Hebb, I). O.: Concerning imagery. Psychol. Rev. 75, 466--477 (1968) Kerr, B.: Processing demands during mental operations. Memory and Cognition 1, 401--412 (1973) Kintsch, W. : Models for free recall and recognition. In: Models of human memory (ed. I). A. Norman), pp. 339--374. New York, London: Academic Press i970 Lindquist, E.: Design and analysis of experiments in psychology and education. Boston: Houghton Mifflin 1953 Mackworth, J. F.: Presentation rate, repetition, and organization in auditory short-term memory. Canadian J. Psychol. 19, 304--315 (1965) Moore, T. V. : The temporal relations of meaning and imagery. Psyehol. l~ev. 22, 177--225 (1915) ~oray, N.: Attention. Selective processes in vision and hearing. New York: Hutchinson 1970 Morton, J.: A functional model for mcmory~ In: models of human memory. (ed I). A. Norman), pp. 203--254. New York, London: Academic Press 1970

226

H.-G. Bosshardt

l~Iorton, J., Holloway, L. M.: On the absence of cross-modal suffix effects. Quart. J. exp. Psychol. 22, 167--t76 (1970) Murdock, B. B., Jr.: Visual and auditory stores in shore-term memory. Quar~. J. exp. Psychol. 18, 206--2tl (1966) Murdock, B. B., Jr.: Auditory and visual stores in short-term memory. Acta psychol. (Amst.) 27, 316--324 (1967a) Murdock, B. B., Jr.: Distractor and probe techniques in short-term memory. Canadian J. Psychol. 12, 25--36 (t967b) 1V[urdock, B. B., Jr. : Serial order effects in shor~derm memory. J. exp. Psychol. 76 (~onogr. Suppl. 4, par~ 2), (1968) Murdock, B. B., Jr. : Where or when: Modality effects as a function of temporal and spatial distribution of information. J. verb. learn, verb. behav. 8, 378--383 (1969) Murdock, B. B., Jr., Walker, K. D.: !Viodality effects in free recall. J. verb. learn verb. behav. 8, 665--676 (1969) Murray, D. J. : The role of speech responses in short-term memory. Canadian J. Psychol. 21, 263--276 (t967) Murray, D. J. : Articulation and acoustic eonfusability in short-term memory. J. exp. Psychol. 78, 679--684 (1968) Murray, D. J., Roberts, B. : Visual and auditory presentation, presentation rate, and shortterm memory in children. Brit. J. Psychol. 59, t19--125 (1968) Neisser, U. : Cognitive Psychology. New York: Appleton-Century-Crofts 1967 Norman, D. A., Rumelhart, D. E.: A system for perception and memory. In: Models of human memory (ed. D. A. Norman), pp. 21--66. New York, London: Academic Press 1970 Paivio, A. : Abstractness, imagery, and meaningfulness in paired-associate learning. J. verb. learn, verb. behav. 4, 32--38 (1965) Paivio, A.: Latency of verbal associations and imagery to noun stimuli as a function of abstractness and generality. Canadian J. Psyehol. 20, 378--387 (1966) Paivio, A.: Paired-associate learning and free recall of nouns as a function of concreteness, specificity, imagery, and meaningfulness. Psyehol. Rep. 20, 239--245 (i967) Paivio, A. : A factor-analytic study of word attributes and verbal learning. J. verb. learn, verb. behav. 7, 41--49 (1968) Paivio, A. : Mental imagery in associative learning and memory. Psyehol. Rev. 76, 241--263 (t969) Paivio, A.: On the functional significance of imagery. Psychol. Bull. 73, 385--392 (~I970) Paivio, h.: Imageryand verbal processes. New¥ork, Chicago: Holt, Rinehart & Winston 1971 a Paivio, A.: Imagery and language. I n : Imagery (ed. S. J. Segal), pp. 9--35. New York, London: Academic Press 1971b Paivio, A., Csapo, K. : Concrete image and verbal memory codes. J. exp. Psychol. 80, 279--285 (1969) Paivio, A., Yuille, J. C., Rogers, T. B. : Noun imagery and meaningfulness in free and serial recall. J. exp. Psyehol. 79, 509--514 (i969) Peterson, L. R., Johnson, S. T.: Some effects of minimizing articulation in short-term retention. J. verb. learn, verb. behav. 10, 346--354 (197t) Posner, M. I.: Short-term memory systems in human information processing. Aeta psychol. (Amst.) 27, 267--284 (1967) Posner, M. I., Konick, A. ~.: On the role of interference in short-term memory. J. exp. Psyehol. 72, 222--231 (t966) Posner, M. I., Rossman, E.: Effect of size and location of informational transforms upon shortterm retention. J. exp. Psychol. 70, 496--505 (1965) Rohwer, W. D., Jr. : Images and pictures in children's learning: Research results and instructional implications. Psychol. Bull. 73, 393--403 (~1970)

The Influence of Visual and Auditory Images

227

Sanders, A. F. : Rehearsal and recall in immediate memory. Ergonomics 4, 25--34 (i96t) Shuhnan, It. G. : Similarity effects in short-term memory. Psychol. Bull. 75, 399--415 (1971) Shulman, H. G. : Semantic confusion errors in short-term memory. J. verb. learn, verb. behav. 1:1, 221--227 (t972) Wickelgren, W. A.: Size of rehearsal groups in short-term memory. J. exp. Psyehol. 68, 413--419 (1964) Winer, B. J. : Statistical principles in experimental design. New York: Melgraw-Hil11962 Dr. H.-G. Bosshard~ Psychologisches InstRut Ruhr-Universit~it D-4630 Bochum Postfach 2148 Federal Republic of Germany

The influence of visual and auditory images on the recall of items of visual and auditory presentation mode.

Psychol. Res. 37, 211--227 (1975) © by Springer-Verlag 1975 The Influence of Visual and Auditory Images on the Recall of Items of Visual and Auditory...
1MB Sizes 0 Downloads 0 Views