VisionRes. Vol. 31, No. 9, pp. 15414548, 1991 Printed in Great Britain. All rights rewvcd
copyright 0
0042-6989/91 $3.00 + 0.00 1991 Pergamon PreS plc
TEMPORAL DIFFERENCES BETWEEN COLOR PATHWAYS WITHIN THE RETINA AS A POSSIBLE ORIGIN OF SUBJECTIVE COLORS* S. M. COURTNEY and G. BUCHSBAUM Department of Bioengineering, School of Engineering and Applied Science, University of Pennsylvania, 220 South 33rd Street, Philadelphia, PA 19104, U.S.A. (Received 11 July 1990; in revised form 2 January 1991) Abstract-We propose a model of temporal signal processing within the retina based on temporal differences between the color pathways which may explain the phenomenon of subjective color. We quantify the model by inferring impulse response functions from physiological data, and predict the output of the different color pathways to temporally modulated achromatic signals which produce the sensation of color. Certain achromatic temporally modulated signals create imbalances between the color pathways which are analogous to those produced by stationary chromatic signals.
INTRODUCTION
For more than a century and a half it has been known that intermittent white light can, under certain conditions, produce the sensation of color. In 1826, Benedict Prevost first observed these so-called “subjective colors” by moving a piece of cardboard in front of a white light in a darkened room. Later, in 1838, G. T. Fechner rediscovered subjective colors when he spun a black and white patterned disk below the luminance fusion frequency. There are many ways of producing color with achromatic stimuli. While spinning patterned disks is the easiest and most common method of producing the color sensation, no movement across the retina is necessary, only temporal luminosity modulation (Motokawa & Ebe, 1952; von Campenhausen, 1968; Festinger, Allyn & White, 1971; Muller, Korth, Rix & Weimer, 1980). In addition, a spatial pattern is not necessary. Many researchers demonstrated that color could be induced by varying only the temporal and luminance contrast properties of the signals (see Cohen & Gordon, 1949 for an extensive review of early work; Stewart, 1924). More recently, Festinger et al. (1971) have found spatially homogeneous, temporally modulated signals which produce a variety of color sensations (see Fig. 1). Subjective colors provide a useful probe into the visual system (von Campenhausen, *This paper was presented at the 1990 ARVO meeting. va
31/9-F
1973; Both & von Campenhausen, 1977). This phenomenon demonstrates an essential nonlinearity in the transformation of visual stimuli in which spatial, temporal, and chromatic information interact. Yet, despite its long history, there is no satisfactory quantitative explanation for subjective colors. Temporal differences have been suggested before as an explanation for subjective colors, originally by Fechner in 1838. Stewart, in 1887, proposed four methods of incorporating different delays, rise times, and fall times, into the three color systems. In 1922, Pieron formulated a theory, similar to one of Stewart’s cases, which depends upon the timeto-peak of each of the receptor types (see Cohen & Gordon, 1949; Stewart, 1924). These explanations require temporal differences between the different color pathways which are much larger (differential latencies on the order of 75 msec) than has been found by physiological or psychophysical studies. We propose a three pathway, color-opponent model for generating subjective colors which requires much smaller temporal differences than Fechner’s and Pieron’s theories. From physiological data, we infer impulse response functions and nonlinearities for the three most common wavelength selective on-center parvocellular ganglion cell types. Using these impulse functions, we calculate the outputs of these ganglion cells to the temporally modulated achromatic signals used by Festinger et al. (1971) to produce subjective colors. The model demonstrates how these signals can create an imbalance in the
1541
1542
S. 60
M. COURTNEY
and G.
BUCHSBAUM
“Neutral”
‘Red”
50
4
40 30 20 g z
10
5 =.
0
I
I
I
:: T x ‘Z
60-
“Green”
E 50
‘ii 3
40
0
I
I
1
50
100
150
0
I
I
I
50
100
150
Time (msec) Fig. 1. Achromatic temporally modulated signals used by Festinger et al. (1971) to elicit the indicated subjective color sensations.
Cones
Opponency
Ganglion cells
Higher stages
~~F===&EJ
.
Fig. 2. An overview of the model showing the stages of calculation. The impulse functions are de&d at the ganglion cell level, but the temporal differences may accumulate through the earlier stages. The receptive fields have inhibitory inputs (dashed lines) from all cone types, although some may have a greater weight than others as indicated. Saturation and rectification occur at the ganglion cell level, before the responses are integrated and compared.
A possible origin )f subjective colors
responses of the different color pathways when integrated over time. This imbalanced response to the temporally modulated achromatic signals is analogous to the response that would occur if the cells were presented with stationary chromatic stimuli. DEVELOPMENT
OF THE MODEL
An overview of the model is presented in Fig. 2, showing the stages of calculation. Impulse functions are defined at the ganglion cell level, however, the temporal differences among the pathways may accumulate in the preceding layers. The impulse function is convolved with temporally modulated signals to predict the temporal response profile. Specific nonlinearities are taken into account, and then the response is integrated over time, normalized, and compared to the responses in other pathways. In the model, the abbreviations “L+/M-“, “M+/L-“, and “S+/(L + M)-” color pathways refer to those pathways in the
1.0
impulse functions
3 .= r 3 2 m
visual system which include ganglion cells whose receptive fields are excited by cones which are most sensitive to the long (L), middle (M), or short (S) wavelength regions of the spectrum respectively. The terms “on-response*’ and “off-response” indicate the output of the ganglion cells in response to an increment and a decrement in luminosity respectively. We used the following findings to conceptualize and constrain the model. Zrenner and Gouras (1981, Fig. 3) recorded from retinal ganglion cells of rhesus monkeys while stimulating the center only of the receptive field of each of these cells. They found marked differences between the temporal response profiles of the L +/Mm ganglion cells and those of the S+/(L + M)- ganglion cells. The L+/Mcells appear linear and we may determine their impulse function directly. However, the S+/(L + M)- cell responses have two nonlinearities which are important in the present context: (1) there is an approx. 30 msec delay in the on-response, while there is no observable
kt+ii
L+IM‘
1.5
1543
s+/(L+Mj
II
r
OS
--
0 I
I
0
L
100
80
aa
.r
';I; -_
lz Square responses
60 c
---40
I
20
0
400
Time (msec) Fig. 3. In the bottom row are the responses of each color pathway to a 2OOmsecsquare pulse of light. The solid line is the total response while the dashed line shows the response if only the center of the receptive field is stimulated. The center responses follow the profile of Zrcnncr's (1983) intracellular recordings. In the top row are the impulse functions used in the model which were calculated so as to produce the response pro&s below when convolved with a square pulse of 200 msec. The dashed line in the S+/(L + M)- impulse function graph is the impulse function for the off-response. The solid line for the S+/(L + M)- impulse function is for the on-response only.
S. M. G~JR~NEY and G. BUCHSBAUM
1544
delay in the off-response; (2) the slope of the off-response is markedly different from that of the on-response. Zrenner (1983, Fig. 5.1) also measured critical flicker frequencies electrophysiologically in rhesus monkey parvocellular ganglion cells and found that, in the fovea, the M+/Lpathway had the highest temporal resolution, followed by L+/M- (about 20% slower) and then S+/(L + M)- (about 30% slower than the M +/L - ). Direct recordings of impulse functions have not been able to confirm these differences between the pathways due to high cell to cell variability and noise problems. Schnapf, Nunn, Meister and Baylor (1990) calculated the time-to-peak and the integration times of several macaque cones. The mean values and standard deviations were: Conetype Time to peak (msec) Integration time (msec) M
51* 13
L
55 + 18
s
61 f 1
19 + 10 28 f 14 34+ 17
While these temporal differences between cone types may not appear significant due to the large standard deviations, the trend of increasing temporal resolution, from S through L to M, agrees with that found by Zrenner. As we will show, the differences required by the model are small enough to be in reasonable agreement with these data. Because Zrenner’s data is based on stimulation of the center only of the ganglion cells’ receptive fields, we must add to these responses the effects of a subtractive surround. Because input to the surround appears to come from all cone types, the same temporal properties were used for the surrounds of all ganglion cell types. Lee, Martin and Valberg (1989) measured from macaque retinal ganglion cells and found center-surround delays of 3-8 msec for both L+IMand M+IL- cells. We have used a delay of 6 msec for all three color pathways. Variations in the center-surround delay of + 2 msec do not change the output of the model significantly. We used these findings to infer temporal responses for the three color pathways defined here including impulse responses which incorporate the differences in temporal resolution by using slightly wider impulse functions for the slower pathways (see Fig. 3). The S+/(L + M)pathway is the slowest (Zrenner & Gouras, 1981). In the model we use an impulse function for this pathway with a full width at half maxi-
mum of 5.3 msec. As indicated above there is evidence which suggests a slightly higher temporal resolution for the M+/L - pathway than for the L+/M- pathway. In the model, we postulate a difference in impulse response width of 0.8 msec in full width at half maximum. The full widths of the L+lM- and M+IL- pathway impulse responses are 3.2 and 2.4 msec respectively, implying a 25% difference. The model impulse function is for an ideal (delta function) impulse. Physiologically measured impulse responses depend on the width of the stimulus impulse used and on the average luminance level (Gielen, van Gisbergen & Vendrick, 1982; Purpura, Tranchina, Kaplan & Shapley, 1990; Lee, private communication). The time delay and nonlinearity of the S+/(L + M)- pathway are considered by using a different impulse function for the off-response than for the on-response. For the S+/(L + M)pathway, the slope of the off-response is different from that for the on-response and the offresponse does not have the 30 msec delay that the on-response does. This is unlike a linear system which would require the same temporal characteristics in both off- and on-responses except for a sign change. Figure 3 shows the impulse functions and square pulse responses (for both total response and for center only stimulation) for the three color pathways. The S+/(L + M)and L +/M - impulse response shapes were chosen so that the square pulse response of the center mechanism follows the response profile reported by Zrenner and Gouras (1981, Fig. 3). It is reasonable to ask if saturation, and rectification from the finite level of spontaneous activity, play a role in determining the response profiles of the ganglion cells under the normal viewing conditions of subjective colors. Saturation and rectification effects have been demonstrated in electrophysiology in cones, ganglion cells, and the LGN (Schnapf et al., 1990; Purpura, Kaplan & Shapley, 1988). The results presented here were calculated using a single stage of saturation taken after the convolution of the input with the impulse functions and after the subtraction of the background level due to surround inhibition. This would correspond to saturation at either the ganglion cells or the LGN. Adding other stages of saturation, such as before the subtraction (at the individual cone level), can enhance the imbalances to certain signals. The cut-offs introduced by the rectification and saturation constitute another nonlinearity in the model.
A possible origin of subjective colors RESULTS
By convolving the impulse response functions shown in Fig. 3 with any arbitrary temporal input signal and considering the nonlinearities discussed above one may predict the output of the three color pathways at the ganglion cells. Figure 1 shows the temporally modulated achromatic sequences used by Festinger et al. (1971) to elicit various color sensations. The outputs, predicted by the model, for these signals are shown in Fig. 4. Festinger’s observations indicate that the colors perceived with a single cycle of the stimulus are approximately the same as those seen for multicycle signals. Therefore, the calculated responses shown here are for single cycles in the interest of clarity. We integrated the calculated response profiles over 150 msec. This covers the time for one full cycle plus sufficient time without any stimulus for the outputs to return to near zero. We thus obtained the area under each curve. (Integrating over a longer period of time, therefore, would
120 r 100
1545
not change the results.) This integration time is consistent with chromatic pulse threshold measurements, which indicate an integration time for color information from 160 to 320 msec depending upon mean retinal illumination (Swanson, Ueno, Smith & Pokomy, 1987). The areas under the response profiles were then normalized so that the area under each curve for the stimulus which appeared neutral was equal to unity. Any deviation from neutral, such as that caused by a physically “colored” stimulus differentially exciting the cone pigment types, would cause an imbalance in these normalized responses. The normalized areas were compared by calculating the fraction of the total area of all three pathway responses for each color over one cycle. For example: Fraction of activity that is due to the L +lM- pathway = A&A, + A,,, + A,) where A,, A,,, and A, refer to normalized areas under the L+/M-, M+/L- and S+/(L +M)response profiles respectively (see Fig. 5). “Red”
“Neutral”
L
80 80
“Green”
.g 100
5 iii
80 80 40
“BlU8”
ILL
R
Time (msec) Fig. 4. Responses of each of the color pathways to Festinger et al.3 achromatic signals as predicted by the model. The solid line is the response of the L+/M- pathway, the long dashed line represents the M+/L- response,and the short dashed line shows the P/(L + M)- response. In the inset of each set of responses is the associated input signal which is also shown in Fig. 1.
1546
S. M. COURTNEYand G. BUCHSLMJM 60 0
L+IM-
M+/i
50
n
s+/(L+ M)‘
li; 5 40
0
“Neutral”
“Red’
“Green” Stimulus
‘Blue”
-
name
Fig. 5. Ratios of the normalized area for each pathway to the sum of the areas for all three pathways are given for each of Festinger et al.3 achromatic input signals. The name of the color which resembles that induced by the signal is indicated below each set of responses.
We considered the composition of the center and surround areas of ganglion cell receptive fields in interpreting the relationship between the relative integrated responses in the three ganglion cell types. Lennie (1980) suggests that, at least in the fovea, color opponency in the ganglion cells is a consequence of the center containing a single cone, while the surround is made of a combination of cone types. The more sharply tuned centers, therefore, are paramount in mediating color information through the retina. For example, a stationary, spatially homogeneous, long-wavelength stimulus will produce a proportionately greater normalized response in the L +/AC retinal pathway than in the other on-center pathways (an imbalance compared to the response to a neutral stimulus). This is interpreted through further processing in higher stages which leads to the perception of “red”. We expect that such an imbalance must occur for the corresponding temporally modulated achromatic signal inducing a “red” subjective color sensation. Figure 5 shows that the model predicts an imbalance among the different color pathways which agrees with the imbalance expected from the color sensation induced by each signal used by Festinger et al. (1971). The psychophysical results with stimuli intended to produce cyan and magenta sensations were not consistent between observers. We were also not able to
produce a significant imbalance for these two signals using the present model. The color names given to each signal follow those in the experiment (Festinger et al., 1971) and are general comparative descriptions only; the colors perceived are very unsaturated. With all the present physiological uncertainties and the qualitative nature of the phenomenon we do not attempt to determine the chromaticity coordinates of or otherwise quantify the actual colors here. The signals used by Festinger et al. (1971) to induce subjective color had a contrast of approx. 74%. Contrast is computed as: (L,A -
Lback~ound)/(Lpcak
+
Lbackground).
The level of hard-limiting saturation was chosen such that signals of more than 60% contrast would have some saturation effect. The model is sensitive to this parameter to within 3% of this value. The model also assumes a spontaneous activity level of 5% of the saturation level. While rectification is required by the model, it is not sensitive to variations of up to 40% of this value. However, it is difficult to compare these levels with physiological data for several reasons. Saturation and spontaneous activity levels vary with background and adaptation conditions (Zrenner, 1983; Purpura et al., 1988). Both are lower in the LGN (saturation at 2040% stimulus contrast) than in retinal
A possibleorigin of subjectivecolors ganglion cells (no saturation until 60% contrast) (Kaplan, Purpura & Shapley, 1987). The sensitivity of the model to the level of saturation and the dependence of the cut-off on average luminance and contrast indicates that not all such temporally modulated signals will produce subjective colors (Festinger et al., 1971; Jarvis, 1977). If the contrast is too low, the signals will not saturate and the effect of this nonlinearity will be lost. In addition, at lower luminance levels the impulse functions become wider, diminishing the relative significance in the model of the small difference between the L+lM- and M+/L- responses. On the other hand, if the contrast is too high, the output signals will oversaturate, cutting off the peaks of the responses where most of the differences occur. This seems to be the case in the Festinger et al. first attempt at producing subjective colors in which the signals, which peaked at 5000 td, were shown against a very dark background (near 100% contrast). Under such conditions, no colors were seen, only “very bright lights”. Such high contrast may also be the reason that Jarvis (1977) did not see color when using temporally modulated signals of 170-4600 td, a contrast of over 96%.
CONCLUSION We have shown that an integrative model, which incorporates the entire temporal response profiles (including time delays and nonlinearities) of the three color pathways defined by ganglion cells, provides a possible explanation for subjective colors induced by spatially homogeneous, temporally modulated achromatic signals such as those used by Festinger et al. (1971). The concept of temporal differences explaining this phenomenon is not a new one (Fechner, Pieron in a review by Cohen & Gordon, 1949; Motokawa & Ebe, 1952) and is considered to be reasonable within the context of the visual system (Boynton, 1979). The earlier models, however, required temporal differences of the order of 75 msec between color pathways. The need for this unreasonably large time delay arose because the earlier models considered only temporal delays between peak responses of the color pathways. In the present explanation, however, we propose a different method of incorporating temporal differences which requires smaller, more physiologically plausible differences. The present model integrates and
1547
compares the entire temporal response, including saturation and rectification, not simply its peak. Small differences in the responses are integrated over time and accumulate considerable effects. Additionally, the present analysis considers the subtractive effect of the inhibition which makes the model more sensitive to small differences in the temporal responses of the three pathways. The temporal parameters in the model are based on data from Zrenner (1983) and Zrenner and Gouras (198 1) recorded from ganglion cell center mechanisms. In particular, these include the profiles of the responses of the L+/M- and S+/(L + M)- pathways and the nonlinearities of the S+/(L + M)pathway. The faster response in the M+/L- pathway, although not shown conclusively by the data, is at least consistent with physiological measurements of flicker frequency (Zrenner, 1983) and with recordings from cones (Schnapf et al., 1990). To the best of our knowledge no physiological study is available which directly compares temporal response profiles in the retina with great enough precision to confirm this small difference. Further refinement of the model is limited by the unsaturated and sometimes elusive nature of subjective colors. The signals used in this simulation, in order to produce the proper imbalances, require a particular range of average luminance, contrast, and temporal frequency. Some signals which produce subjective colors require a two-part signal, each part shown to adjacent areas of the retina. For example, the well known Benham’s Disk (Benham, 1884, 1885) may be simulated with a small center area which varies between white, grey, and black, and a surrounding area which alternates between white and black at the same frequency as the center signal. The relative phase between the two signals can alter both the perceived chromaticity and luminance of the center signal (Festinger et al., 1971). Clearly spatial interactions are important here as well as temporal processing. However, more information about the necessary sizes and spatial configurations of the two signals as they relate to the sizes of ganglion cell receptive field centers and surrounds is necessary. Further investigation of the interaction of ganglion cell centers and surrounds and their retinal circuitry, as well as more precise recordings of temporal properties of cells within the retina, will enable such refinement.
1548
S. M. COURMEYanId G. BIJCHSBAUM
Acknowledgemenr-We
thank Dr Peter Sterling for his
helpful criticism.
and phase of responses of macaque retinal ganglion cells to flickering stimuli. Journal of Physiology, London, 414, 245-263.
Lennie, P. (1980). Parallel visual pathways: REFERENCES Benham, C. E. (1884). The London, 51, 200. Benharn, C. E. (1885). The London, 52, 321. Both, R. 8cCampenhausen, change pattern induced
artificial spectrum top. Nature, artificial spectrum top. Nature,
A review.
Vision Research, 20, 561-594.
Lennie, P. & D’Zmura, M. (1988). Mechanisms of color vision. CRC Critical Reviews in Neurobiology, 3, 333-340. Motokawa, K. & Ebe, M. (1952). Retinal colour processes caused by intermittent white light. Nature, London, f70, 79-80.
C. von (1977). Short time delays flicker colors (PIFCs). Vision
Research, 17, 651-660.
Boynton, R. M. (1979). Human color vision (pp. 327-333). New York: Holt, Rinehart & Winston. Campenhausen, C. von (1968). Uber die farbcn der Benhamschen scheibe. Zeitschrifrfur verleichende Physiologic, 60, 351-374.
Campenhausen, C. von (1973). Detection of short time delays between photic stimuli by means of pattern induced flicker colors (PIFCs). Vision Research, 13, 2261-2271.
Cohen, J. & Gordon, D. A. (1949). The PrevostFechner-Benham subjective colors. Psychofogical Bulletin, 46, 97-136.
Festinger, L., Allyn, M. R. & White, C. W. (1971). The perception of color with achromatic stimulation. Vision Research, II, 591-611. Gielen, C. C. A. M., Gisbcrgen, J. A. M. van & Vendrik, A. J. H. (1982). Reconstruction of cone-system contributions to responses of colour-opponent neurones in monkey lateral geniculate. BioZogical Cybernetics, 44, 211-221.
Jarvis, J. R. (1977). On Fechner-Benham subjective colour. Vision Research, 17, 445-45 1. Kaplan, E., Purpura, K. & Shapley, R. M. (1987) Contrast affects the transmission of visual information through the mammalian lateral geniculate nucleus. Journal of Physiology, London, 391, 267-288. Lee, B. B., Martin, P. R. & Valberg, A. (1989). Amplitude
Miiller, G., Korth, M., Rix, R. & Weimer E. (1980). Information coding in the visual system: A new hypothesis. Aibrechr von Graefes Archiv fur Klinihale Oplhalmologie 214, l-7.
Purpura, K., Kaplan, E. & Shapley, R. M. (1988). Background light and the contrast gain of primate P and M retinal ganglion cells. Proceedings of the National Academy of Sciences, U.S.A., 85, 4534-4537. Purpura, K., Tranchina, D., Kaplan, E. & Shapley, R. M. (1990). Light adaptation in the primate retina: Analysis of changes in gain and dynamics of monkey retinal ganglion cells. Visual Neuroscience, 4, 75-93. Schnapf, J. L., Nunn, B. J., Meister, M. & Baylor, D. A. (1990). Visual transduction in cones of the monkey Macaca fascicularis. Journal of Physiology, London, 427, 681-713.
Stewart, G. N. (1924). Colour phenomena caused by intermittent stimulation with white light. American Journal of Physiology, 69, 337-353.
Swanson, W. H., Ueno, T., Smith, V. C. & Pokomy, J. (1987). Temporal modulation sensitivity and pulse detection thresholds for chromatic and luminance perturbations. Journal of the Optical Society of America, A, 4> 1992-2005.
Zrenner, E. (1983) Neurophysiological aspects of color vision in primates: Comparative studies on simian retinal ganglion cells and the human visual system. New York: Springer.
Zrenner, E. & Gouras, P. (1981) Characteristics of the blue sensitive cone mechanism in primate retinal ganglion cells. Vision Research, 21, 1605-1609.
copyright 0
0042-6989/91 $3.00 + 0.00 1991 Pergamon PreS plc
TEMPORAL DIFFERENCES BETWEEN COLOR PATHWAYS WITHIN THE RETINA AS A POSSIBLE ORIGIN OF SUBJECTIVE COLORS* S. M. COURTNEY and G. BUCHSBAUM Department of Bioengineering, School of Engineering and Applied Science, University of Pennsylvania, 220 South 33rd Street, Philadelphia, PA 19104, U.S.A. (Received 11 July 1990; in revised form 2 January 1991) Abstract-We propose a model of temporal signal processing within the retina based on temporal differences between the color pathways which may explain the phenomenon of subjective color. We quantify the model by inferring impulse response functions from physiological data, and predict the output of the different color pathways to temporally modulated achromatic signals which produce the sensation of color. Certain achromatic temporally modulated signals create imbalances between the color pathways which are analogous to those produced by stationary chromatic signals.
INTRODUCTION
For more than a century and a half it has been known that intermittent white light can, under certain conditions, produce the sensation of color. In 1826, Benedict Prevost first observed these so-called “subjective colors” by moving a piece of cardboard in front of a white light in a darkened room. Later, in 1838, G. T. Fechner rediscovered subjective colors when he spun a black and white patterned disk below the luminance fusion frequency. There are many ways of producing color with achromatic stimuli. While spinning patterned disks is the easiest and most common method of producing the color sensation, no movement across the retina is necessary, only temporal luminosity modulation (Motokawa & Ebe, 1952; von Campenhausen, 1968; Festinger, Allyn & White, 1971; Muller, Korth, Rix & Weimer, 1980). In addition, a spatial pattern is not necessary. Many researchers demonstrated that color could be induced by varying only the temporal and luminance contrast properties of the signals (see Cohen & Gordon, 1949 for an extensive review of early work; Stewart, 1924). More recently, Festinger et al. (1971) have found spatially homogeneous, temporally modulated signals which produce a variety of color sensations (see Fig. 1). Subjective colors provide a useful probe into the visual system (von Campenhausen, *This paper was presented at the 1990 ARVO meeting. va
31/9-F
1973; Both & von Campenhausen, 1977). This phenomenon demonstrates an essential nonlinearity in the transformation of visual stimuli in which spatial, temporal, and chromatic information interact. Yet, despite its long history, there is no satisfactory quantitative explanation for subjective colors. Temporal differences have been suggested before as an explanation for subjective colors, originally by Fechner in 1838. Stewart, in 1887, proposed four methods of incorporating different delays, rise times, and fall times, into the three color systems. In 1922, Pieron formulated a theory, similar to one of Stewart’s cases, which depends upon the timeto-peak of each of the receptor types (see Cohen & Gordon, 1949; Stewart, 1924). These explanations require temporal differences between the different color pathways which are much larger (differential latencies on the order of 75 msec) than has been found by physiological or psychophysical studies. We propose a three pathway, color-opponent model for generating subjective colors which requires much smaller temporal differences than Fechner’s and Pieron’s theories. From physiological data, we infer impulse response functions and nonlinearities for the three most common wavelength selective on-center parvocellular ganglion cell types. Using these impulse functions, we calculate the outputs of these ganglion cells to the temporally modulated achromatic signals used by Festinger et al. (1971) to produce subjective colors. The model demonstrates how these signals can create an imbalance in the
1541
1542
S. 60
M. COURTNEY
and G.
BUCHSBAUM
“Neutral”
‘Red”
50
4
40 30 20 g z
10
5 =.
0
I
I
I
:: T x ‘Z
60-
“Green”
E 50
‘ii 3
40
0
I
I
1
50
100
150
0
I
I
I
50
100
150
Time (msec) Fig. 1. Achromatic temporally modulated signals used by Festinger et al. (1971) to elicit the indicated subjective color sensations.
Cones
Opponency
Ganglion cells
Higher stages
~~F===&EJ
.
Fig. 2. An overview of the model showing the stages of calculation. The impulse functions are de&d at the ganglion cell level, but the temporal differences may accumulate through the earlier stages. The receptive fields have inhibitory inputs (dashed lines) from all cone types, although some may have a greater weight than others as indicated. Saturation and rectification occur at the ganglion cell level, before the responses are integrated and compared.
A possible origin )f subjective colors
responses of the different color pathways when integrated over time. This imbalanced response to the temporally modulated achromatic signals is analogous to the response that would occur if the cells were presented with stationary chromatic stimuli. DEVELOPMENT
OF THE MODEL
An overview of the model is presented in Fig. 2, showing the stages of calculation. Impulse functions are defined at the ganglion cell level, however, the temporal differences among the pathways may accumulate in the preceding layers. The impulse function is convolved with temporally modulated signals to predict the temporal response profile. Specific nonlinearities are taken into account, and then the response is integrated over time, normalized, and compared to the responses in other pathways. In the model, the abbreviations “L+/M-“, “M+/L-“, and “S+/(L + M)-” color pathways refer to those pathways in the
1.0
impulse functions
3 .= r 3 2 m
visual system which include ganglion cells whose receptive fields are excited by cones which are most sensitive to the long (L), middle (M), or short (S) wavelength regions of the spectrum respectively. The terms “on-response*’ and “off-response” indicate the output of the ganglion cells in response to an increment and a decrement in luminosity respectively. We used the following findings to conceptualize and constrain the model. Zrenner and Gouras (1981, Fig. 3) recorded from retinal ganglion cells of rhesus monkeys while stimulating the center only of the receptive field of each of these cells. They found marked differences between the temporal response profiles of the L +/Mm ganglion cells and those of the S+/(L + M)- ganglion cells. The L+/Mcells appear linear and we may determine their impulse function directly. However, the S+/(L + M)- cell responses have two nonlinearities which are important in the present context: (1) there is an approx. 30 msec delay in the on-response, while there is no observable
kt+ii
L+IM‘
1.5
1543
s+/(L+Mj
II
r
OS
--
0 I
I
0
L
100
80
aa
.r
';I; -_
lz Square responses
60 c
---40
I
20
0
400
Time (msec) Fig. 3. In the bottom row are the responses of each color pathway to a 2OOmsecsquare pulse of light. The solid line is the total response while the dashed line shows the response if only the center of the receptive field is stimulated. The center responses follow the profile of Zrcnncr's (1983) intracellular recordings. In the top row are the impulse functions used in the model which were calculated so as to produce the response pro&s below when convolved with a square pulse of 200 msec. The dashed line in the S+/(L + M)- impulse function graph is the impulse function for the off-response. The solid line for the S+/(L + M)- impulse function is for the on-response only.
S. M. G~JR~NEY and G. BUCHSBAUM
1544
delay in the off-response; (2) the slope of the off-response is markedly different from that of the on-response. Zrenner (1983, Fig. 5.1) also measured critical flicker frequencies electrophysiologically in rhesus monkey parvocellular ganglion cells and found that, in the fovea, the M+/Lpathway had the highest temporal resolution, followed by L+/M- (about 20% slower) and then S+/(L + M)- (about 30% slower than the M +/L - ). Direct recordings of impulse functions have not been able to confirm these differences between the pathways due to high cell to cell variability and noise problems. Schnapf, Nunn, Meister and Baylor (1990) calculated the time-to-peak and the integration times of several macaque cones. The mean values and standard deviations were: Conetype Time to peak (msec) Integration time (msec) M
51* 13
L
55 + 18
s
61 f 1
19 + 10 28 f 14 34+ 17
While these temporal differences between cone types may not appear significant due to the large standard deviations, the trend of increasing temporal resolution, from S through L to M, agrees with that found by Zrenner. As we will show, the differences required by the model are small enough to be in reasonable agreement with these data. Because Zrenner’s data is based on stimulation of the center only of the ganglion cells’ receptive fields, we must add to these responses the effects of a subtractive surround. Because input to the surround appears to come from all cone types, the same temporal properties were used for the surrounds of all ganglion cell types. Lee, Martin and Valberg (1989) measured from macaque retinal ganglion cells and found center-surround delays of 3-8 msec for both L+IMand M+IL- cells. We have used a delay of 6 msec for all three color pathways. Variations in the center-surround delay of + 2 msec do not change the output of the model significantly. We used these findings to infer temporal responses for the three color pathways defined here including impulse responses which incorporate the differences in temporal resolution by using slightly wider impulse functions for the slower pathways (see Fig. 3). The S+/(L + M)pathway is the slowest (Zrenner & Gouras, 1981). In the model we use an impulse function for this pathway with a full width at half maxi-
mum of 5.3 msec. As indicated above there is evidence which suggests a slightly higher temporal resolution for the M+/L - pathway than for the L+/M- pathway. In the model, we postulate a difference in impulse response width of 0.8 msec in full width at half maximum. The full widths of the L+lM- and M+IL- pathway impulse responses are 3.2 and 2.4 msec respectively, implying a 25% difference. The model impulse function is for an ideal (delta function) impulse. Physiologically measured impulse responses depend on the width of the stimulus impulse used and on the average luminance level (Gielen, van Gisbergen & Vendrick, 1982; Purpura, Tranchina, Kaplan & Shapley, 1990; Lee, private communication). The time delay and nonlinearity of the S+/(L + M)- pathway are considered by using a different impulse function for the off-response than for the on-response. For the S+/(L + M)pathway, the slope of the off-response is different from that for the on-response and the offresponse does not have the 30 msec delay that the on-response does. This is unlike a linear system which would require the same temporal characteristics in both off- and on-responses except for a sign change. Figure 3 shows the impulse functions and square pulse responses (for both total response and for center only stimulation) for the three color pathways. The S+/(L + M)and L +/M - impulse response shapes were chosen so that the square pulse response of the center mechanism follows the response profile reported by Zrenner and Gouras (1981, Fig. 3). It is reasonable to ask if saturation, and rectification from the finite level of spontaneous activity, play a role in determining the response profiles of the ganglion cells under the normal viewing conditions of subjective colors. Saturation and rectification effects have been demonstrated in electrophysiology in cones, ganglion cells, and the LGN (Schnapf et al., 1990; Purpura, Kaplan & Shapley, 1988). The results presented here were calculated using a single stage of saturation taken after the convolution of the input with the impulse functions and after the subtraction of the background level due to surround inhibition. This would correspond to saturation at either the ganglion cells or the LGN. Adding other stages of saturation, such as before the subtraction (at the individual cone level), can enhance the imbalances to certain signals. The cut-offs introduced by the rectification and saturation constitute another nonlinearity in the model.
A possible origin of subjective colors RESULTS
By convolving the impulse response functions shown in Fig. 3 with any arbitrary temporal input signal and considering the nonlinearities discussed above one may predict the output of the three color pathways at the ganglion cells. Figure 1 shows the temporally modulated achromatic sequences used by Festinger et al. (1971) to elicit various color sensations. The outputs, predicted by the model, for these signals are shown in Fig. 4. Festinger’s observations indicate that the colors perceived with a single cycle of the stimulus are approximately the same as those seen for multicycle signals. Therefore, the calculated responses shown here are for single cycles in the interest of clarity. We integrated the calculated response profiles over 150 msec. This covers the time for one full cycle plus sufficient time without any stimulus for the outputs to return to near zero. We thus obtained the area under each curve. (Integrating over a longer period of time, therefore, would
120 r 100
1545
not change the results.) This integration time is consistent with chromatic pulse threshold measurements, which indicate an integration time for color information from 160 to 320 msec depending upon mean retinal illumination (Swanson, Ueno, Smith & Pokomy, 1987). The areas under the response profiles were then normalized so that the area under each curve for the stimulus which appeared neutral was equal to unity. Any deviation from neutral, such as that caused by a physically “colored” stimulus differentially exciting the cone pigment types, would cause an imbalance in these normalized responses. The normalized areas were compared by calculating the fraction of the total area of all three pathway responses for each color over one cycle. For example: Fraction of activity that is due to the L +lM- pathway = A&A, + A,,, + A,) where A,, A,,, and A, refer to normalized areas under the L+/M-, M+/L- and S+/(L +M)response profiles respectively (see Fig. 5). “Red”
“Neutral”
L
80 80
“Green”
.g 100
5 iii
80 80 40
“BlU8”
ILL
R
Time (msec) Fig. 4. Responses of each of the color pathways to Festinger et al.3 achromatic signals as predicted by the model. The solid line is the response of the L+/M- pathway, the long dashed line represents the M+/L- response,and the short dashed line shows the P/(L + M)- response. In the inset of each set of responses is the associated input signal which is also shown in Fig. 1.
1546
S. M. COURTNEYand G. BUCHSLMJM 60 0
L+IM-
M+/i
50
n
s+/(L+ M)‘
li; 5 40
0
“Neutral”
“Red’
“Green” Stimulus
‘Blue”
-
name
Fig. 5. Ratios of the normalized area for each pathway to the sum of the areas for all three pathways are given for each of Festinger et al.3 achromatic input signals. The name of the color which resembles that induced by the signal is indicated below each set of responses.
We considered the composition of the center and surround areas of ganglion cell receptive fields in interpreting the relationship between the relative integrated responses in the three ganglion cell types. Lennie (1980) suggests that, at least in the fovea, color opponency in the ganglion cells is a consequence of the center containing a single cone, while the surround is made of a combination of cone types. The more sharply tuned centers, therefore, are paramount in mediating color information through the retina. For example, a stationary, spatially homogeneous, long-wavelength stimulus will produce a proportionately greater normalized response in the L +/AC retinal pathway than in the other on-center pathways (an imbalance compared to the response to a neutral stimulus). This is interpreted through further processing in higher stages which leads to the perception of “red”. We expect that such an imbalance must occur for the corresponding temporally modulated achromatic signal inducing a “red” subjective color sensation. Figure 5 shows that the model predicts an imbalance among the different color pathways which agrees with the imbalance expected from the color sensation induced by each signal used by Festinger et al. (1971). The psychophysical results with stimuli intended to produce cyan and magenta sensations were not consistent between observers. We were also not able to
produce a significant imbalance for these two signals using the present model. The color names given to each signal follow those in the experiment (Festinger et al., 1971) and are general comparative descriptions only; the colors perceived are very unsaturated. With all the present physiological uncertainties and the qualitative nature of the phenomenon we do not attempt to determine the chromaticity coordinates of or otherwise quantify the actual colors here. The signals used by Festinger et al. (1971) to induce subjective color had a contrast of approx. 74%. Contrast is computed as: (L,A -
Lback~ound)/(Lpcak
+
Lbackground).
The level of hard-limiting saturation was chosen such that signals of more than 60% contrast would have some saturation effect. The model is sensitive to this parameter to within 3% of this value. The model also assumes a spontaneous activity level of 5% of the saturation level. While rectification is required by the model, it is not sensitive to variations of up to 40% of this value. However, it is difficult to compare these levels with physiological data for several reasons. Saturation and spontaneous activity levels vary with background and adaptation conditions (Zrenner, 1983; Purpura et al., 1988). Both are lower in the LGN (saturation at 2040% stimulus contrast) than in retinal
A possibleorigin of subjectivecolors ganglion cells (no saturation until 60% contrast) (Kaplan, Purpura & Shapley, 1987). The sensitivity of the model to the level of saturation and the dependence of the cut-off on average luminance and contrast indicates that not all such temporally modulated signals will produce subjective colors (Festinger et al., 1971; Jarvis, 1977). If the contrast is too low, the signals will not saturate and the effect of this nonlinearity will be lost. In addition, at lower luminance levels the impulse functions become wider, diminishing the relative significance in the model of the small difference between the L+lM- and M+/L- responses. On the other hand, if the contrast is too high, the output signals will oversaturate, cutting off the peaks of the responses where most of the differences occur. This seems to be the case in the Festinger et al. first attempt at producing subjective colors in which the signals, which peaked at 5000 td, were shown against a very dark background (near 100% contrast). Under such conditions, no colors were seen, only “very bright lights”. Such high contrast may also be the reason that Jarvis (1977) did not see color when using temporally modulated signals of 170-4600 td, a contrast of over 96%.
CONCLUSION We have shown that an integrative model, which incorporates the entire temporal response profiles (including time delays and nonlinearities) of the three color pathways defined by ganglion cells, provides a possible explanation for subjective colors induced by spatially homogeneous, temporally modulated achromatic signals such as those used by Festinger et al. (1971). The concept of temporal differences explaining this phenomenon is not a new one (Fechner, Pieron in a review by Cohen & Gordon, 1949; Motokawa & Ebe, 1952) and is considered to be reasonable within the context of the visual system (Boynton, 1979). The earlier models, however, required temporal differences of the order of 75 msec between color pathways. The need for this unreasonably large time delay arose because the earlier models considered only temporal delays between peak responses of the color pathways. In the present explanation, however, we propose a different method of incorporating temporal differences which requires smaller, more physiologically plausible differences. The present model integrates and
1547
compares the entire temporal response, including saturation and rectification, not simply its peak. Small differences in the responses are integrated over time and accumulate considerable effects. Additionally, the present analysis considers the subtractive effect of the inhibition which makes the model more sensitive to small differences in the temporal responses of the three pathways. The temporal parameters in the model are based on data from Zrenner (1983) and Zrenner and Gouras (198 1) recorded from ganglion cell center mechanisms. In particular, these include the profiles of the responses of the L+/M- and S+/(L + M)- pathways and the nonlinearities of the S+/(L + M)pathway. The faster response in the M+/L- pathway, although not shown conclusively by the data, is at least consistent with physiological measurements of flicker frequency (Zrenner, 1983) and with recordings from cones (Schnapf et al., 1990). To the best of our knowledge no physiological study is available which directly compares temporal response profiles in the retina with great enough precision to confirm this small difference. Further refinement of the model is limited by the unsaturated and sometimes elusive nature of subjective colors. The signals used in this simulation, in order to produce the proper imbalances, require a particular range of average luminance, contrast, and temporal frequency. Some signals which produce subjective colors require a two-part signal, each part shown to adjacent areas of the retina. For example, the well known Benham’s Disk (Benham, 1884, 1885) may be simulated with a small center area which varies between white, grey, and black, and a surrounding area which alternates between white and black at the same frequency as the center signal. The relative phase between the two signals can alter both the perceived chromaticity and luminance of the center signal (Festinger et al., 1971). Clearly spatial interactions are important here as well as temporal processing. However, more information about the necessary sizes and spatial configurations of the two signals as they relate to the sizes of ganglion cell receptive field centers and surrounds is necessary. Further investigation of the interaction of ganglion cell centers and surrounds and their retinal circuitry, as well as more precise recordings of temporal properties of cells within the retina, will enable such refinement.
1548
S. M. COURMEYanId G. BIJCHSBAUM
Acknowledgemenr-We
thank Dr Peter Sterling for his
helpful criticism.
and phase of responses of macaque retinal ganglion cells to flickering stimuli. Journal of Physiology, London, 414, 245-263.
Lennie, P. (1980). Parallel visual pathways: REFERENCES Benham, C. E. (1884). The London, 51, 200. Benharn, C. E. (1885). The London, 52, 321. Both, R. 8cCampenhausen, change pattern induced
artificial spectrum top. Nature, artificial spectrum top. Nature,
A review.
Vision Research, 20, 561-594.
Lennie, P. & D’Zmura, M. (1988). Mechanisms of color vision. CRC Critical Reviews in Neurobiology, 3, 333-340. Motokawa, K. & Ebe, M. (1952). Retinal colour processes caused by intermittent white light. Nature, London, f70, 79-80.
C. von (1977). Short time delays flicker colors (PIFCs). Vision
Research, 17, 651-660.
Boynton, R. M. (1979). Human color vision (pp. 327-333). New York: Holt, Rinehart & Winston. Campenhausen, C. von (1968). Uber die farbcn der Benhamschen scheibe. Zeitschrifrfur verleichende Physiologic, 60, 351-374.
Campenhausen, C. von (1973). Detection of short time delays between photic stimuli by means of pattern induced flicker colors (PIFCs). Vision Research, 13, 2261-2271.
Cohen, J. & Gordon, D. A. (1949). The PrevostFechner-Benham subjective colors. Psychofogical Bulletin, 46, 97-136.
Festinger, L., Allyn, M. R. & White, C. W. (1971). The perception of color with achromatic stimulation. Vision Research, II, 591-611. Gielen, C. C. A. M., Gisbcrgen, J. A. M. van & Vendrik, A. J. H. (1982). Reconstruction of cone-system contributions to responses of colour-opponent neurones in monkey lateral geniculate. BioZogical Cybernetics, 44, 211-221.
Jarvis, J. R. (1977). On Fechner-Benham subjective colour. Vision Research, 17, 445-45 1. Kaplan, E., Purpura, K. & Shapley, R. M. (1987) Contrast affects the transmission of visual information through the mammalian lateral geniculate nucleus. Journal of Physiology, London, 391, 267-288. Lee, B. B., Martin, P. R. & Valberg, A. (1989). Amplitude
Miiller, G., Korth, M., Rix, R. & Weimer E. (1980). Information coding in the visual system: A new hypothesis. Aibrechr von Graefes Archiv fur Klinihale Oplhalmologie 214, l-7.
Purpura, K., Kaplan, E. & Shapley, R. M. (1988). Background light and the contrast gain of primate P and M retinal ganglion cells. Proceedings of the National Academy of Sciences, U.S.A., 85, 4534-4537. Purpura, K., Tranchina, D., Kaplan, E. & Shapley, R. M. (1990). Light adaptation in the primate retina: Analysis of changes in gain and dynamics of monkey retinal ganglion cells. Visual Neuroscience, 4, 75-93. Schnapf, J. L., Nunn, B. J., Meister, M. & Baylor, D. A. (1990). Visual transduction in cones of the monkey Macaca fascicularis. Journal of Physiology, London, 427, 681-713.
Stewart, G. N. (1924). Colour phenomena caused by intermittent stimulation with white light. American Journal of Physiology, 69, 337-353.
Swanson, W. H., Ueno, T., Smith, V. C. & Pokomy, J. (1987). Temporal modulation sensitivity and pulse detection thresholds for chromatic and luminance perturbations. Journal of the Optical Society of America, A, 4> 1992-2005.
Zrenner, E. (1983) Neurophysiological aspects of color vision in primates: Comparative studies on simian retinal ganglion cells and the human visual system. New York: Springer.
Zrenner, E. & Gouras, P. (1981) Characteristics of the blue sensitive cone mechanism in primate retinal ganglion cells. Vision Research, 21, 1605-1609.
