V. S. Ramachandran & R. L Gregory*

Reprinted from Nature. Vol. 350, No. 6320. pp. 699- 702, 25th April. 1991 with the kind permission of the Editor

Psychology Department 0109, University of California, San Diego, La Jolla. California 92092-0109, USA
* Department of Psychology, University of Bristol, Bristol, UK

Patients with scotomas or blind-spots in their visual field [REF 1-5] resulting from damage to the visual pathways often report that the pattern from the rest of the visual field ‘fills in’ to occupy the scotoma. Here we describe a novel technique for generating an artificial perceptual scotoma which enabled us to study the spatial and temporal characteristics of this filling-in process. A homogeneous grey square subtending 1.5° was displayed against a background of twinkling two-dimensional noise of equal mean luminance (Fig. 1). On steady eccentric fixation for 10s the square vanished and was filled in by the twinkling noise from the surround. Using this display we found that ‘filling in’ is an active visual process that probably involves creating an actual neural representation of the surround rather than merely ignoring the absence of information from the scotoma; filling in can occur separately for colour and texture, suggesting separate mechanisms; the filling-in process does not completely suppress information from the scotoma, even after an image has faded completely from consciousness - it can nevertheless contribute to motion perception; and the process can be strongly influenced by illusory contours.

FIG. 1 Stimulus used to produce an artificial perceptual scotoma. The background consisted of twinkling spots of eight different grey levels. The square subtended 1.5° x 1.5° and it had the same mean luminance (50 cd-m-2) as the twinkling texture. The fixation spot was about 6° away from the border of the square. On steady fixation the square vanished in about 5s and was filled in by the twinkling noise in the surround. A similar fading and filling in of texture was originally described by Ramachandran and Anstis [REF 7] and Ramachandran [REF 8] but in their stimulus the square was a ‘window' filled with horizontally moving dots rather than a homogeneous grey. Also, we found that if the square was very small (<0.2°) it could be seen to vanish even if it was very bright or dark, that is non-equiluminous with the surround. (The effect could then be seen even if the fixation spot was only 2° from the square.) The fading occurred even more quickly (< 2 s) if the square was in a different stereoscopic plane (nearer or further) than the twinkling texture. This was confirmed by all four subjects.

Patients with discrete lesions in their retinas or in primary visual cortex usually have a ‘blind spot’ or scotoma in their visual field as measured by conventional perimetry [REF 1-5]. Oddly enough, if the patient gazes at a companion seen against a background of wallpaper, the companion’s head may disappear but the gap gets ‘filled in’ by the surrounding wallpaper. It is said that King Charles II used to decapitate his courtiers using this benign procedure, although he used his natural blind spot rather than a scotoma.

Of the natural blind spot (corresponding to the optic nerve head), Sir David Brewster [6] has written: "We should expect, whether we use one or both eyes, to see a black or dark spot upon every landscape within 15° of the point which most particularly attracts our notice. The Divine Artificer, however, has not left his work thus imperfect ... the spot, in place of being black, has always the same colour as the ground."

We know surprisingly little about the nature of the neural representation that corresponds to the filling in of scotomas and blind spots. To study the spatial and temporal characteristics of this process we created a novel stimulus that results in a temporary reversible scotoma in normal individuals [REF 7,8]. A homogeneous grey square subtending 1.5°x 1.5° was displayed against a background of dynamic twinkling noise and the subjects were asked to fixate steadily on a spot 6° to the right (or left) of the square (Fig. 1). The twinkle had the same mean luminance (25 cd m2) as the square. Four naive subjects were used in all the experiments described here. After a brief period of steady fixation they reported that the square vanished completely and was filled in by the surrounding twinkle. The average time for fading and filling in was 5.04s (n = 40; four subjects, 10 trials each.)

Does this filling-in process involve creating an actual neural representation of the ‘twinkle’ in the brain areas to which the grey square projects, or does it merely involve ignoring the absence of information? (After all, one is unaware of the huge gaping hole behind one’s head, but one would not want to conclude, for example, that the wallpaper had been filled in behind the head.) One hint comes from the fact that the filling-in process often appeared gradual; after the square’s borders faded, the twinkle filled in from outside to inside slowly, taking 2 or 3 s. This suggests an active process.

FIG. 2 Schematic illustration of stimulus used to demonstrate independent filling-in mechanisms for colour and twinkling texture. The homogeneous grey square subtended 2.6° x 2.8°. The black spots in the square subtended 5 minutes of arc and moved horizontally as in a ‘conveyor belt’. The surround consisted of randomly flickering (‘twinkling’) black spots on a homogeneous pink background (depicted dark grey) whose mean luminance was the same (80 cd m2) as the grey square. The background subtended 19° x 15°. Four subjects were asked to view this display and asked whether the twinkle and colour both filled in at the same time, the twinkle filled in first, or the colour filled in first. On most trials the subjects reported that the colour filled in first (mean = 2.83 s; n = 40) followed a few seconds later by the twinkling noise (mean = 4.05 s).

Is filling-in a unitary process or can it occur separately for colour and texture? To find out, we modified Fig. 1 so that the surround consisted of sparse black dots twinkling on a homogeneous pink background, and the square consisted of black spots moving horizontally against a grey background (Fig. 2) that was equiluminous with the pink. This square, defined by a difference in motion as well as colour, also faded completely in a few seconds, but we found that the filling-in occurred in two distinct stages.

First, the homogeneous grey region in the square vanished and was filled in by the pink from the surround, so that one now had the experience of seeing the black dots moving against the filled-in pink that did not actually exist on the retina. Once this had occurred, the moving spots also faded and were replaced a few seconds later by the twinkling spots in the surround. Thus, there may be separate fill mechanisms for colour and texture corresponding, perhaps, to the different extrastriate visual areas [REF 9-12] that are thought to be specialized for colour or motion (twinkle). Perhaps the colour border fades first and the surround colour is then assigned to this region (in the ‘colour areas’ such as V4). Because the moving spots have not yet faded, the visual system ‘assumes’ that the spots must be moving against the filled-in pink.

Does the fading of the square (and subsequent filling in) depend mainly on the adaptation of neural detectors that extract borders of the square or of the homogeneous region inside it? To answer this question we allowed the filling in to occur and then suddenly switched off and replaced the square with a slightly smaller concentric square (1.3°x 1.3°). Because the region corresponding to this small square has already been filled in with twinkle, does one have to start the filling in again? The answer is yes. The small square did become clearly visible and took several seconds to fade and become ‘filled in’ again (mean = 2.29 s). Thus the fading of contours is critical to the whole process.

How deep is the suppression of information within the scotoma? To investigate this we waited until the square had faded completely, then switched off the square and replaced it with an identical square that was shifted by about 0.4°. To our surprise, we found that the second square seems to ‘move’ or ‘arrive’ at its location, instead of just appearing out of nowhere. Even though the first square was not consciously visible it could nevertheless provide an input to apparent motion perception. A more formal experiment was conducted on the four subjects using a smaller square (1°x 1°).

On each trial we waited until the subject reported complete fading, then we switched off the square and replaced it with a new square that was either to the left or to the right of the original. Left-right trials were presented in random order and the subject’s task was to simply report the direction. Performance on this task was well above chance (75 out of 80 trials; four subjects, 20 trials each). Furthermore, it was clear from the subjects’ phenomenological reports that they were actually experiencing a sensation of motion and not simply reporting a change of location. We postulate, therefore, that the fading of the square from consciousness occurs from area V4, the dorsolateral area (DL) and other ‘form’ areas, but that information about its motion is processed separately in the middle temporal area (MT).

Our next experiment also explored the depth of suppression within the scotoma. We waited until the square was replaced with twinkle, then we suddenly introduced a small (0.2°) red test-probe in the centre of the grey square. All four subjects reported that the spot instantly ‘overcame’ the filling in, but only in the small region corresponding to the spot, so that they saw a red spot surrounded by the filled-in twinkle that did not actually exist around it on the retina. That the red spot is seen to be actually surrounded by the filled-in twinkle is additional evidence for an active filling-in process. If no such process exists, one might have simply experienced a vague inability to decide what was immediately around the red spot.

FIG. 3a, In this display the surround consisted of a counterphase flickering grating instead of twinkling noise. Fading and filling in took longer but could still be seen.
3b, In this display we have stationary continuously visible horizontal lines instead of a counterphase flickering grating. Fading and filling in can be seen after 15-20 s of steady eccentric fixation. We find that one out of every five or six subjects cannot achieve the steady fixation required for this effect. Also, on some trials the entire pattern fades (as in ‘Troxler fading’ [REF 30]) instead of the square, an effect that is rarely seen in Fig. 1.
3c, The square Is displayed against a ‘nonsense’ matrix of English letters. Again, the square fades and gets filled in by the alphabets, but subjects could not actually read the letters in the filled-in region.

Next, we explored the temporal characteristics of the filling-in process using Fig. 1. After filling in had occurred, we switched off the entire display and replaced it with a homogeneous grey field of equal luminance to the square (the square subtended 1.3°x 1.3°). We now found that there was a square patch of twinkly noise in the area that was originally occupied by the square, and this patch persisted for as long as 2-3 s (mean for four subjects = 2.44 s; n = 40). We conclude that a time-varyingneural representation of the surrounding twinkle is created within the scotoma and perhaps this representation persists for 2-3 s after the display is switched off. An alternative possibility is that the twinkle creates some peculiar state of adaptation in the surround, and this adaptation subsequently ‘induces’ a spatially heterogeneous twinkle in the region corresponding to the square. Even so, whatever mechanism is responsible for this induction of twinkle in the region originally occupied by the square is unlikely to be very different from the process causing the filling in of the scotoma in the first place. What is truly remarkable about the percept is its dynamic nature, that it is actually seen to twinkle within the scotoma. When we changed the temporal frequency of the surrounding twinkle to about 10 Hz, from 30 Hz, we found that the perceived temporal frequency of the persistent twinkling patch was also reduced correspondingly.

Is the filling-in process specific to the pattern in the surround? We tried to answer this by using displays such as Fig. 3a, in which the surround was a diagonal twinkling (‘counterphase flickering’) square wave grating. We waited for fill to occur, then replaced the background lines with twinkling two-dimensional noise. Subjects typically reported that this procedure restored the grey square momentarily (1 s) and that it had to be filled in again by the new twinkling texture in the surround. Scotomas can also be produced, although less vividly, using stationary patterns such as Fig. 3b and c. In Fig. 3b the surround consists of a stationary horizontal square wave grating, and in Fig. 3c it was composed of ordinary type-written English, Latin, or ‘nonsense’ text. The filling in of text was especially striking and was reported by all four subjects, although, needless to say, none of them could actually read the text within the filled-in region.

Equiluminous chromatic borders [REF 13,14] and luminance edges [REF15] also tend to fade during optical image stabilization or steady fixation [REF 12]. We found that this was true even if an achromatic (grey) square was displayed on an equiluminous yellow background. The square (subtending 2°) faded in about 3-4 s and was replaced by the yellow from the background. The presence of the yellow within the filled-in region cannot be explained in terms of conventional colour contrast or adaptation effects. Again, if a black spot (0.3° diameter) was introduced in the centre of the square, the filling-in of yellow was observed right up to, but not beyond, the edges of the black spot. Yet, if we had a thin black ring (0.3° outer diameter) instead of a spot, the yellow filled the interior of the ring as well, its spread was not 'blocked' by the ring. So the word 'paint' might be a more appropriate metaphor than 'filling in'.

Our next experiment demonstrates an intriguing interaction between illusory contours [REF 16-22] and the filling-in mechanism. The display consisted of a circular grey disk that straddled the vertical border between two equiluminous regions of different colour (pink and yellow). A vertical illusory contour was then introduced that coincided with the chromatic border and also continued across the centre of the grey disk (Fig. 4a). On steady eccentric fixation, subjects now reported that the colours filled n the disks from the two sides to form an illusory colour border corresponding to the illusory contour. If the illusory contour was not present the colour border within the scotoma usually appeared ‘washed out’, indistinct and unstable. (If the illusory contours coincided with the borders of the disk itself (Fig. 4b) the fading or filling-in processes were delayed.) We may conclude from this that the filling-in process is strongly influenced by early image segmentation, produced by illusory contours.

FIG. 4 Stimulus configuration used to demonstrate the effect of illusory contours on the filling-in process. a, the background was pink on the left (depicted light grey here) and green (depicted dark grey here) on the right and the circular disk was an equiluminous grey. Filling-in occurred from both sides and formed a 'border' between the two filled-in colours that coincided with the illusory contour. In the absence of the central vertical illusory contour (b) the border appeared indistinct and unstable. The filling-in of the disk also took longer in b, perhaps because of the illusory contours associated with the chromatic borders. Eleven naive subjects reported this effect; filling in took place earlier in a than b in 195 out of 220 trials.

Finally, we wondered whether the filling in of the natural blind spot is also an active process that involves spatial integration. To explore this we used a 10°-wide red annulus (ring) whose inner margins just overlapped the outer margins of the blind spot, viewed monocularly. The red colour filled in the blind spot so that the stimulus looked like a homogeneous red disk rather than a ring. Yet when we introduced several concentric thin red rings around the annulus the blind spot appeared filled with rings instead of a homogeneous red, an effect that was confirmed by four naive subjects. Thus what fills in the blind spot is not merely what is immediately around it but also he pattern from the remote surround.

Considerable progress has recently been made in understanding the nature of neural representation underlying perception [REF 23-29]. Our results suggest that the rules used by the visual system to fill in perceptual scotomas may help to elucidate further some of these processes. One intriguing possibility is that filling in simply reflects the activity of higher extrastriate neurons whose large receptive fields may straddle the boundaries of the grey square. Once the neurons that signal the square’s border become fatigued, the signals from the surround may get misattributed to the region corresponding to the scotoma. A second possibility is that feedback pathways [REF 11,24] from extrastriate areas are used to create a representation of the surround in the portion of area 17 that corresponds to the scotoma.

This theory complements, rather than negates, Sir David Brewster’s hypothesis [REF 6] that the filling in is achieved by ‘the Divine Artificer’, and it has the additional merit of being more easily testable.

Note added in proof. In patients with small (1°) retinal scotomas we find that filling in occurs for homogeneous colours and fine-grained static noise but not for dynamic twinkling noise (Fig. 1).

Received 27 December 1990; accepted 27 February 1991.

Acknowledgements

We thank W. Aiken, A. Saluda, D. Rogers-Ramachandran, F. Crick, T. Sejnowsky, P. Churchland and C. Ramachandran for stimulating discussions, and the Airforce Office of Scientific Research for support. R.L.G. was a McDonnel-Pew visiting scholar at UCSD.

References