How does our recognition ability change when we go from color to grayscale images? The fact that we can perfectly well recognize things in old black and white movies and photographs suggests that there should be little effect of removing color information from an image. Indeed, that is what we find when we test normally sighted children on an object naming task. The children are shown many images, either in color or grayscale, and asked to name the object shown.

The performance of normally sighted children does not change when color information is removed. However, a very different result is seen with newly sighted children born with congenital cataract (Prakash children).

The Prakash children show a decrease in performance when color information is removed from images. In each case, performance with grayscale images is worse than performance with color. We would like to understand why is this? The quest to understand this clinical observation has seeded a basic science investigation. Here is our explanation.

First some background. The first stage of sensing color is the cone photoreceptor in our retina. The brain combines information from three types of cone cells to derive color percepts. Importantly, the cone cells have a developmental progression.

As shown below, a newborn’s cones are very immature and over the course of the first couple of years, they acquire their adult forms. At the beginning of life, the cones are very limited in signaling color information. As the image progression at the bottom shows, a newborn has poor color vision and gains color richness over several months.

A Prakash child, when he or she surgically gains sight at the age of several years, already has mature cones. Thus, right after surgery, a Prakash child begins to sense the visual world in rich color. We might think that this is a good thing. But, counterintuitively, this overly good color perception at the outset may be detrimental for the Prakash child.

So, to the question of why Prakash children show a significant drop in performance when color information is removed, our hypothesis is that starting with rich color images makes the Prakash child’s visual system overly reliant on color. Hence, the child performs poorly when color is removed (gray-scale images).

Computationally testing the idea that color restriction early in development may lead to greater robustness to color changes later.

Although this hypothesis nicely explains our clinical data, we want to test it a little more rigorously. We use a computational visual system, specifically a deep network, to examine the impact of different training regimens. In particular, we want to determine if initial color restriction can lead to greater robustness to color changes later.

A network trained only with color images (the C2C regimen) performs poorly when color information is shifted.

However, networks trained on grayscale images, or those that start with grayscale and then see color images have no trouble dealing with color variations. They are robust and their performance is unchanged by color perturbations.

Interestingly, the C2G trained network does not perform as well as the G2C network, demonstrating that the order in which images are shown during training (first grayscale and then color, rather than the other way around) matters for determining final network performance. The biomimetic regimen (gray then color) is better than the non-biomimetic regimen (color then gray). The inference from these results is quite clear.

Both the clinical data from human subjects and the computational data from deep networks suggest that initial experience with achromatic inputs is useful for reducing the reliance on color cues.

“Our findings are significant from three perspectives: 1. They help explain a pattern of deficits observed in newly sighted children, 2. They provide an account for why normal visual development proceeds in the way that it does, and 3. They suggest that the performance of AI vision systems can be improved by incorporating insights from human development”, said Dr. Tapan Gandhi, Professor in Electrical Engineering Department, IIT Delhi and Cadence Chair Professor of Automation & AI.

(The work published in research journal SCIENCE, is a joint collaboration between IIT Delhi; Shroff Charitable Eye Hospital Daryaganj; Dept. of Brain and Cognitive Science, MIT USA; and Project Prakash Charitable Trust, New Delhi. Research paper- https://www.science.org/doi/10.1126/science.adk9587)