Seeing the unseen, coloring anime: AI and physics shaping the future of imaging

Cameras have long been regarded as tools for capturing what the human eye can see. But that assumption may soon be challenged.

“If we change the way we capture data, the potential of Artificial Intelligence (AI) expands dramatically,” says Associate Professor Hiroyuki Kubo of the Graduate School of Informatics. Dr. Kubo is developing methods to visualize what is normally invisible—such as blood vessels beneath the skin or vehicles hidden in fog—using not specialized equipment, but systems built with ordinary projectors and cameras. His approach could significantly accelerate advances in AI. Beyond imaging, Dr. Kubo is also exploring new possibilities in AI-assisted coloring. By designing systems in which humans and AI collaborate to efficiently color large volumes of animation cels, his research aims to enhance productivity in anime production.

We spoke with Dr. Kubo about his work and his vision for the future as he continues to push the boundaries of what cameras—and AI—can do.

Visualizing the invisible with innovative imaging technology

Please tell us about your research

I specialize in image and information processing. While much of the research in this field focuses on using AI and software to analyze data that has already been captured, I focus on the capture process itself. By rethinking how cameras, lighting, and lens settings are used, I aim to extract visual information that cannot be perceived by the human eye or obtained through conventional imaging methods. I believe that image data captured in this way has strong potential to enhance AI performance.

As a simple example, imagine a photograph of a white liquid in a glass. If that image is analyzed by an AI system, it will most likely be identified as milk. However, the liquid could just be liquid soap. A standard photograph records only the light reflected from an object’s surface, providing limited information about its material composition or internal structure. As a result, even AI systems struggle to reliably distinguish between substances like milk and liquid soap.

To address this limitation, I modify the imaging process itself. By changing the way an image is captured, it becomes possible to obtain visual data that reflects information from inside an object, rather than only its surface. Milk and liquid soap differ in how light travels and scatters within them. By selectively capturing light that penetrates the liquid and returns to the camera, I can visualize these internal differences. If AI systems are trained using this type of data, they may be able to distinguish between materials that appear identical to the human eye—such as milk and liquid soap—based on their internal properties.

What have you been able to visualize using this approach?

A representative example is the visualization of blood vessels beneath the skin. When a standard photograph of the skin is taken, the light reflected from its surface is too strong to reveal the blood vessels underneath. By precisely controlling both the projector and the camera, and introducing a slight spatial offset between the illumination and imaging positions, I selectively capture light that has passed through the tissue and interacted with the underlying blood vessels. This makes it possible to visualize blood vessels in real time without exposing patients to radiation.

Because the system is compact and relatively inexpensive, it has potential applications not only in clinical settings but also in home healthcare and disaster-response environments. In addition, by training AI on the images obtained through this method, it may be possible to support the diagnosis of vascular conditions such as varicose veins.

Are there any other examples of your research?

One example is a collaborative project with a company that uses AI to study methods for visualizing water flow.

From a conventional photograph, it is impossible to determine whether water is flowing or in which direction it is moving. To address this challenge, I disperse extremely fine fibers—cellulose nanofibers—into the water. When there is no flow, the fibers are oriented randomly, and the polarization of the light reflected and scattered from them—the direction in which the light waves vibrate—also varies. However, when the water flows in a specific direction, the fibers align with the flow. This alignment produces a characteristic polarization pattern. By capturing these changes in polarization, I was able to visualize the pattern of water flow.

Combining the images obtained in this way with AI could support a wide range of applications, including more efficient maintenance and management of infrastructure such as bridges, the design of fuel-efficient ships, and the analysis of swimming techniques.

Streamlining anime production through AI-human collaboration

In addition to your research on visualizing the invisible, you are also working on introducing AI into anime production.

Since around 2018, I have been conducting joint research with an anime production company on the theme of improving the efficiency of the anime production process using AI.

Anime production involves many different stages, but the process of coloring line drawings in particular requires a significant amount of labor. My goal is to use AI to streamline this coloring process and improve the overall efficiency of anime production.

Is the idea to have AI automatically apply the colors?

At first, I pursued this approach. I prepared thousands—sometimes tens of thousands—of pairs of line drawings and their fully colored versions, and used them to train the AI. However, we soon realized that achieving 100 percent accuracy would be extremely difficult. In anime production, even a one-percent coloring error can be critical. This realization led me to rethink my research approach.

After considerable trial and error, I developed a system in which AI and humans collaborate in the coloring process. In this system, the AI first proposes several candidate colors, and a human artist then selects the appropriate one. This approach can significantly reduce the amount of work required while maintaining a high level of accuracy in coloring. My goal is to create a workflow in which AI and humans complement each other’s strengths.

It sounds like an interactive system in which AI and humans communicate with each other.

Yes, I believe this kind of collaboration represents one possible model for the relationship between humans and AI in the next generation. Since December 2024, I have also been participating in the research project “Generative AI Accelerator Challenge: GENIAC.” This initiative is implemented by Japan’s Ministry of Economy, Trade and Industry, and the New Energy and Industrial Technology Development Organization (NEDO) to strengthen the country’s capabilities in generative AI development. Through this project, I hope to further accelerate my research. I plan to continue this work with the aim of supporting the future of Japan’s anime industry.

The joy of research: Juggling very different things while sprinting at full speed

I understand that you originally majored in physics. What led you to move into the field of image and information processing?

When I first entered university, I was interested in areas of physics such as cosmology and elementary particle physics. However, as I progressed through my studies, I began to feel a stronger desire to work on problems that were closer to everyday life. Around the time I was about to enter my third year as an undergraduate, I met a professor who specialized in research on facial image analysis. Through my work in his laboratory, I became deeply interested in the field of image and information processing. I eventually earned my doctorate through research on computer graphics using software-based approaches.

After that, I worked for a camera manufacturer, where I was exposed to the technical expertise behind the photographic process. This experience gradually sparked my interest in hardware as well. Later, at the Nara Institute of Science and Technology and Carnegie Mellon University in the United States, I discovered the excitement of research that combines software and hardware, which ultimately led me to the work I do today.

Most researchers in image and information processing specialize primarily in software, and relatively few—probably less than ten percent—also work on hardware, as I do. Looking back, my background in physics may have made it easier for me to move into hardware research involving optics without hesitation.

Is there anything you keep in mind when choosing or developing your research topics?

One thing I try to focus on is identifying problems that few people have noticed or attempted to tackle.

In many areas of research, a large number of researchers tend to focus on the same topic. This often leads to a situation in which everyone competes for speed under the same set of rules, like runners sprinting in a 100-meter race. That kind of competition is, of course, important. However, if researchers only compete within existing frameworks, it can be difficult to generate truly new perspectives or ideas.

By contrast, if you choose a problem that few others have explored, you are free to set your own rules and approach it creatively. For example, you might imagine an unusual challenge—like trying to juggle several very different things while sprinting at full speed.

I believe that one of the great attractions of research is the opportunity to explore unexplored questions from a unique perspective.

The importance of active engagement and clear communication

Finally, do you have a message for students and early-career researchers?

I encourage you to engage actively with the wider community. Rather than focusing only on your own research topic, try attending a variety of events, academic conferences, and research meetings. By meeting many different people, you may find that unexpected connections lead to valuable support.

I also recommend practicing how to explain your research in a way that even your parents or siblings can understand. If you truly understand your work, you should be able to describe it using everyday language and familiar examples, without relying on technical terminology or complex mathematical formulas. This ability to communicate clearly will be valuable in many situations throughout your career. I sincerely hope that each of you will enjoy a fulfilling research life.