Turning artificial intelligence research into real-world tools isn’t always straightforward, despite the remarkable progress in recent years ranging from self-driving cars to large language models. Systems that perform well in controlled lab settings often fall short when deployed in homes, in hospitals or on city streets.

Vision-based systems, for example, hold promise for improving safety, monitoring health and aiding rehabilitation. For example, a camera-based system could help a clinician assess a patient’s gait or rehabilitation progress.

In the future, an assistive robot in an older adult’s home might need to recognize when the person is struggling to stand or reaching for an object. But a major challenge is ensuring these robots perform reliably beyond the lab, particularly in messy, unpredictable or low-visibility conditions.

As a computer scientist who studies AI systems designed to understand human behavior – especially through visual cues like body movements and interactions with objects – I see three main challenges to taking AI from the lab to the real world: generalization, human behavior and resources.

Challenge 1: The generalization gap

Most AI vision systems are trained on well-lit, high-quality images, often from curated sources like motion capture studios or daytime recordings. But when these systems are deployed in real-world settings, such as in a dimly lit home, hospital room or nighttime street, they often struggle. This is not because the technology is flawed, but because the data used to train it cannot fully represent the enormous variability of the real world.

Take human pose estimation, for example. This task involves detecting key points on a person’s body, such as joints, to understand how they’re moving. The result resembles a digital skeleton that represents the positions of a person’s arms, legs and other body parts. Pose estimation has been studied for health and rehabilitation applications, including measuring a person’s gait, monitoring rehabilitation exercises and assessing movement patterns associated with the risk of falling. In well-lit environments, modern AI models do this with impressive accuracy. But in low or uneven lighting, performance can drop sharply. That’s because the model has learned from clean, consistent data and isn’t prepared to generalize to tougher conditions.

Would collecting more nighttime training data solve the problem? More labeled data would certainly help, but collecting it is especially difficult. When a person’s joints are hard to see in an image, they are also hard for a human annotator to label accurately. Low light can also obscure visual information rather than simply make an image darker, and nighttime conditions vary with cameras, shadows, glare and other sources of noise. Collecting and manually labeling examples that cover all these conditions is therefore difficult and expensive.

We’ve been working to close this gap by developing AI systems that perform better in low-light scenarios. One method we use is unsupervised domain adaptation, where a model trained on well-lit, labeled data is adapted to handle low-light settings, without needing manual body-joint labels for the low-light images. This helps bridge the “domain gap” between clean training data and the messier world where these systems are actually used.

In our 2026 study on low-light human pose estimation, we generated realistic low-light training images and designed the system to balance uncertain visual evidence with its learned knowledge of human body structure. The method substantially improved performance on two low-light benchmarks, although the broader challenge is far from solved.

Challenge 2: Human behavior

Another major hurdle is the sheer variety of human behaviors, especially when people interact with objects. In computer vision, this is known as human-object interaction detection. Teaching an AI system to recognize actions like cutting a tomato or passing a basketball requires not just object detection, but an understanding of context and intent.

The real challenge is scale. The number of objects is large, and the ways humans interact with them are even greater. You can sit on a chair, carry it, drag it or stack it. Each is a distinct interaction. A cup might be used for drinking, washing, pouring or handing something to another person. An assistive robot would need to distinguish among these actions to respond appropriately. It’s simply not feasible to collect and label data for every possible combination.

In my research, we study how an AI system can detect interactions it has never seen during training. Researchers call this a generalization-to-unseen-classes problem, and it’s a key challenge for building models that don’t just memorize patterns, but can adapt to new situations. Our work has explored how foundation models can help recognize previously unseen interactions. In a more recent study, we used large vision-language models, models that combine computer vision and natural language processing, to identify the parts of an object that support particular actions, without training the system on labeled examples for that specific task. This matters in areas like assistive robotics and safety monitoring, where AI needs to respond intelligently to behaviors it wasn’t specifically trained on.

This challenge is different from the low-light problem. In low-light pose estimation, the system is trying to recognize a familiar type of movement under unfamiliar environmental conditions. In human-object interaction detection, the action-and-object combination itself may be new. Both are generalization problems, but they require different solutions.

Challenge 3: The resource gap

Even when researchers know how to make models more robust, applying those improvements isn’t always easy, especially in academic or smaller research settings. Today’s most powerful AI systems depend on massive computational resources in the form of data centers that contain huge numbers of specialized computer chips.

These are often too expensive for university labs or public-sector researchers, and the computing power needed to train leading models has continued to rise. According to the 2026 AI Index Report, the technology industry produced more than 90% of the notable AI models introduced in 2025.

At the same time, a model developed using large computing clusters may ultimately need to operate in a clinic, school, robot or home device, which means running on much smaller hardware. Such systems may also need to respond quickly without continuously sending sensitive data to the cloud. This mismatch creates a widening gap between what’s possible in theory and what’s practical in deployment. Bridging that gap requires not just better models, but more efficient algorithms and open-source tools that broaden access to advanced AI.

Why it matters

These challenges aren’t just academic. They affect how and whether AI tools can be trusted in the real world. Imagine a fall-detection system that fails at night, a workplace safety monitor that misses a crucial action or a robot that misunderstands what a person is doing because it’s never seen that exact interaction before. These aren’t just minor bugs – they’re breakdowns in trust, usability and sometimes even safety.

AI systems are being deployed in homes, hospitals, factories, vehicles and other everyday environments. If they can’t adapt to different environments or recognize new human behaviors, their usefulness becomes limited and their risks amplified. That’s why researchers are increasingly focusing not just on accuracy in ideal conditions, but on robustness, generalization and real-world readiness.

Looking ahead: Closing the gap

Closing the gap between lab-trained AI models and messy, unpredictable real-world environments won’t happen overnight, but it’s possible. In my research, we’ve explored techniques like using augmented data to simulate challenging conditions and help models adapt. We’ve also worked on enabling AI systems to better generalize to new, unseen situations by using knowledge from foundation models to distinguish unfamiliar interactions from similar ones encountered during training.

Ultimately, I believe that making AI systems more reliable in everyday settings is a challenge worth solving, not just to improve gadgets and apps, but because AI has the potential to support human well-being, whether that means safer streets, smarter healthcare or more secure homes.

Bo Wang has received research support from the Mississippi Impact Grant, Office for Research and Economic Development at the University of Mississippi. Research discussed in this article also received support from the National Research Foundation, Singapore, under its Strategic Capability Research Centres Funding Initiative.