Imagine a home robot trained to perform tasks in a factory. It might struggle when deployed in a user’s kitchen because the environment differs from its training space. Engineers usually try to match the training environment as closely as possible with the real world.
However, MIT and other institutions discovered that training AI in a completely different, noise-free environment can sometimes yield better results than training in the same noisy environment where the agent will be tested.
This unexpected phenomenon is known as the indoor training effect.
The researchers trained AI agents to play Atari games with added unpredictability. They found that the indoor training effect appeared consistently across different games. They hope this will inspire further research to improve AI training methods.
The researchers wanted to understand why AI agents trained in one environment perform poorly in different environments. They used reinforcement learning, which involves trial and error to learn actions that maximize rewards.
They added noise to a key part of the training, the transition function, which defines the probability of moving from one state to another. For example, Pac-Man might determine the ghosts’ movements.
When they trained the agent with added noise, performance dropped. However, when they trained the agent in a noise-free environment and tested it with noise, the agent performed better than the one trained with noise from the start.
The researchers tested various environments by adding different noise levels to the transition function. This made the games less realistic, with more noise-causing ghosts in Pac-Man to teleport randomly.
They adjusted probabilities to check if the indoor training effect worked in normal Pac-Man games, so ghosts moved up and down more often. AI agents trained in noise-free environments still performed better in these adjusted, realistic games.
The researchers were surprised that this effect seemed to be a general property of reinforcement learning, not just due to their noise adjustments.
When the researchers investigated further, they noticed patterns in how AI agents explore their training spaces.
If both agents explore similar areas, the one trained in a noise-free environment does better because it can learn the game’s rules more easily.
If their exploration patterns differ, the agent trained in the noisy environment performs better, likely because it learns patterns it wouldn’t encounter in the noise-free environment.
Serena Bono, a research assistant in the MIT Media Lab and lead author of a paper, said, “If I learn to play tennis only with my forehand in a non-noisy environment, but then in a noisy one, I have to also play with my backhand, I won’t play as well in the non-noisy environment.”
Looking ahead, the researchers plan to study the indoor training effect in more complex reinforcement learning environments and apply it to other areas like computer vision and natural language processing. They aim to create training environments that use the indoor training effect to help AI agents perform better in unpredictable settings. This could significantly enhance the versatility and robustness of AI systems in the real world.
Journal Reference:
- Serena Bono, Spandan Madan, Ishaan Grover, Mao Yasueda, Cynthia Breazeal, Hanspeter Pfister, Gabriel Kreiman. The Indoor-Training Effect: unexpected gains from distribution shifts in the transition function. arXiv: 2401.15856v2
Source: Tech Explorist