Constrained Reinforcement Learning Helps Robot Swarms Transfer from Simulation to Reality
- Published
- Author
- IRIDIA
- Reads
- 463
A 20-robot study found that learning when to switch between preset behaviors improved real-world performance across five swarm missions.


For robot swarms, a learning system with fewer choices may produce behavior that works better on physical machines. Researchers at IRIDIA, the artificial intelligence laboratory of the Université libre de Bruxelles (ULB), tested a reinforcement learning method that selects among preset robot behaviors instead of directly controlling wheel speeds. In experiments with 20 e-puck robots, the resulting policies transferred reliably from simulation to hardware and substantially outperformed a method that learned motor commands directly.
The difference addresses the reality gap: simulated robots cannot reproduce every detail of physical operation. Friction, sensor noise and mechanical imprecision can cause a machine to behave differently from its simulated counterpart. A policy that performs well in the computer may depend on details that disappear on real hardware, leaving the robot unable to carry out its task as intended.
Learning when to switch
The IRIDIA team’s method limits what the learning algorithm can change. Rather than setting each wheel’s speed, the algorithm chooses, from moment to moment, among a small set of established behaviors. Those options include moving toward a light, aggregating with other robots and exploring. The learning task is therefore to determine when the group should switch between behaviors, rather than inventing every motor command from scratch.
That constraint is intended to reduce overfitting to the simulation. A highly flexible algorithm can find strategies that exploit quirks of the simulated environment; those shortcuts may not work once sensors and motors encounter real-world variation. With a smaller set of available actions, the algorithm has less scope to discover such simulator-specific tricks. The researchers’ approach keeps learning focused on how the group coordinates its existing behaviors.
Mauro Birattari, who led the research, describes the reality gap as an overfitting problem: a design method with too much flexibility can learn the simulator instead of the task. Reinforcement learning can locate whatever succeeds in simulation, including solutions that depend on conditions that a physical robot will not reproduce. Restricting the available decisions is intended to make the learned choices more dependable on hardware.
Results on physical robots
The researchers assessed the method across five collective missions, running 250 experiments on physical robots. The swarms trained with behavior selection transferred reliably from simulation to the e-puck robots. They substantially outperformed a state-of-the-art approach in which the learning algorithm directly controlled the motors; that more flexible method struggled after transfer to real machines.
The experiments also found that simpler, more constrained policies transferred better than larger, more flexible ones. That result challenges the idea that greater model flexibility is always an advantage in artificial intelligence. For this class of swarm task, the relevant question is not just how much a policy can learn in simulation, but whether its decisions remain useful when the robots operate with physical sensors and mechanisms.
The result matters because swarm robotics has often favored modular design: engineers assemble group behavior from predefined components. Those methods have a track record of transferring to real robots, but finding effective designs can involve slow, costly search. Reinforcement learning offers a way to automate part of the design process. Combining it with a predefined behavior set gives the algorithm room to adapt coordination while retaining practical limits on what each robot can do.
That trade-off shapes what the method can offer. It learns how and when to combine available behaviors; the behavior set itself must already contain actions relevant to the mission. The study therefore points toward a middle ground between hand-designed modular systems and unrestricted learning of motor commands. For practitioners, a useful next question is whether a suitable set of behaviors can be defined for a new task and robot platform.
What comes next
First author Ilyes Gharbi said the approach could make reinforcement learning more practical for swarm robotics, where designing collective behaviors by hand takes expert time. The researchers identify larger swarms, other robot platforms and less controlled environments as the next settings to test. Those trials will show how well the results extend beyond the 20 e-puck robots and the five missions used in this study.
The work brings together two approaches with different strengths: reinforcement learning can search for useful behavior, while modular methods constrain that search in ways that support transfer to real machines. Search and rescue, environmental monitoring and exploration in places that are difficult for people are among the possible application areas described by the team. The immediate engineering takeaway is narrower: constraining what a swarm can learn may help keep its simulated coordination intact when it moves onto physical robots.
The research was partially supported by the Belgian Fonds de la Recherche Scientifique—FNRS through the PDR project “Reinforcement Learning for Robot Swarms.” The study by I. Gharbi, M. Kegeleirs, G. Legarda Herranz, J. Szpirer, K. Hasselmann and M. Birattari, “Structural bias enables reliable sim-to-real transfer in reinforcement learning for robot swarms,” appeared in Science Robotics in 2026.
