RoboCup@Home’s 50-Object Vision Challenge: Raphael Memmesheimer on NimbRo’s Award-Winning Method

Team NimbRo’s robot during the RoboCup@Home competition at RoboCup 2026. Raphael Memmesheimer leads the team.

For a domestic service robot, recognizing an object is only useful if it can do so in the room where it must act. At RoboCup@Home, teams get two preparation days to assemble their robots and adapt them to an unfamiliar apartment stocked with objects they may never have seen. Team NimbRo’s work on semi-automatic image labeling targets that bottleneck: it helps build a vision model for as many as 50 competition objects in a short window. The team received the RoboCup 2026 symposium best paper award for the method.

Raphael Memmesheimer, a postdoctoral researcher at the University of Bonn and leader of NimbRo@Home, has competed in the league since his student years. His first RoboCup World Cup in 2013 helped set the direction of his academic career. He values the event as a test of autonomous systems in a physical setting: a robot must enter the arena and operate on its own, rather than demonstrate performance through a prepared video or a fixed dataset.

Building a vision model in two days

The apartment changes between events, and organizers typically source groceries locally. Teams can anticipate broad categories such as drinks, snacks and cutlery, but not the particular packaging or products placed around the venue. Some objects come from the publicly available YCB Dataset, so teams can collect examples in advance. General object data can also support pre-training. Those resources help, but they cannot cover every unfamiliar item.

NimbRo’s approach combines object segmentation with reference images of the actual items. First, the team trains a model using objects seen at earlier editions, teaching it to identify regions that may contain an object. It then supplies reference images for the objects at hand and uses semi-supervised labeling to connect the detected regions with those references. The method produced automatic labels for about 60% of the data described in the paper, reducing the amount of manual annotation needed.

“Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation” describes the work. Its practical target is fine-grained recognition across a large set of classes. Memmesheimer says open-vocabulary systems can struggle when asked to distinguish many similar items, such as different drinks. Prompt-based detection and segmentation worked for the team with 5 to 10 classes, but did not scale reliably to all 50 objects in the competition.

The team used the method at RoboCup 2025 in Brazil and at the RoboCup GermanOpen. Memmesheimer says better vision preparation leaves more of the two-day setup period for testing and task execution. For the 2026 World Cup in Korea, NimbRo developed the approach further by using video to generate additional training data.

Example labeled images from different competition venues. Figure from “Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation”.

Laundry put perception and manipulation together

NimbRo finished third at RoboCup 2026, with its strongest performance in a new “Doing Laundry” task. The robot had to move a laundry basket from a washing machine to a folding table, locate and grasp the washer door handle, open the door, remove T-shirts and fold them. It cannot see inside the machine, so it must rely on touch to determine whether it has grasped a shirt before pulling it out.

The task brings several capabilities together: recognizing the machine and its handle, positioning the gripper, manipulating a door, sensing an item in the hand and folding fabric. NimbRo’s robot opened the washer and began folding. Memmesheimer credited the semi-automatic labeling work as part of the preparation that supported this performance.

Team NimbRo’s robot removing laundry from a washing machine.

Five tasks span a home and a restaurant

Before a robot can take part, it must pass an initial inspection covering safety and interactive capabilities. The event then tests teams across five task areas, with varied demands on navigation, perception, communication and manipulation.

In the human-robot interaction task, a robot welcomes arriving guests. It must reach and open the door, let a person enter, maintain eye contact while speaking, guide the guest to a free seat and introduce them to others. The sequence repeats as new guests arrive, making reliable interaction part of the task rather than a one-time demonstration.

A breakfast-clearance pick-and-place task tests household manipulation. The robot must move cutlery to a dishwasher, load a dishwasher tablet, close the machine and set the table for the next meal. Memmesheimer describes this as particularly intensive in its manipulation requirements.

The general-purpose service robot task gives the robot a natural-language command that may involve different parts of the apartment. A request might involve retrieving a particular drink or snack and handing it to someone. In some past events, audience members have supplied commands, testing whether robots can respond to unfamiliar people as well as to team members.

Restaurant service moves the robot to an unfamiliar location. Teams know the menu in advance, but the robot must scan the venue, find customers who are waving and take spoken orders in an often noisy environment. The setting makes the task a public test of how well the system handles a less controlled space.

The laundry task completes the set. Its hidden load makes tactile feedback especially relevant: the robot has to tell by feel whether it is holding a T-shirt, remove it and then fold garments at a table.

Team NimbRo’s robot opening a door.

A broader league raises the bar

Memmesheimer recalls winning the open-platform league at the Eindhoven event he attended. At that time, RoboCup@Home was divided into three leagues: two based on standard platforms and one open-platform league. Those divisions have since merged. RoboCup 2026 brought together 24 teams from around the world in one league, increasing the breadth of the field.

Across his time in the league, Memmesheimer has seen perception improve substantially. When he began competing, deep learning had not yet produced vision models that worked well on unfamiliar objects. Mapping and navigation were already functioning well, but robots could mistake similarly shaped objects, such as a coat hanger or a potted plant, for a person. Human perception and speech recognition have since improved, while task planning remains difficult.

New methods, including large language models, open-vocabulary systems and agentic systems, continue to extend what teams can try. Yet a home robot must combine many demanding capabilities: navigating an apartment, interacting with people, handling objects and planning tasks. Each is a substantial research problem, and a robot also has to perform reliably across a limited number of attempts. Memmesheimer sees periods of slower progress when teams are stretched across that range of challenges.

Team NimbRo.

Next steps for NimbRo

The team’s next priority is manipulation. Memmesheimer says NimbRo is working to make the robot pick a wider variety of objects more quickly and improve two-arm handling. Continued testing also aims to make its behavior more human-like.

For now, the image-labeling method addresses a specific but consequential constraint: teams must teach a robot about local objects before the real tasks begin. By shrinking the manual annotation workload, NimbRo can spend more of its brief preparation period on the full system. Memmesheimer’s long-term goal is autonomous service robots that learn from people through multiple kinds of sensory information and prove useful outside the lab. The next gains will depend on connecting better object knowledge to faster, more capable manipulation.