When Embodied AI Meets Millimeter-Scale Assembly: A Recap of the 2nd RoCo Challenge
For an assembly robot, picking up a part is the easy part. The hard part is getting every step right, from the first to the last. A brick placed half a stud off will not seat, and any error made early on is carried into every step that follows.
On September 30, 2026, the 2nd RoCo Challenge (Robotic Collaborative Assembly Challenge) concluded at IROS 2026 in Pittsburgh, USA. The event was organized by researchers from Carnegie Mellon University and Nanyang Technological University, together with Tsinghua University, École Centrale de Lyon, Sharpa, UC Berkeley, A*STAR, MIT, and other institutions. It drew 55 registered teams from more than 8 countries, 7 of which advanced to the onsite finals, competing for a total prize pool of $20,000.
Following the inaugural edition at AAAI 2026 in January, this year’s RoCo kept the Industrial Board Assembly track and added a new Brick Assembly track. Teams had to complete full long-horizon assembly tasks on real bimanual robots, from perception and planning to fine, contact-rich manipulation.
- Competition website: https://rocochallenge.github.io/RoCo-IROS2026/
- Official GitHub: https://github.com/rocochallenge
- Contact: rocochallenge@gmail.com
After the competition, we interviewed several participating teams about the challenges they faced, the technical trade-offs they made, and what they hope to see next.

1. Overview: Two Tracks, One Long-Horizon Test on Real Robots
This year’s RoCo featured two tracks:
- Track 1: Industrial Board Assembly. Using the NIST Task Board from the U.S. National Institute of Standards and Technology, robots had to complete nine assembly operations in order: placing two gears, inserting a rod and a bolt, plugging in USB-A and HDMI connectors, inserting a pin, and seating two batteries. The track tests precision, dexterity, and practicality.
- Track 2: Brick Assembly. Given a target structure, robots had to build it from seven brick types ranging from 1×2 to 2×6, through precise manipulation and long-horizon planning. The track tests long-horizon reasoning, generalization, and creativity.
All onsite hardware was provided by sponsors:
- Dexmate Vega U bimanual robot + Tars grippers: two 7-DoF arms fitted with parallel grippers from Tars, an RGB-D camera in the head, and a fisheye RGB camera on each wrist. This platform was used in both tracks.
- Sharpa North humanoid: equipped with two 22-DoF dexterous hands, serving as the dexterous-hand sub-track of Industrial Board Assembly on the same task board.
Because every team competed on the same organizer-provided hardware, the contest came down to algorithms and systems engineering rather than hardware.
Competition Format: From Simulation Leaderboard to Real Robots
Stage 1: Registration and proposal review. Registration opened on April 27, and 55 teams signed up. Teams submitted technical proposals by July 12, and pre-finalists were announced on July 18.
Stage 2: Simulation phase (June–August). The organizers provided a simulation environment for each track, with all code open-sourced on the official RoCo GitHub. The simulated robot matches the Dexmate Vega U used onsite.
- The Industrial Board Assembly track is built on NVIDIA Isaac Sim. The robot assembles nine parts in a fixed order. Gears and batteries must be placed within 10 mm of the target, while connectors and the pin count only when they snap into their slots. Each trial is scored by the number of parts assembled (0–9), and the final score is the average over 200 evaluation trials. During evaluation, the initial positions of the parts and the board are randomly offset by up to ±1 cm to prevent overfitting to a single initial state.
- The Brick Assembly track is built on BrickSim, a brick physics simulator developed by the organizers. There are two task types. “Step assembly” asks the robot to place a single brick at a target location, worth 1 point on success. “Sequence assembly” gives only the final structure; the robot must plan the order itself and build it from scratch, scoring one point per correctly placed brick. The total score is the step-assembly score plus the square of each sequence-assembly score, so returns grow quickly as structures become more complete. The simulation tasks, mostly shaped like letters and digits, were released at the start of the competition.

For data, the Industrial Board Assembly track provided an official dataset: 200 demonstrations collected with the Dexmate Vega U in Isaac Sim, each a complete trajectory in which all nine parts are assembled successfully, totaling about 120,000 frames. It is released in LeRobot format on Hugging Face and includes three RGB streams (head and both wrists), proprioceptive state, and action labels. The demonstrations were generated by a scripted expert policy (hand-specified end-effector waypoints executed through inverse kinematics), with random perturbations added to each trajectory for diversity, while part positions and assembly order stayed fixed. The organizers also released real-robot data from the Sharpa North and handheld UMI data, along with three reference baselines: the scripted expert policy, Diffusion Policy, and π0.5. The Brick Assembly track placed no restrictions on data sources; teams could collect their own training data using privileged simulator information, teleoperation, or synthetic generation.
Stage 3: Onsite finals (September 26–30). The simulation leaderboard closed on August 31, and 7 teams advanced to the onsite finals. Hardware was set up on September 26. On September 27, teams checked in, attended a rules briefing, and joined hardware bootcamps for Sharpa and Dexmate. September 27–29 was the onsite real-robot hackathon, and the hidden Brick Assembly tasks were released only on the morning of September 29. The final competition took place on the morning of September 30, followed by scoring and the award ceremony.
The two tracks used different onsite formats:
- Industrial Board Assembly: Both the hackathon and the finals used the same NIST Task Board and nine parts as in simulation, with the Dexmate Vega U and Sharpa North platforms scored separately. In the finals, each team had a two-hour window (10:00–12:00 or 12:00–14:00) for up to three official attempts, each capped at 15 minutes. Unlike simulation, which only checks whether a part ends up assembled, onsite scoring split each part into two steps, pick and place/insert, worth 1 point each for a maximum of 18. A pick counts when the part is fully lifted from its initial pose and stably held; a placement counts when the part is stably inserted or seated at the target and stays in place after release, with no partial credit. Robots could autonomously re-grasp and re-insert within the time limit, and only the final state was scored. Elapsed time was used only to break ties.
- Brick Assembly: Onsite target structures came in two kinds. “Known structures” were shared with teams in advance so they could practice during the hackathon, while “unknown structures” were released only on the morning of September 29 to test how well systems generalize to new structures. In the finals, each team chose which structures to attempt, and each structure scored the square of the number of bricks completed: 4 bricks earn 16 points, and 10 bricks earn 100. Squared scoring rewards building one structure further far more than building a little of each, which makes task selection especially important.

A few hard rules applied onsite. Once an attempt began, team members could not touch the robot, the board, the keyboard, or the mouse, or send any control commands; doing so ended the attempt immediately, keeping only the points already earned. Pressing the emergency stop also ended the attempt and counted toward the attempt limit. Policies had to use real-time online perception, and any attempt judged to be an open-loop replay of demonstrations was voided. If a part fell out of the workspace, a judge returned it to its initial position and the robot could keep trying.
Compared with simulation, moving to the real robot is more than swapping machines. Information that comes for free in simulation, such as wrist depth and exact poses, is either missing on the real robot or carries calibration error. Systems refined for months in simulation had just three days to make the jump to physical hardware.
Award Winners
Industrial Board Assembly
- 🥇 1st place: GeoVAST-HIC (UC Berkeley) Members: Xingjian Wang, Mutian Shen, Jiakai Chen, Xinyi Cao, Jingdong Zhou, Chenrui Tie, Haoran Lu, Shuo Yin
- 🥈 2nd place: SteadyHand (University of Alberta) Members: Eunice Ding, Matvey O.
- 🥉 3rd place: V-Team (Carnegie Mellon University) Members: Kamya Singh, Mohammed Rajkotwala
Brick Assembly
- 🥇 1st place: GeoVAST-HIC (UC Berkeley) Members: Jinzhong Gu, Max Wang, Fan Xu (in alphabetical order by last name)
- 🥈 2nd place: IDEAL-RoCo (Wuhan University of Technology, Huazhong University of Science and Technology, Rutgers University) Members: Zhi Wang, Yan Zeng, Hao Wang, Chang Zhu, Yixing Zhai, Yuhao Chen, Tingcong Jiang, Haixin Wang, Bolong Zheng
- 🥉 3rd place: Jiaqi (Christina) Tan (Simon Fraser University)
2. The Core Challenge: Errors Accumulate, and One Stud Off Means Failure
In our interviews, nearly every team pointed to the same biggest challenge: Sim2Real.
“Everything worked well in sims, but even getting our policy to run on real hardware was really really difficult,” said V-Team from Carnegie Mellon University.
In the Brick Assembly track, that difficulty shrinks to the millimeter scale. Tingcong Jiang, an onsite member of IDEAL-RoCo, put concrete numbers on it:
“A brick has to land within roughly half a stud to seat reliably, but errors in the arm kinematics, hand-eye calibration, and head-camera calibration ranged from about 3 to 13 mm depending on the arm configuration. The wrist cameras also provide only fisheye RGB images, without depth. Since the simulation policy assumed accurate poses and wrist depth, directly transferring it to the real robot often resulted in placements that were off by one stud.”
Perception was no easier. The IDEAL-RoCo team noted that bricks are usually single-colored and largely textureless, with few stable local features. The studs on top are arranged periodically, which provides useful geometric information but also introduces repeated appearance and matching ambiguity. Add occlusion, shadows, and viewpoint changes, and precise pose estimation from RGB images alone becomes very hard.
3. Winning Solutions: How GeoVAST-HIC Took Both Titles
GeoVAST-HIC from UC Berkeley was the biggest winner of the event. The team fielded two groups, one per track, and won both Industrial Board Assembly and Brick Assembly.
Industrial Board Assembly: Back to Perception and Planning
At the inaugural RoCo at AAAI 2026, the top three teams in the real-robot round all chose end-to-end VLA approaches. This year’s Industrial Board Assembly champion took a different path.
“It is extremely difficult to learn actions that can successfully complete tasks from the official dataset,” said Jingdong Zhou of GeoVAST-HIC. Based on that judgment, the team built its approach around depth-image recognition and path planning: depth perception first identifies targets accurately, and a planner then generates executable assembly motions.
With limited data, tight tolerances, and only three days of onsite tuning, an interpretable perception-planning pipeline that can be debugged stage by stage is easier to reproduce reliably on a real robot.
Brick Assembly: Winning Through Task Selection
The decisive move by the Brick Assembly champion came at the strategy level. The finals offered several target structures to choose from, and while analyzing the tasks, GeoVAST-HIC noticed that tasks O and U share part of the same structure, so a single build could score for both.

Tingcong Jiang of runner-up IDEAL-RoCo recalled this as the most memorable moment of the event:
“It showed that sim-to-real performance was only part of the challenge. Task selection and scoring strategy mattered just as much. Seeing a single plan pay off twice was impressive.”
To be clear, this reuse was a strategy the team worked out by hand after analyzing the tasks, not a capability their robot policy discovered on its own. Yet it points to exactly the kind of capability robots should learn: recognizing shared substructures across goals in long-horizon assembly, and ordering the build to get the most out of every step.
Approaches from Other Winning Teams
- V-Team (Carnegie Mellon University, 3rd in Industrial Board Assembly): a deterministic pipeline: FoundationPose pose estimation → deterministic inverse kinematics → manually tuned grasps → force-feedback-based retries.
-
IDEAL-RoCo (2nd in Brick Assembly): a modular perception-planning-control framework. The perception layer fuses head RGB-D, wrist RGB, robot forward kinematics, brick CAD models, and stud geometry, using multi-frame observations and quality checks for stability. For example, SAM-based segmentation first finds a candidate region for the baseplate, and the regular stud pattern is then used for geometric localization. The task layer decides assembly order, arm assignment, and whether a handover is needed. The control layer breaks the task into stages such as grasping, transport, alignment, and insertion, switching to smaller, more constrained end-effector motions near the target for fine adjustment.
For real-robot deployment, the team’s key idea was using relative measurements to cancel absolute errors. The system was first developed in Isaac Sim and then transferred to the Dexmate Vega U. Because kinematic and calibration errors were enough to misplace bricks, the team stopped relying on absolute poses. They localized the baseplate by touching its studs with the fingertips, started every pick from a fixed pickup location, and before each placement paused the arm above the target so the wrist camera could directly measure the offset between the held brick and the baseplate (or the brick below). After correcting the target, the robot pressed the brick down vertically. “Because both quantities were measured in the same local frame, much of the calibration error was shared and therefore canceled out.” The team also recorded every run, testing changes on replayed data before deploying them on the robot, which greatly reduced unnecessary hardware trials.
4. Walls Hit on Real Robots: What Embodied Foundation Models Still Need to Learn
One phenomenon deserves a closer look: nearly every team we interviewed chose a structured approach rather than an end-to-end embodied foundation model, and that choice was already made in the simulation phase. IDEAL-RoCo built its system in Isaac Sim around build-order planning and inverse kinematics. On the Industrial Board Assembly simulation leaderboard, the top submissions were almost all policy search, deterministic IK skills, motion planning, or CAD-based state-machine scripts, with the best reaching 8.62 out of 9. By contrast, the organizers’ π0.5 and Diffusion Policy baselines trained on the official dataset, as well as a GR00T N1.7-based submission, all scored close to zero. Once precision requirements reach the millimeter scale, what are today’s embodied foundation models still missing? The walls teams hit on real robots offer fairly concrete answers.
Five Walls the Teams Hit
Wall 1: If you can’t see it clearly, you can’t place it precisely. Precise placement depends on the wrist camera seeing the target clearly, which covered only about two thirds of the bricks. For white-on-white bricks, same-color stacks, or finger occlusion, the system had to fall back on arm kinematics, with errors of up to a full stud.
Wall 2: Once a brick is misplaced, there is no recovery. Several teams admitted their systems still lacked reliable failure detection and recovery. A single misplaced brick affects every step after it.
Wall 3: It’s hard to tell where the error comes from. Offline analysis suggested that most of the error comes from arm kinematics and the head camera, but the data collected onsite was not diverse enough to separate the error sources.
Wall 4: Bimanual coordination is still rudimentary. Take the Type1/C structure: the robot must first build a local subassembly and then attach it to the existing structure from below. IDEAL-RoCo explored local pre-assembly, bimanual handover, and bottom-up insertion, but this part “was not yet solved robustly.”
Wall 5: Getting the system to run at all is a hurdle. Asked what capability they would most like to add, V-Team answered simply: “just getting hardware to run on the machine.”
Six Lessons for Embodied Foundation Models
- Act in relative terms, not absolute ones. Many foundation models output end-effector poses directly in the robot’s base frame, which carries kinematic and calibration errors straight into execution. The most effective engineering trick this year was relative measurement: observing the held object and the target in the same local view so that shared errors cancel out. Conditioning on local wrist observations and outputting relative displacements during contact could make foundation models inherently more robust to calibration drift and embodiment differences.
- What pre-training data lacks is the “last millimeter.” Existing large-scale robot data is dominated by pick-and-place with centimeter-level tolerances, and millimeter-level alignment, insertion, and seating are barely covered. This year’s official dataset has a similar limitation: part positions are fixed across its 200 demonstrations, while simulation evaluation offsets initial positions by up to ±1 cm. This may be one reason teams found it “extremely difficult to learn actions that can successfully complete tasks from the official dataset.” Fine-tuning data for precision assembly needs to cover more positional variation, be denser in the contact phase, and add modalities such as force and touch.
- Failure and recovery must be in the data. “No recovery after a misplacement” was the most common weakness this year. As the inaugural RoCo champion showed by systematically adding failure and recovery trajectories, a foundation model must see enough failures before it can learn closed-loop correction.
- Geometric and physical priors belong in the representation. Textureless bricks with periodic studs make it hard for vision encoders pre-trained on internet images to localize precisely, while structured approaches rely heavily on geometric information such as the stud grid, regular dimensions, and connection relationships. Giving foundation models visual representations that understand 3D geometry and contact structure is a prerequisite for precision manipulation.
- Perception must adapt to real sensor configurations. Wrist depth in simulation but only fisheye RGB on the real robot is enough to break an entire approach. Foundation models need to stay robust when modalities are missing or camera setups change, rather than depending on one ideal sensor combination.
- Beyond a “brain,” you need a reliable “cerebellum.” The champion spotting a substructure shared by two tasks, and the runner-up deciding in advance where to stop, are both high-level planning and risk judgment, exactly where foundation models excel. The last millimeter of alignment and insertion, however, is still handled by geometric measurement and fine control. IDEAL-RoCo also plans to try imitation learning from local visual observations for the final insertion stage. Letting foundation models handle task understanding and sequence planning, paired with high-precision local control, may be the more pragmatic near-term path to industrial use.
5. Highlights from the Floor
For many teams, the most memorable moments were simple and genuine.
“When our robot picked up the first object,” said V-Team.
IDEAL-RoCo described the moment a system that had gone through a long period of simulation, development, and testing finally executed its assembly strategy on the competition robot: “seeing perception, bimanual motion, and precision insertion work together on the real robot was especially rewarding.” The team also enjoyed the onsite exchange: “The atmosphere was very welcoming, and we made new connections, learned about different technical approaches, and gained a lot of inspiration for our own research.”
6. Looking Ahead to RoCo 2027
The teams we interviewed generally found this year’s difficulty “about right”: challenging, but not out of reach. Drawing on their feedback, here is what we have in mind for the next RoCo.
From “well-tuned” to “well-generalized.” We hope to reveal some previously unseen target structures on competition day, encourage teams to attempt them onsite, and award bonus points for complete or partial builds. That way, the competition will test not only how well a system is tuned for a fixed task list, but how well it generalizes to new structures.
Perception across the whole assembly process. This year, bricks were pre-staged at fixed pickup locations. In the future, we are considering having robots locate and pick up loose bricks directly from the table, extending perception from placing to picking.
Tasks that truly require two arms. For example, one arm holding part of a bridge while the other completes the assembly. We hope such tasks will make fuller use of bimanual platforms, rather than having two arms simply take turns.
A better show. We are also considering a short head-to-head round in which two robots build the same structure at the same time, along with an improved livestream with real-time comments and interaction so that remote team members and viewers can take part more easily.
Better preparation conditions for teams. We plan to release sensor specifications, calibration information, and hardware interface details earlier and in more detail, extend onsite tuning time, and improve onsite networking for remote collaboration and data transfer.
We look forward to seeing more teams at RoCo 2027 turn ideas from simulation into reliable, generalizable assembly capabilities on real robots.
7. Core Organizers
Ruixuan Liu
Ph.D. student at the Robotics Institute, Carnegie Mellon University, advised by Prof. Changliu Liu, and onsite organizer of this year's challenge. Research interests include robot manipulation, generative intelligence, and human-robot interaction, with a long-standing focus on robotic brick assembly. Related work includes customized bimanual brick assembly, APEX-MR for multi-robot collaborative assembly, BrickSim, a physics simulator for interlocking bricks, BrickGPT for generating buildable brick structures from text (ICCV 2025 Best Paper Award, Marr Prize), and Prompt-to-Product, which turns natural language into physical assembly products.
Haichao Liu
Postdoctoral Research Fellow at the School of Electrical and Electronic Engineering, Nanyang Technological University, in Prof. Ziwei Wang's PINE (Perception and embodied INtElligence) Lab. Received a Ph.D. from the Hong Kong University of Science and Technology, a master's degree from Harbin Institute of Technology, and a bachelor's degree in robotics engineering from Northeastern University (China). Research spans dexterous robot manipulation with open-vocabulary commands as well as decision-making, planning, and control for autonomous driving, using multimodal large models, reinforcement and imitation learning, and convex and non-convex optimization. Overall lead of the inaugural RoCo Challenge at AAAI 2026 and onsite organizer of this year's challenge.
Ziwei Wang
Assistant Professor at the School of Electrical and Electronic Engineering, Nanyang Technological University. Received a B.S. from the Department of Physics and a Ph.D. from the Department of Automation at Tsinghua University, followed by postdoctoral research at the Robotics Institute, Carnegie Mellon University. Research focuses on foundation models for robot manipulation, including grounding foundation models in the physical scene and deploying them on resource-limited robots.
Changliu Liu
Associate Professor at the Robotics Institute, Carnegie Mellon University, and director of the Intelligent Control Lab. Received a bachelor's degree from Tsinghua University and a Ph.D. from UC Berkeley, followed by postdoctoral research at the Stanford Intelligent Systems Laboratory. Research focuses on the design and verification of human-centered intelligent systems, with an emphasis on lifelong safety and efficient human-robot collaboration in uncertain and interactive environments, applied to manufacturing and transportation. Honors include the 2025 IEEE RAS Early Academic Career Award in Robotics and Automation, the 2025 IFAC Robotics Outstanding Young Researcher Award, the NSF CAREER Award, and an Amazon Research Award.
The full organizing team of this year’s challenge:

8. Closing: Assembly Is the Ultimate Test for Physical AI
If the inaugural RoCo showed the potential of end-to-end VLAs on long-horizon tasks, the 2nd RoCo put the spotlight on another equally critical problem: precision.
When the tolerance is half a stud but errors can exceed ten millimeters, whether a system succeeds depends on whether it knows where its errors come from, whether it can make them cancel out through local measurement, and whether it can detect and recover when something goes wrong. These questions do not belong to any single technical approach; structured pipelines and learning-based systems alike have to answer them.
Technical reports from the participating teams will be released soon, with more details on their methods and lessons learned.
Thank you to all participating teams for their hard work; to our hardware sponsors Dexmate, Sharpa, and Tars, and our funding sponsors Avant Robotics, Eastworlds, and RoboForce; and to the volunteers from the Intelligent Control Lab (ICL) at Carnegie Mellon University.

See you at RoCo 2027!