Vision

Robots must act in physical environments where objects, contacts, and task details vary from one attempt to the next. A fixed pipeline can be brittle when its assumptions do not match the scene, while a general-purpose agent without physical grounding can propose plans that are impossible to execute. We aim to bring the adaptability of agentic reasoning together with the reliability of robotics tools.

Our vision is for robotic agents to replace hand-authored, task-specific reasoning with flexible orchestration of reusable tools. An agent interprets a goal, decides what information is missing, and composes capabilities such as perception, simulation, geometric reasoning, motion planning, skill libraries, and feedback control. These tools serve as structured priors: they encode useful physical knowledge and validated procedures, so the agent can reason and act without having to rediscover geometry, dynamics, or execution constraints from scratch.

This is not a move from structure to unconstrained language generation. The agent chooses and adapts the structure supplied by its tools; physics checks, grounded geometry, and controllers keep its proposals tied to what a robot can do. The resulting loop can inspect a scene, build or update a model, test a candidate plan, execute it, and use observed outcomes to revise the next action.

Research Directions

Agentic Real2Sim2Real Data Generation

Simulation can make robot learning faster and safer, but building a useful simulation from a real task is itself labor-intensive. ARSTAG uses coordinated language agents to turn a single RGB image and an instruction into task-specific simulation scenes, feasible robot demonstrations, and task-consistent randomized data. The generated experience can then support policy training and evaluation before deployment on physical robots.

This direction treats simulation as an active tool in the agent’s workflow, not merely a source of synthetic images. Agents use scene reconstruction and demonstration generation to create data that reflects the task, then use real-world outcomes to measure how well the resulting policies transfer. The work evaluates seven manipulation tasks across three visuomotor policy architectures on a dual-arm robot; the reported pi0.5 policy reaches 74.6% average real-world success.

Publication:

1. [U] ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation
   Bowei Li, Yuner Zhang and Changliu Liu
   arXiv:2609.24563, 2026
   

   Teaser Preprint Page Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   ARSTAG automates the creation of robot policy training data from one RGB image and a language instruction. Coordinated language agents reconstruct task-relevant simulation scenes, produce feasible robot demonstrations, and diversify the data using task-consistent randomization. Evaluation covers seven manipulation tasks and three visuomotor policy architectures on a dual-arm robot. The pi0.5 policy achieves 74.6 percent average real-world success. Ablations indicate improved robustness from randomization and better performance with larger generated datasets.

   Video:
   

   Teaser:
   

   

Geometry-Grounded Skill Composition

Long-horizon manipulation requires more than selecting a familiar skill: the robot must determine whether a skill’s requirements match the current scene and instantiate its actions for the objects at hand. ManiSkillFormer connects reusable skill schemas to observations through explicit geometric contracts. An LLM agent specifies the geometry a skill needs and corresponding motion templates; perception grounds task-relevant 3D geometry, and the system uses it to instantiate a reusable skill.

These contracts act as a prior over what a skill needs to succeed. They make the agent’s reasoning inspectable and connect semantic task descriptions to concrete geometry and motion, without requiring new demonstrations or per-object policy fine-tuning. Evaluations include pick-and-place across 30 instances from eight object categories, functional manipulation, and long-horizon tasks on a dual-arm robot.

Publication:

1. [U] ManiSkillFormer: Demonstration-Free Compositional Manipulation via Task-Conditioned Geometric Contracts
   Peiqi Yu, Mosam Dabhi, Shangtao Li, Bowei Li, Laszlo Jeni and Changliu Liu
   arXiv:2609.16331, 2026
   

   Preprint Page Code Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Adapting robotic manipulation to new objects and tasks often requires additional demonstrations, policy fine-tuning, or manual engineering. Reusable manipulation skills can reduce this effort, but connecting their execution requirements to scene-specific geometry remains challenging. We present ManiSkillFormer, a framework for demonstration-free and compositional manipulation that connects perception and action through explicit geometric contracts. Building on reusable skill schemas, LLM agents generate contracts specifying the geometry primitives required by each skill, together with corresponding motion templates for semantic objects and task contexts. These contracts guide a perception module to ground task-relevant 3D geometry from observations, which is then used to instantiate reusable motion templates in a skill library. We evaluate ManiSkillFormer on a dual-arm robot across demonstration-free pick-and-place with 30 instances from 8 object categories, functional manipulation including unscrewing, pouring, pressing, and folding, and three long-horizon tasks. ManiSkillFormer achieves an average success rate of 88.97% for pick-and-place, 75.00% for functional manipulation, and completion rates of 50–80% across the long-horizon tasks, outperforming the evaluated baselines and two ablated pipelines. These results demonstrate the potential of explicit geometric contracts to support skill reuse and composition across objects and tasks without per-object policy fine-tuning or additional robot demonstrations.

   Video:
   

Unifying Goal

Together, these directions explore how agents can use simulation, geometric contracts, reusable skills, and robot feedback as complementary sources of structure. The long-term goal is a system that can turn a new instruction and a changing scene into a grounded model, a feasible plan, and reliable physical action, while using simulation and execution feedback to improve transfer. Rather than asking one model to learn every capability end to end, we aim to let an agent compose the right tools and priors for each task and expose enough of that process to validate its decisions in the real world.