Imagine a day when you want to create a product, all you need to do is describe your wildest imagination. An AI system brings your imagination into reality within minutes.

"What I cannot create, I do not understand." - Richard Feynman

We believe the future of intelligence is physically grounded — a form of physical AI that truly understands the world we live in. It goes beyond simply following commands. At its deepest form, it comprehends the physical world so profoundly that it can interpret us, interact with us, and create with us in endless ways. When all you have is an abstract idea for a prototype, this physical AI becomes your creative partner, brainstorming, designing, and building alongside you, seamlessly turning imagination into reality.

This line of research aims to harness cutting-edge AI and robotics technologies to take a step toward the future of physical AI. To this end, we study the problem of creating physical assembly products directly from user text prompts. To achieve this, we use LEGO bricks as our assembly platform. These bricks offer a rich variety of components, enabling users to freely customize their desired creations. In addition, it is a cost-effective and easily replicable environment, making it an ideal choice for benchmarking. More importantly, brick assembly itself is a highly challenging problem due to its customizability, semantic complexity, tiny component sizes, precision requirements, ultra-long horizons, and passive connections, i.e., the connections between bricks are not rigid and may break during the assembly process. Successfully accomplishing brick assembly therefore requires the physical AI system to possess dexterous manipulation skills, long-horizon reasoning, semantic understanding, and, most critically, a deep understanding of our physical world and its underlying physical principles.

We present Prompt-to-Product, a proof-of-concept physical AI system that transforms abstract human imagination into tangible physical creations. Users simply describe their ideas in natural language, and Prompt-to-Product interprets those descriptions to generate concrete, physically buildable assembly designs. The system then reasons and constructs the final product, brick by brick, bringing imagination seamlessly into reality.

Our Prompt-to-Product demonstrates a profound understanding of the physical world. First, all generated assembly designs are physically realizable in the real world. Second, it understands the structural properties of assemblies and knows how to accomplish the construction process, determining the proper order and manipulation skills of operations. Third, it masters ultra-long-horizon tasks, involving hundreds of components and assembly processes that can take more than half an hour to complete.

Publications:

1. [U] Prompt-to-Product: Generative Assembly via Bimanual Manipulation
   Ruixuan Liu, Philip Huang, Ava Pun, Kangle Deng, Shobhit Aggarwal, Zhenran Tang, Michelle Liu, Deva Ramanan, Jun-Yan Zhu, Jiaoyang Li and Changliu Liu
   arXiv preprint arXiv:2508.21063, 2025
   

   Preprint Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Creating assembly products demands significant manual effort and expert knowledge in 1) designing the assembly and 2) constructing the product. This paper introduces Prompt-to-Product, an automated pipeline that generates real-world assembly products from natural language prompts. Specifically, we leverage LEGO bricks as the assembly platform and automate the process of creating brick assembly structures. Given the user design requirements, Prompt-to-Product generates physically buildable brick designs, and then leverages a bimanual robotic system to construct the real assembly products, bringing user imaginations into the real world. We conduct a comprehensive user study, and the results demonstrate that Prompt-to-Product significantly lowers the barrier and reduces manual effort in creating assembly products from imaginative ideas.

   Video:
   

It is critical to ground AI systems in real-world physics, enabling them to truly understand and operate within our physical world. In this line of research, we explore ways to integrate physics awareness into modern AI and robotic systems, bridging the gap between virtuality and reality.

This paper proposes a new optimization formulation, which optimizes over force-balancing equations, to infer the structural stability of block stacking assembly. In addition, we provide StableLego: a comprehensive brick assembly dataset, which includes a wide variety of brick assembly designs for real-world objects. The dataset includes more than 50k assembly structures built using standardized toy bricks with different dimensions along with their stability inferences generated by the proposed algorithm.

Publications:

1. [J27] StableLego: Stability Analysis of Block Stacking Assembly
   Ruixuan Liu, Kangle Deng, Ziwei Wang and Changliu Liu
   IEEE Robotics and Automation Letters, 2024
   

   Link Code Cite

   Citation Formats: BibTeX Plain Text

Modern AI models excel at understanding user ideas and generating virtual content, such as text, images, and 3D meshes. However, they often struggle to produce physically buildable assemblies, where the interactions between components and the overall structure must be valid in the real world. In this line of research, we explore approaches to interpreting user intent and generating assembly designs that are not only creative but also physically realizable.

Creating Brick Designs from Demonstrations

This project aims to extract and generate brick assembly designs from human demonstrations. By directly demonstrating the steps to construct a desired customized assembly object, users can communicate their needs in a more intuitive way. More importantly, generating brick assembly designs from human demonstrations enables users to create entirely new structures that have never existed before, offering greater flexibility for customising the assembly.

Publications:

1. [U] Simulation-aided Learning from Demonstration for Robotic LEGO Construction
   Ruixuan Liu, Alan Chen, Xusheng Luo and Changliu Liu
   arXiv:2309.11010, 2023
   

   Teaser Preprint Video Cite

   Citation Formats: BibTeX Plain Text

   Video:
   

   Teaser:
   

   

1. [W] Robotic LEGO Assembly and Disassembly from Human Demonstration
   Ruixuan Liu, Yifan Sun and Changliu Liu
   ACC Workshop on Recent Advancement of Human Autonomy Interaction and Integration, 2023
   

   Preprint Code Video Cite

   Citation Formats: BibTeX Plain Text

   Video:
   

BrickGPT

We explore the end-to-end generation of brick assembly designs using modern generative AI and foundation models. In this project, we aim to develop generative models capable of interpreting user specifications through intuitive prompts, such as text descriptions, and generating brick assembly designs that satisfy the users' customization requirements. Additionally, this project focuses on integrating physics awareness into the generative model, with the goal of producing physically buildable brick assembly designs.

Publications:

1. [C106] Generating Physically Stable and Buildable Brick Structures from Text
   Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu and Jun-Yan Zhu
   International Conference on Computer Vision, 2025
   Best Paper Award (Marr Prize)
   

   Teaser Preprint Page Code Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts. We also develop a text-based LEGO texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Lego, containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models at the project website: this URL.

   Teaser:
   

   

Coming soon: online demo and design gallery:

Building brick assembly structures with robots is challenging due to the small size of the components and the high precision required. Additionally, brick structures are highly customizable, making it difficult for robots to generalize policies across different designs. Constructing an assembly can involve hundreds of steps, creating an ultra-long-horizon manipulation task. Furthermore, the connections between bricks are not rigid, so improper actions can affect or even break the existing structure. As a result, the system must possess a comprehensive understanding of the physical properties of the assembly. These challenges make brick assembly a complex and still largely unsolved problem. In this line of research, we aim to enable general-purpose robots to construct customized brick structures with accuracy and reliability.

Manipulation Capability

Bricks are significantly smaller compared to existing robots and grippers. Additionally, brick assembly requires extremely high-precision manipulation, on the order of millimeters or even sub-millimeters, which presents challenges for manipulating bricks with current grippers and robots. This research focuses on the design of mechanical end-of-arm tools (EOAT) to enable robust brick manipulation. Specifically, our goal is to design 1) low-cost and 2) transferable EOATs that allow general-purpose robots to reliably manipulate, assemble, and disassemble bricks.

Publications:

1. [C76] A Lightweight and Transferable Design for Robust LEGO Manipulation
   Ruixuan Liu, Yifan Sun and Changliu Liu
   International Symposium of Flexible Automation, 2024
   

   Teaser Link Video Cite

   Citation Formats: BibTeX Plain Text

   Video:
   

   Teaser:
   

   

1. [C103] Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO
   Zhenran Tang, Ruixuan Liu and Changliu Liu
   IEEE/RSJ International Conference on Intelligent Robots and Systems, 2025
   

   Preprint Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Manipulation and insertion of small and tight-toleranced objects in robotic assembly remain a critical challenge for vision-based robotics systems due to the required precision and cluttered environment. Conventional global or wrist-mounted cameras often suffer from occlusions when either assembling or disassembling from an existing structure. To address the challenge, this paper introduces "Eye-in-Finger", a novel tool design approach that enhances robotic manipulation by embedding low-cost, high-resolution perception directly at the tool tip. We validate our approach using LEGO assembly and disassembly tasks, which require the robot to manipulate in a cluttered environment and achieve sub-millimeter accuracy and robust error correction due to the tight tolerances. Experimental results demonstrate that our proposed system enables real-time, fine corrections to alignment error, increasing the tolerance of calibration error from 0.4mm to up to 2.0mm for the LEGO manipulation robot.

Design Reasoning and Execution

Constructing customized brick structures is challenging because it is a super-long-horizon task and requires a comprehensive understanding of the physical structural properties to ensure that operations are performed correctly. In addition, brick assembly structures often require cooperative manipulation due to their complex geometric designs. In this project, we explore methods for reasoning through a given customized brick design and executing appropriate, efficient multi-robot manipulations to achieve the physical assembly.

Publications:

1. [J29] Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking
   Ruixuan Liu, Alan Chen, Weiye Zhao and Changliu Liu
   IEEE Robotics and Automation Letters, 2025
   

   Teaser Preprint Code Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Combinatorial assembly uses standardized unit primitives to build objects that satisfy user specifications. This paper studies assembly sequence planning (ASP) for physical combinatorial assembly. Given the shape of the desired object, the goal is to find a sequence of actions for placing unit primitives to build the target object. In particular, we aim to ensure the planned assembly sequence is physically executable. However, ASP for combinatorial assembly is particularly challenging due to its combinatorial nature. To address the challenge, we employ deep reinforcement learning to learn a construction policy for placing unit primitives sequentially to build the desired object. Specifically, we design an online physics-aware action mask that filters out invalid actions, which effectively guides policy learning and ensures violation-free deployment. In the end, we apply the proposed method to Lego assembly with more than 250 3D structures. The experiment results demonstrate that the proposed method plans physically valid assembly sequences to build all structures, achieving a 100% success rate, whereas the best comparable baseline fails more than 40 structures.

   Video:
   

   Teaser:
   

   

1. [C96] APEX-MR: Multi-Robot Asynchronous Planning and Execution for Cooperative Assembly
   Philip Huang, Ruixuan Liu, Shobhit Aggarwal, Changliu Liu and Jiaoyang Li
   Robotics: Science and Systems, 2025
   

   Teaser Preprint Page Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Compared to a single-robot workstation, a multi-robot system offers several advantages: 1) it expands the system’s workspace, 2) improves task efficiency, and more importantly, 3) enables robots to achieve significantly more complex and dexterous tasks, such as cooperative assembly. However, coordinating the tasks and motions of multiple robots is challenging due to issues, e.g., system uncertainty, task efficiency, algorithm scalability, and safety concerns. To address these challenges, this paper studies multi-robot coordination and proposes APEX-MR, an asynchronous planning and execution framework designed to safely and efficiently coordinate multiple robots to achieve cooperative assembly, e.g., LEGO assembly. In particular, APEX-MR provides a systematic approach to post-process multi-robot tasks and motion plans to enable robust asynchronous execution under uncertainty. Experimental results demonstrate that APEX-MR can significantly speed up the execution time of many long-horizon LEGO assembly tasks by 48% compared to sequential planning and 36% compared to synchronous planning on average. To further demonstrate the performance, we deploy APEX-MR to a dual-arm system to perform physical LEGO assembly. To our knowledge, this is the first robotic system capable of performing customized LEGO assembly using commercial LEGO bricks. The experiment results demonstrate that the dual-arm system, with APEX-MR, can safely coordinate robot motions, efficiently collaborate, and construct complex LEGO structures.

   Video:
   

   Teaser:
   

   

System Robustness and Failure Recovery

Detecting and recovering from failures are crucial to ensuring a robust and reliable robotic assembly system. In this project, we explore approaches, such as multimodality and foundation models, to reliably detect potential failures during the assembly process and develop policies for effective recovery.

Publications:

1. [C87] Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts
   Hongyi Chen, Yunchao Yao, Ruixuan Liu, Changliu Liu and Jeffrey Ichnowski
   American Control Conference, 2025
   

   Preprint Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Current robot autonomy struggles to operate beyond the assumed Operational Design Domain (ODD), the specific set of conditions and environments in which the system is designed to function, while the real-world is rife with uncertainties that may lead to failures. Automating recovery remains a significant challenge. Traditional methods often rely on human intervention to manually address failures or require exhaustive enumeration of failure cases and the design of specific recovery policies for each scenario, both of which are labor-intensive. Foundational Vision-Language Models (VLMs), which demonstrate remarkable common-sense generalization and reasoning capabilities, have broader, potentially unbounded ODDs. However, limitations in spatial reasoning continue to be a common challenge for many VLMs when applied to robot control and motion-level error recovery. In this paper, we investigate how optimizing visual and text prompts can enhance the spatial reasoning of VLMs, enabling them to function effectively as black-box controllers for both motion-level position correction and task-level recovery from unknown failures. Specifically, the optimizations include identifying key visual elements in visual prompts, highlighting these elements in text prompts for querying, and decomposing the reasoning process for failure detection and control generation. In experiments, prompt optimizations significantly outperform pre-trained Vision-Language-Action Models in correcting motion-level position errors and improve accuracy by 65.78% compared to VLMs with unoptimized prompts. Additionally, for task-level failures, optimized prompts enhanced the success rate by 5.8%, 5.8%, and 7.5% in VLMs’ abilities to detect failures, analyze issues, and generate recovery plans, respectively, across a wide range of unknown errors in Lego assembly.