Assembly is a critical category of tasks in contemporary manufacturing. In particular, we focus on high-mix, low-volume (HMLV) assemblies, which are highly customizable, often involving complex operations, such as Lego assembly. Creating an HMLV assembly product is generally time-consuming due to two key stages: 1) designing the assembly and 2) constructing the product. Both of these stages are non-trivial and demand significant human effort.

Designing HMLV assembly products is time-intensive because the design must accommodate a wide range of user requirements. Expert knowledge is necessary to thoroughly analyze and refine the design, ensuring that it is both physically valid and meets the diverse specifications set by users.

Constructing HMLV assemblies is equally time-consuming, as it is a labor-intensive process that requires specialized expertise. This includes a deep understanding of the various customized designs and components involved.

This line of research aims to harness AI and robotics technology to reduce the human effort required in creating HMLV assembly products, i.e. automating the process from design to execution. To achieve this, these projects use Lego as the benchmarking platform. Lego offers a diverse range of components, allowing users to freely customize their desired products. In addition, it is a cost-effective and replicable platform, making it an ideal choice for benchmarking. More importantly, despite being a well-known toy brand, Lego assembly is a non-trivial and, actually challenging, problem due to its customizability, tiny sizes, and most importantly, the passive connections, i.e., connections between bricks are not rigid and could break throughout the assembly process. Thus, accomplishing Lego assembly requires the system to have a comprehensive understanding of the underlying physics as well as dexterous manipulation skills.

Understanding the physical properties of Lego assemblies is crucial for both designing Lego structures and constructing Lego products. Conventional approaches rely on physics engines to simulate assembly properties and provide cost-effective feedback. However, existing simulations often fail to accurately model the physical interactions between components in a Lego assembly. To address the challenge, this research aims to develop computational tools that can reliably and efficiently capture the physical dynamics of Lego assemblies.

This paper proposes a new optimization formulation, which optimizes over force-balancing equations, to infer the structural stability of block stacking assembly. In addition, we provide StableLego: a comprehensive Lego assembly dataset, which includes a wide variety of Lego assembly designs for real-world objects. The dataset includes more than 50k Lego structures built using standardized Lego bricks with different dimensions along with their stability inferences generated by the proposed algorithm.
Contributors: Ruixuan Liu, Kangle Deng, Ziwei Wang.

Publications:

1. [J27] StableLego: Stability Analysis of Block Stacking Assembly
   Ruixuan Liu, Kangle Deng, Ziwei Wang and Changliu Liu
   IEEE Robotics and Automation Letters, 2024
   

   Link Code Cite

   Citation Formats: BibTeX Plain Text

Designing Lego assemblies is challenging because it is essential to ensure that the generated designs are physically sound, meaning the interactions between components and the overall structure must be valid in the real world. Additionally, the assembly design must meet the user's requirements. In this research, we explore various approaches to interpreting user needs and generating Lego assembly designs, with the goal of minimizing human effort as much as possible.

Lego UI

This project aims to design a user-friendly interface that allows users to select their desired Lego designs for the robot to assemble. Specifically, we are developing a Lego design wiki, where users can input text descriptions to retrieve Lego designs that match their expectations. Post

Contributors: Ruixuan Liu, Shobhit Aggarwal, Kareem Segizekov.

Creating Lego Design from Demonstration

This project aims to extract and generate Lego designs from human demonstrations. By directly demonstrating the steps to construct a desired customized Lego object, users can reduce the effort required to search for a design and communicate their needs in a more intuitive way. More importantly, generating Lego designs from human demonstrations enables users to create entirely new Lego structures that have never existed before, offering greater flexibility for customising the assembly.

Publications:

1. [U] Simulation-aided Learning from Demonstration for Robotic LEGO Construction
   Ruixuan Liu, Alan Chen, Xusheng Luo and Changliu Liu
   arXiv:2309.11010, 2023
   

   Teaser Preprint Video Cite

   Citation Formats: BibTeX Plain Text

   Video:
   

   Teaser:
   

   

1. [W] Robotic LEGO Assembly and Disassembly from Human Demonstration
   Ruixuan Liu, Yifan Sun and Changliu Liu
   ACC Workshop on Recent Advancement of Human Autonomy Interaction and Integration, 2023
   

   Preprint Code Video Cite

   Citation Formats: BibTeX Plain Text

   Video:
   

Generative Lego Assembly Design

To further reduce the human effort and expert knowledge required in designing assemblies, we explore the end-to-end generation of Lego assembly designs using generative AI. In this project, we aim to develop generative models capable of interpreting user specifications through intuitive prompts, such as text descriptions, and generating Lego designs that satisfy the users' customization requirements. Additionally, this project focuses on integrating physical awareness into the generative model, with the goal of producing physically buildable Lego designs.

Contributors: Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Jun-Yan Zhu.

Publications: Coming soon!

Design Gallery (In progress):

Building Lego structures with robots is challenging due to the small sizes of the components and the high precision required. Additionally, Lego structures are highly customizable, making it difficult for the robot to generalize its policy across different designs. Furthermore, the connections between bricks are not rigid, meaning that subsequent operations can affect or even break the existing structure if performed incorrectly. As a result, the system must have a comprehensive understanding of the physical properties of the Lego design. These challenges make robotic Lego assembly a complex problem. In this research, we aim to enable general-purpose robots to construct customized Lego structures.

Manipulation Capability

Lego components are significantly smaller compared to existing robots and grippers. Additionally, Lego assembly requires extremely high-precision manipulation, on the order of millimeters or even sub-millimeters, which presents challenges for manipulating Lego bricks with current grippers and robots. This research focuses on the design of mechanical end-of-arm tools (EOAT) to enable robust Lego manipulation. Specifically, our goal is to design 1) low-cost and 2) transferable EOATs that allow general-purpose robots to reliably manipulate, assemble, and disassemble Lego bricks.

Publications:

1. [C76] A Lightweight and Transferable Design for Robust LEGO Manipulation
   Ruixuan Liu, Yifan Sun and Changliu Liu
   International Symposium of Flexible Automation, 2024
   

   Teaser Link Video Cite

   Citation Formats: BibTeX Plain Text

   Video:
   

   Teaser:
   

   

1. [U] Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO
   Zhenran Tang, Ruixuan Liu and Changliu Liu
   arXiv preprint arXiv:2503.06848, 2025
   

   Preprint Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Manipulation and insertion of small and tight-toleranced objects in robotic assembly remain a critical challenge for vision-based robotics systems due to the required precision and cluttered environment. Conventional global or wrist-mounted cameras often suffer from occlusions when either assembling or disassembling from an existing structure. To address the challenge, this paper introduces "Eye-in-Finger", a novel tool design approach that enhances robotic manipulation by embedding low-cost, high-resolution perception directly at the tool tip. We validate our approach using LEGO assembly and disassembly tasks, which require the robot to manipulate in a cluttered environment and achieve sub-millimeter accuracy and robust error correction due to the tight tolerances. Experimental results demonstrate that our proposed system enables real-time, fine corrections to alignment error, increasing the tolerance of calibration error from 0.4mm to up to 2.0mm for the LEGO manipulation robot.

Design Reasoning, Planning and Execution

Constructing customized Lego structures is challenging because it requires a comprehensive understanding of the structural physical properties to ensure that operations are performed correctly. In addition, Lego structures often require cooperative manipulation due to their complex geometric designs. In this project, we explore methods for reasoning through a given customized Lego design and planning appropriate, efficient multi-robot motions to achieve the physical assembly.

Publications:

1. [J29] Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking
   Ruixuan Liu, Alan Chen, Weiye Zhao and Changliu Liu
   IEEE Robotics and Automation Letters, 2025
   

   Teaser Preprint Code Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Combinatorial assembly uses standardized unit primitives to build objects that satisfy user specifications. This paper studies assembly sequence planning (ASP) for physical combinatorial assembly. Given the shape of the desired object, the goal is to find a sequence of actions for placing unit primitives to build the target object. In particular, we aim to ensure the planned assembly sequence is physically executable. However, ASP for combinatorial assembly is particularly challenging due to its combinatorial nature. To address the challenge, we employ deep reinforcement learning to learn a construction policy for placing unit primitives sequentially to build the desired object. Specifically, we design an online physics-aware action mask that filters out invalid actions, which effectively guides policy learning and ensures violation-free deployment. In the end, we apply the proposed method to Lego assembly with more than 250 3D structures. The experiment results demonstrate that the proposed method plans physically valid assembly sequences to build all structures, achieving a 100% success rate, whereas the best comparable baseline fails more than 40 structures.

   Video:
   

   Teaser:
   

   

1. [U] APEX-MR: Multi-Robot Asynchronous Planning and Execution for Cooperative Assembly
   Philip Huang, Ruixuan Liu, Shobhit Aggarwal, Changliu Liu and Jiaoyang Li
   arXiv preprint arXiv:2503.15836, 2025
   

   Teaser Preprint Page Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Compared to a single-robot workstation, a multi-robot system offers several advantages: 1) it expands the system’s workspace, 2) improves task efficiency, and more importantly, 3) enables robots to achieve significantly more complex and dexterous tasks, such as cooperative assembly. However, coordinating the tasks and motions of multiple robots is challenging due to issues, e.g., system uncertainty, task efficiency, algorithm scalability, and safety concerns. To address these challenges, this paper studies multi-robot coordination and proposes APEX-MR, an asynchronous planning and execution framework designed to safely and efficiently coordinate multiple robots to achieve cooperative assembly, e.g., LEGO assembly. In particular, APEX-MR provides a systematic approach to post-process multi-robot tasks and motion plans to enable robust asynchronous execution under uncertainty. Experimental results demonstrate that APEX-MR can significantly speed up the execution time of many long-horizon LEGO assembly tasks by 48% compared to sequential planning and 36% compared to synchronous planning on average. To further demonstrate the performance, we deploy APEX-MR to a dual-arm system to perform physical LEGO assembly. To our knowledge, this is the first robotic system capable of performing customized LEGO assembly using commercial LEGO bricks. The experiment results demonstrate that the dual-arm system, with APEX-MR, can safely coordinate robot motions, efficiently collaborate, and construct complex LEGO structures.

   Video:
   

   Teaser:
   

   

System Robustness and Failure Recovery

Detecting and recovering from failures are crucial to ensuring a robust and reliable robotic assembly system. In this project, we explore approaches, such as multimodality and foundation models, to reliably detect potential failures during the assembly process and develop policies for effective recovery.

Publications:

1. [C87] Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts
   Hongyi Chen, Yunchao Yao, Ruixuan Liu, Changliu Liu and Jeffrey Ichnowski
   American Control Conference, 2025
   

   Preprint Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Current robot autonomy struggles to operate beyond the assumed Operational Design Domain (ODD), the specific set of conditions and environments in which the system is designed to function, while the real-world is rife with uncertainties that may lead to failures. Automating recovery remains a significant challenge. Traditional methods often rely on human intervention to manually address failures or require exhaustive enumeration of failure cases and the design of specific recovery policies for each scenario, both of which are labor-intensive. Foundational Vision-Language Models (VLMs), which demonstrate remarkable common-sense generalization and reasoning capabilities, have broader, potentially unbounded ODDs. However, limitations in spatial reasoning continue to be a common challenge for many VLMs when applied to robot control and motion-level error recovery. In this paper, we investigate how optimizing visual and text prompts can enhance the spatial reasoning of VLMs, enabling them to function effectively as black-box controllers for both motion-level position correction and task-level recovery from unknown failures. Specifically, the optimizations include identifying key visual elements in visual prompts, highlighting these elements in text prompts for querying, and decomposing the reasoning process for failure detection and control generation. In experiments, prompt optimizations significantly outperform pre-trained Vision-Language-Action Models in correcting motion-level position errors and improve accuracy by 65.78% compared to VLMs with unoptimized prompts. Additionally, for task-level failures, optimized prompts enhanced the success rate by 5.8%, 5.8%, and 7.5% in VLMs’ abilities to detect failures, analyze issues, and generate recovery plans, respectively, across a wide range of unknown errors in Lego assembly.