Overview:

Assembly is a critical category of tasks in contemporary manufacturing. In particular, we focus on hihg-mix, low-volume (HMLV) assemblies, which are highly customizable, often involving complex operations, such as Lego assembly. Creating an HMLV assembly product is generally time-consuming due to two key factors: 1) designing the assembly and 2) constructing the product. Both of these stages are non-trivial and demand significant human effort.
  • Designing HMLV assembly products is time-intensive because the design must accommodate a wide range of user requirements. Expert knowledge is necessary to thoroughly analyze and refine the design, ensuring that it is both physically valid and meets the diverse specifications set by users.
  • Constructing HMLV assemblies is similarly time-consuming, as it is a labor-intensive process that requires specialized expertise. This includes a deep understanding of the various customized designs and components involved.

    • Image description This line of research aims to harness AI and robotics technology to reduce the human effort required in creating HMLV assembly products, i.e. automating the process from design to execution. To achieve this, these projects use Lego as the benchmarking platform. Lego offers a diverse range of components, allowing users to freely customize their desired products. More importantly, it is a cost-effective and replicable platform, making it an ideal choice for benchmarking.

    Research Topics

    Physics Awareness

    Understanding the physical properties of Lego assemblies is crucial for both designing Lego structures and constructing Lego products. Conventional approaches rely on physics engines to simulate assembly properties and provide cost-effective feedback. However, existing simulations often fail to accurately model the physical interactions between components in a Lego assembly. To address the challenge, this research aims to develop computational tools that can reliably and efficiently capture the physical dynamics of Lego assemblies.


    Structural Stability Analysis for Block Stacking Assembly Image description

    This paper proposes a new optimization formulation, which optimizes over force-balancing equations, to infer the structural stability of block stacking assembly. In addition, we provide StableLego: a comprehensive Lego assembly dataset, which includes a wide variety of Lego assembly designs for real-world objects. The dataset includes more than 50k Lego structures built using standardized Lego bricks with different dimensions along with their stability inferences generated by the proposed algorithm.
    Contributors: Ruixuan Liu, Kangle Deng, Ziwei Wang.

    Publications:

    1. [J27] StableLego: Stability Analysis of Block Stacking Assembly
      Ruixuan Liu, Kangle Deng, Ziwei Wang and Changliu Liu
      IEEE Robotics and Automation Letters, 2024


    Creating Customized Assembly Design

    Designing Lego assemblies is challenging because it is essential to ensure that the generated designs are physically sound, meaning the interactions between components and the overall structure must be valid in the real world. Additionally, the assembly design must meet the user's requirements. In this research, we explore various approaches to interpreting user needs and generating Lego assembly designs, with the goal of minimizing human effort as much as possible.


    Lego UI Image description

    This project aims to design a user-friendly interface that allows users to select their desired Lego designs for the robot to assemble. Specifically, we are developing a Lego design wiki, where users can input text descriptions to retrieve Lego designs that match their expectations. Post

    Contributors: Ruixuan Liu, Shobhit Aggarwal, Kareem Segizekov.



    Creating Lego Design from Demonstration Image description

    This project aims to extract and generate Lego designs from human demonstrations. By directly demonstrating the steps to construct a desired customized Lego object, users can reduce the effort required to search for a design and communicate their needs in a more intuitive way. More importantly, generating Lego designs from human demonstrations enables users to create entirely new Lego structures that have never existed before, offering greater flexibility for customising the assembly.

    Contributors: Ruixuan Liu, Alan Chen, Xusheng Luo.

    Publications:

    1. [U] Simulation-aided Learning from Demonstration for Robotic LEGO Construction
      Ruixuan Liu, Alan Chen, Xusheng Luo and Changliu Liu
      arXiv:2309.11010, 2023
    1. [W] Robotic LEGO Assembly and Disassembly from Human Demonstration
      Ruixuan Liu, Yifan Sun and Changliu Liu
      ACC Workshop on Recent Advancement of Human Autonomy Interaction and Integration, 2023



    Generative Lego Assembly Design Image description

    To further reduce the human effort and expert knowledge required in designing assemblies, we explore the end-to-end generation of Lego assembly designs using generative AI. In this project, we aim to develop generative models capable of interpreting user specifications through intuitive prompts, such as text descriptions, and generating Lego designs that meet the customization requirements. Additionally, this project focuses on integrating physical awareness into the generative model, with the goal of producing physically buildable Lego designs.

    Contributors: Ava Pun, Kangle Deng, Ruixuan Liu.

    Publications: Coming soon!



    Constructing Customized Assembly

    Building Lego structures with robots is challenging due to the small size of the components and the high accuracy required. Additionally, Lego structures are highly customizable, making it difficult for the robot to generalize its policy across different designs. Furthermore, the connections between bricks are not rigid, meaning that subsequent operations can affect or even break the existing structure if performed incorrectly. As a result, the system must have a comprehensive understanding of the physical properties of the Lego design. These challenges make robotic Lego assembly a complex problem. In this research, we aim to enable general-purpose robots to construct customized Lego structures.


    Manipulation Capability Image description

    Lego components are significantly smaller compared to existing robots and grippers. Additionally, Lego assembly requires extremely high-precision manipulation, on the order of millimeters or even sub-millimeters, which presents challenges for manipulating Lego bricks with current grippers and robots. This research focuses on the design of mechanical end-of-arm tools (EOAT) to enable robust Lego manipulation. Specifically, our goal is to design 1) low-cost and 2) transferable EOATs that allow general-purpose robots to reliably manipulate, assemble, and disassemble Lego bricks.

    Contributors: Ruixuan Liu, Kevin Tang, Yifan Sun.

    Publications:

    1. [C76] A Lightweight and Transferable Design for Robust LEGO Manipulation
      Ruixuan Liu, Yifan Sun and Changliu Liu
      International Symposium of Flexible Automation, 2024



    Design Reasoning, Planning and Execution Image description

    Constructing customized Lego structures is challenging because it requires a comprehensive understanding of the structural physical properties to ensure that operations are performed correctly. In addition, Lego structures often require cooperative manipulation due to their complex geometric designs. In this project, we explore methods for reasoning through a given customized Lego design and planning appropriate, efficient multi-robot motions to achieve the physical assembly.

    Contributors: Philip Huang, Ruixuan Liu, Shobhit Aggarwal, Zhongqi Wei, Alan Chen.

    Publications:

    1. [U] Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking
      Ruixuan Liu, Alan Chen, Weiye Zhao and Changliu Liu
      arXiv:2408.10162, 2024



    System Robustness and Failure Recovery Image description

    Detecting and recovering from failures are crucial to ensuring a robust and reliable robotic assembly system. In this project, we explore approaches, such as multimodality and foundation models, to reliably detect potential failures during the assembly process and develop policies for effective recovery.

    Contributors: Ruixuan Liu, Hongyi Chen, Philip Huang.

    Publications:

    1. [U] Robustifying Long-term Human-Robot Collaboration through a Hierarchical and Multimodal Framework
      Peiqi Yu, Abulikemu Abuduweili, Ruixuan Liu and Changliu Liu
      arXiv:2411.15711, 2024


    Sponsor: CMU Manufacturing Futures Institute

    Period of Performance: 2023 ~ Now

    Point of Contact: Ruixuan Liu