Talk@AutoDesk - Unleashing Creativity with Generative AI and Robotics

Changliu gave a talk at AutoDesk, titled Unleashing Creativity with Generative AI and Robotics.

Abstract: This talk explores how generative AI and robotics can automate the journey from design prompt to physical prototype, reducing human effort in customized product assembly. Using Lego as a testbed, we present an end-to-end pipeline that integrates text-to-3D generative design—accounting for physical buildability—with a bimanual robotic system for assembly. By reasoning over structure, stability, and task sequencing, the system transforms creative ideas into tangible products.

This talk highlights the following work from ICL:

[J27] StableLego: Stability Analysis of Block Stacking Assembly
Ruixuan Liu, Kangle Deng, Ziwei Wang and Changliu Liu
IEEE Robotics and Automation Letters, 2024
Citation Formats:
```
    
```

[U] Generating Physically Stable and Buildable LEGO Designs from Text
Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu and Jun-Yan Zhu
arXiv preprint arXiv:2505.05469, 2025
Citation Formats:
```
    
```
Abstract:

We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts. We also develop a text-based LEGO texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Lego, containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models at the project website: this URL.

Teaser:

[C76] A Lightweight and Transferable Design for Robust LEGO Manipulation
Ruixuan Liu, Yifan Sun and Changliu Liu
International Symposium of Flexible Automation, 2024
Citation Formats:
```
    
```
Video:

Teaser:

[U] Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO
Zhenran Tang, Ruixuan Liu and Changliu Liu
arXiv preprint arXiv:2503.06848, 2025
Citation Formats:
```
    
```
Abstract:

Manipulation and insertion of small and tight-toleranced objects in robotic assembly remain a critical challenge for vision-based robotics systems due to the required precision and cluttered environment. Conventional global or wrist-mounted cameras often suffer from occlusions when either assembling or disassembling from an existing structure. To address the challenge, this paper introduces "Eye-in-Finger", a novel tool design approach that enhances robotic manipulation by embedding low-cost, high-resolution perception directly at the tool tip. We validate our approach using LEGO assembly and disassembly tasks, which require the robot to manipulate in a cluttered environment and achieve sub-millimeter accuracy and robust error correction due to the tight tolerances. Experimental results demonstrate that our proposed system enables real-time, fine corrections to alignment error, increasing the tolerance of calibration error from 0.4mm to up to 2.0mm for the LEGO manipulation robot.

[J29] Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking
Ruixuan Liu, Alan Chen, Weiye Zhao and Changliu Liu
IEEE Robotics and Automation Letters, 2025
Citation Formats:
```
    
```
Abstract:

Combinatorial assembly uses standardized unit primitives to build objects that satisfy user specifications. This paper studies assembly sequence planning (ASP) for physical combinatorial assembly. Given the shape of the desired object, the goal is to find a sequence of actions for placing unit primitives to build the target object. In particular, we aim to ensure the planned assembly sequence is physically executable. However, ASP for combinatorial assembly is particularly challenging due to its combinatorial nature. To address the challenge, we employ deep reinforcement learning to learn a construction policy for placing unit primitives sequentially to build the desired object. Specifically, we design an online physics-aware action mask that filters out invalid actions, which effectively guides policy learning and ensures violation-free deployment. In the end, we apply the proposed method to Lego assembly with more than 250 3D structures. The experiment results demonstrate that the proposed method plans physically valid assembly sequences to build all structures, achieving a 100% success rate, whereas the best comparable baseline fails more than 40 structures.

Video:

Teaser:

[C96] APEX-MR: Multi-Robot Asynchronous Planning and Execution for Cooperative Assembly
Philip Huang, Ruixuan Liu, Shobhit Aggarwal, Changliu Liu and Jiaoyang Li
Robotics: Science and Systems, 2025
Citation Formats:
```
    
```
Abstract:

Compared to a single-robot workstation, a multi-robot system offers several advantages: 1) it expands the system’s workspace, 2) improves task efficiency, and more importantly, 3) enables robots to achieve significantly more complex and dexterous tasks, such as cooperative assembly. However, coordinating the tasks and motions of multiple robots is challenging due to issues, e.g., system uncertainty, task efficiency, algorithm scalability, and safety concerns. To address these challenges, this paper studies multi-robot coordination and proposes APEX-MR, an asynchronous planning and execution framework designed to safely and efficiently coordinate multiple robots to achieve cooperative assembly, e.g., LEGO assembly. In particular, APEX-MR provides a systematic approach to post-process multi-robot tasks and motion plans to enable robust asynchronous execution under uncertainty. Experimental results demonstrate that APEX-MR can significantly speed up the execution time of many long-horizon LEGO assembly tasks by 48% compared to sequential planning and 36% compared to synchronous planning on average. To further demonstrate the performance, we deploy APEX-MR to a dual-arm system to perform physical LEGO assembly. To our knowledge, this is the first robotic system capable of performing customized LEGO assembly using commercial LEGO bricks. The experiment results demonstrate that the dual-arm system, with APEX-MR, can safely coordinate robot motions, efficiently collaborate, and construct complex LEGO structures.

Video:

Teaser:

Read more at this research project page.