News
HRI Demo for President Biden
During President Biden’s visit, we demonstrated our work on human-robot collaboration.
Our research is to build intelligent robots that interact with the world like humans so that they can better assist and collaborate with people. While there are various tasks that humans and robots can collaborate in a manufacturing environment, we set up a FANUC industrial robot to demonstrate human-robot handover. These kinds of tasks arise naturally when the human needs the robot’s assistance to fetch an object that is either out-of-reach, too heavy, or in some hazardous environment. Ruixuan Liu and Rui Chen participated in this demo.
There are two major challenges in this handover task: 1. safety, which is how to make sure the powerful robot does not hurt humans; 2. efficiency, that is how to correctly interpret the human’s intention to better assist the human. We developed intelligent perception, AI, and control software for the robot to achieve both safety and efficiency. The software contains a digital twin which tracks and supervises the real-time human and robot behaviors via a camera. The real-time distance and momentum between the human body and the robot arm are monitored. When the human gets close, a safety controller will be activated to move the robot away from the human. In this way, safety is guaranteed even when the robot is in idle mode. In terms of efficiency, when the robot sees the human’s palm up, it quickly infers that the human needs the screwdriver. Then the robot locates the screwdriver, picks it up, and hands it to the human. After the human finishes the task, he hands the screwdriver back to the robot and the robot then places it to the correct location for efficient tool management. This handover loop can be repeated as many times as the human desires without external reset operations. A pre-recorded video of the demo is shown below.
President Biden was very interested in this idea and asked how to best integrate this kind of method in real manufacturing environments. We explained that this can be used to support multi-purpose assembly lines where the human workers can rely on the robot to manage all the tools. We are collaborating with Chengtao Wen – a Senior Research Scientist at Siemens in the hopes of making the transition from prototype to fully commercialized automatic tool management for flexible assembly lines a seamless one.

We would like to acknowledge the support from FANUC, Siemens, and Ford in this research. FANUC generously donated the robot arm to our lab. Siemens supported this research through a sponsored project “Task agnostic real-time perception and control with few-shot cross-platform adaptation”. And Ford supported this research through a University Research Program Award on “Safe uncaged industrial robots”.
Read more on CMU News, Twitter@POTUS.
January 29, 2022
NSF Career Award!
Our proposal “Toward Lifelong Safety of Autonomous Systems in Uncertain and Interactive Environments” gets selected by NSF Career Award.
This Faculty Early Career Development Program (CAREER) grant will fund research that enables autonomous systems to operate safely in close interaction with humans, as required, for example, in next generation manufacturing infrastructure, thereby promoting the progress of science, and advancing the national prosperity. Until recently, humans were physically separated from robots to prevent injuries and fatalities. Modern robotics focuses on humans and collaborative robotic systems working together on the same tasks. A safety hazard in such interactive environments is the occurrence of human errors. It is critical that safety conscious responses be programmed into collaborative robotic systems to guarantee safe behavior even when tasks or environments change. This project will develop a new algorithmic framework for safety assurance of autonomous robotic systems that aims for optimal performance when safety can be managed, anticipates and compensates for inevitable failures when it cannot, and learns from past mistakes. This framework will increase trustworthiness of autonomous systems while minimizing human efforts in deployment and maintenance, critical steps toward granting full autonomy to intelligent robots in uncertain and interactive environments, including such application domains as industrial robotics and autonomous driving. Through close integration of research and education, this project will contribute to new interdisciplinary training in robotics and autonomy, accessible dissemination of research in robot safety to the public, and opportunities for interactive learning through a remotely operated robotic platform. Partnerships with the Advanced Robotics for Manufacturing Institute, the Girls of Steel Robotics program, and the Choate Rosemary Hall college-preparatory school will be leveraged to provide opportunities for graduate student internships with small manufacturers and broaden participation in research of individuals from currently underrepresented groups.
This research aims to make fundamental contributions to a theory of cross-task safe guardians that augment existing hardware platforms without manual tuning, monitor and optimally modify their nominal task-oriented control actions to satisfy constraints representing safety requirements, and accomplish these objectives under time-varying uncertainty. It achieves this aim by investigating data-efficient model learning algorithms that accurately track the dynamics of an interactive environment, as well as by designing adaptive controllers that safely adjust the control strategy according to newly learned dynamic models. A responsibility-based evolutionary adversarial learning approach is developed to enable the adaptive safe control algorithm to achieve optimal performance given limits on available resources. Evaluation of the safe guardian and intelligent optimizer approaches is achieved in simulation and experimentally using autonomous vehicles interacting with human-operated vehicles in different traffic conditions, as well as in space-sharing applications involving robot arm manipulators and other human or robotic agents.
More details can be found on the NSF Webpage.
January 26, 2022
ATI's Article on Our Work
ATI shared the story of our project on robotic weld bead removal project. Read the post here.
December 9, 2021
Congratulations Tianhao for Passing Qualification Exam!
Congratulations Tianhao!
November 18, 2021
Weiye Presented Our Paper in CoRL 2021
Check our paper
November 7, 2021
ICL presence at MECC 2021
- System Identification of Safe Controllers Using Inverse Optimization
- Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithm (preprint)
October 25, 2021
ICL Mini Workshop on Recent Projects
Learn more of our recent work through the mini workshop.
0:00 Tianhao Wei: Robotic safety with Neural Network Components
11:50 Ruixuan Liu: Towards safe human-robot collaboration
25:40 Rui Chen and Alvin Shek: Real-time Collaborative Robot Handling with Recurrent Encoder-Decoder Attentive Neural Process
39:06 Letian Wang: Hierarchical, Adaptable and Transferable Networks (HATN) forDriving Behavior Generation
August 14, 2021
Ruixuan's Master Thesis Talk on Data-Efficient Behavior Prediction for Safe Human-Robot Collaboration
Ruixuan finishes his master study on Data-Efficient Behavior Prediction for Safe Human-Robot Collaboration. His thesis can be found here.
August 14, 2021