News

During President Biden’s visit, we demonstrated our work on human-robot collaboration.

Our research is to build intelligent robots that interact with the world like humans so that they can better assist and collaborate with people. While there are various tasks that humans and robots can collaborate in a manufacturing environment, we set up a FANUC industrial robot to demonstrate human-robot handover. These kinds of tasks arise naturally when the human needs the robot’s assistance to fetch an object that is either out-of-reach, too heavy, or in some hazardous environment. Ruixuan Liu and Rui Chen participated in this demo.

There are two major challenges in this handover task: 1. safety, which is how to make sure the powerful robot does not hurt humans; 2. efficiency, that is how to correctly interpret the human’s intention to better assist the human. We developed intelligent perception, AI, and control software for the robot to achieve both safety and efficiency. The software contains a digital twin which tracks and supervises the real-time human and robot behaviors via a camera. The real-time distance and momentum between the human body and the robot arm are monitored. When the human gets close, a safety controller will be activated to move the robot away from the human. In this way, safety is guaranteed even when the robot is in idle mode. In terms of efficiency, when the robot sees the human’s palm up, it quickly infers that the human needs the screwdriver. Then the robot locates the screwdriver, picks it up, and hands it to the human. After the human finishes the task, he hands the screwdriver back to the robot and the robot then places it to the correct location for efficient tool management. This handover loop can be repeated as many times as the human desires without external reset operations. A pre-recorded video of the demo is shown below.

President Biden was very interested in this idea and asked how to best integrate this kind of method in real manufacturing environments. We explained that this can be used to support multi-purpose assembly lines where the human workers can rely on the robot to manage all the tools. We are collaborating with Chengtao Wen – a Senior Research Scientist at Siemens in the hopes of making the transition from prototype to fully commercialized automatic tool management for flexible assembly lines a seamless one.

We would like to acknowledge the support from FANUC, Siemens, and Ford in this research. FANUC generously donated the robot arm to our lab. Siemens supported this research through a sponsored project “Task agnostic real-time perception and control with few-shot cross-platform adaptation”. And Ford supported this research through a University Research Program Award on “Safe uncaged industrial robots”.

Read more on CMU News, Twitter@POTUS.

Our proposal “Toward Lifelong Safety of Autonomous Systems in Uncertain and Interactive Environments” gets selected by NSF Career Award.

This Faculty Early Career Development Program (CAREER) grant will fund research that enables autonomous systems to operate safely in close interaction with humans, as required, for example, in next generation manufacturing infrastructure, thereby promoting the progress of science, and advancing the national prosperity. Until recently, humans were physically separated from robots to prevent injuries and fatalities. Modern robotics focuses on humans and collaborative robotic systems working together on the same tasks. A safety hazard in such interactive environments is the occurrence of human errors. It is critical that safety conscious responses be programmed into collaborative robotic systems to guarantee safe behavior even when tasks or environments change. This project will develop a new algorithmic framework for safety assurance of autonomous robotic systems that aims for optimal performance when safety can be managed, anticipates and compensates for inevitable failures when it cannot, and learns from past mistakes. This framework will increase trustworthiness of autonomous systems while minimizing human efforts in deployment and maintenance, critical steps toward granting full autonomy to intelligent robots in uncertain and interactive environments, including such application domains as industrial robotics and autonomous driving. Through close integration of research and education, this project will contribute to new interdisciplinary training in robotics and autonomy, accessible dissemination of research in robot safety to the public, and opportunities for interactive learning through a remotely operated robotic platform. Partnerships with the Advanced Robotics for Manufacturing Institute, the Girls of Steel Robotics program, and the Choate Rosemary Hall college-preparatory school will be leveraged to provide opportunities for graduate student internships with small manufacturers and broaden participation in research of individuals from currently underrepresented groups.

This research aims to make fundamental contributions to a theory of cross-task safe guardians that augment existing hardware platforms without manual tuning, monitor and optimally modify their nominal task-oriented control actions to satisfy constraints representing safety requirements, and accomplish these objectives under time-varying uncertainty. It achieves this aim by investigating data-efficient model learning algorithms that accurately track the dynamics of an interactive environment, as well as by designing adaptive controllers that safely adjust the control strategy according to newly learned dynamic models. A responsibility-based evolutionary adversarial learning approach is developed to enable the adaptive safe control algorithm to achieve optimal performance given limits on available resources. Evaluation of the safe guardian and intelligent optimizer approaches is achieved in simulation and experimentally using autonomous vehicles interacting with human-operated vehicles in different traffic conditions, as well as in space-sharing applications involving robot arm manipulators and other human or robotic agents.

More details can be found on the NSF Webpage.

ATI shared the story of our project on robotic weld bead removal project. Read the post here.

Congratulations Tianhao!

Check our paper

1. Model-free Safe Control for Zero-Violation Reinforcement Learning.

1. System Identification of Safe Controllers Using Inverse Optimization
2. Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithm (preprint)

Learn more of our recent work through the mini workshop.

0:00 Tianhao Wei: Robotic safety with Neural Network Components

11:50 Ruixuan Liu: Towards safe human-robot collaboration

25:40 Rui Chen and Alvin Shek: Real-time Collaborative Robot Handling with Recurrent Encoder-Decoder Attentive Neural Process

39:06 Letian Wang: Hierarchical, Adaptable and Transferable Networks (HATN) forDriving Behavior Generation

Ruixuan finishes his master study on Data-Efficient Behavior Prediction for Safe Human-Robot Collaboration. His thesis can be found here.

The organizers reported the results of VNN-COMP’21 on 4th Workshop on Formal Methods for ML-Enabled Autonomous Systems (FoMLAS) affiliated with CAV 2021. The benchmarks used in the competition can be found here. The final slides can be found here. The final report will be out soon.

Flexible MPC-based Conflict Resolution Using Online Adaptive ADMM

Feasible Region-based Identification Using Duality

Distributed Motion Coordination Using Convex Feasible Set Based Model Predictive Control

Congratulations Jaskaran and Weiye!

Human Motion Prediction Using Adaptable Recurrent Neural Networks and Inverse Kinematics

Chase Noren, 2nd year PhD student in the lab, gets the prestigious Uber PhD Fellowship.

Tolerance-guided Policy Learning for Adaptable and Transferrable Delicate Industrial Insertion

Contact-Rich Trajectory Generation in Confined Environment using Iterative Convex Optimization

Multi-car Convex Feasible Set Algorithm in Trajectory Planning

Weiye’s paper Contact-Rich Trajectory Generation in Confined Environments Using Iterative Convex Optimization gets the Best Student Paper Finalist in DSCC 2020.

Experimental Evaluation of Human Motion Prediction Toward Safe and Efficient Human Robot Collaboration

Robust online model adaptation by EKF with exponential moving average and dynamic multi-epoch strategy (L4DC)

Why does symmetry cause deadlocks? (IFAC World Congress)

The project “Hierarchical Motion Planning for Efficient and Provably Safe Human-Robot Interactions” received 2019 Amazon Research Reward.

Tianhao presented the paper Safe control algorithms using energy functions: A unified framework, benchmark, and new directions in the 58th IEEE Conference on Decision and Control. The code is available here.

Siyan presented the paper Adaptable Human Intention and Trajectory Prediction for Human-Robot Collaboration in the AAAI Fall Symposium Series - AI for HRI. This work was done by Abulikemu and Siyan during their internships in ICL. More information can be found in the poster and the presentation slides.

The book “Designing Robot Behavior in Human-Robot Interactions” is available here.

Jaskaran Grover, 2nd year PhD student in the lab, gets the prestigious Uber Presidential Fellowship.

The Robotiq Blog posted an interview with Changliu (link). In this interview, Changliu talks cobot safety, the importance of having realistic expectations of what cobots can and can’t do and her vision of a manufacturing world after cobots.

We will organize a workshop at Stanford on June 7 and June 10 introducing the toolbox NeuralVerification.jl. Agenda can be found here.

We will present our paper “AGen - Adaptable generative prediction networks for autonomous driving” in IV2019 in June. Preprints will be available soon. This work is supported by Holomatic. We would like to thank our sponsor for their generous support.

A group of high school students and teachers from Choate visited the Intelligent Control Lab.

Our paper “NeuralVerification.jl: Algorithms for Verifying Deep Neural Networks” has won the Best Applied Paper award at the ICLR 2019 Workshop on Debugging Machine Learning Models. This paper will be presented at the workshop on May 6.

FANUC generously donated an industrial robot arm LR Mate 200iD/7L to the Intelligent Control Lab to support our research. We look forward to developing more pratical methodologies and industrial applications with the robot arm.

This paper surveys emerging algorithms to verify whether a deep neural network satisfies certain input-output properties. We will present the work at AAAI 2019 Spring Symposium Verification of Neural Networks (VNN19) on March 25, and ICLR 2019 Workshop on Debugging Machine Learning Models (DebugML-19) on May 6.