HRI Demo for President Biden

During President Biden’s visit, we demonstrated our work on human-robot collaboration.

Our research is to build intelligent robots that interact with the world like humans so that they can better assist and collaborate with people. While there are various tasks that humans and robots can collaborate in a manufacturing environment, we set up a FANUC industrial robot to demonstrate human-robot handover. These kinds of tasks arise naturally when the human needs the robot’s assistance to fetch an object that is either out-of-reach, too heavy, or in some hazardous environment. Ruixuan Liu and Rui Chen participated in this demo.

There are two major challenges in this handover task: 1. safety, which is how to make sure the powerful robot does not hurt humans; 2. efficiency, that is how to correctly interpret the human’s intention to better assist the human. We developed intelligent perception, AI, and control software for the robot to achieve both safety and efficiency. The software contains a digital twin which tracks and supervises the real-time human and robot behaviors via a camera. The real-time distance and momentum between the human body and the robot arm are monitored. When the human gets close, a safety controller will be activated to move the robot away from the human. In this way, safety is guaranteed even when the robot is in idle mode. In terms of efficiency, when the robot sees the human’s palm up, it quickly infers that the human needs the screwdriver. Then the robot locates the screwdriver, picks it up, and hands it to the human. After the human finishes the task, he hands the screwdriver back to the robot and the robot then places it to the correct location for efficient tool management. This handover loop can be repeated as many times as the human desires without external reset operations. A pre-recorded video of the demo is shown below.

President Biden was very interested in this idea and asked how to best integrate this kind of method in real manufacturing environments. We explained that this can be used to support multi-purpose assembly lines where the human workers can rely on the robot to manage all the tools. We are collaborating with Chengtao Wen – a Senior Research Scientist at Siemens in the hopes of making the transition from prototype to fully commercialized automatic tool management for flexible assembly lines a seamless one.

Explaining the handover demo to President Biden

We would like to acknowledge the support from FANUC, Siemens, and Ford in this research. FANUC generously donated the robot arm to our lab. Siemens supported this research through a sponsored project “Task agnostic real-time perception and control with few-shot cross-platform adaptation”. And Ford supported this research through a University Research Program Award on “Safe uncaged industrial robots”.

Read more on CMU News, Twitter@POTUS.

January 29, 2022

NSF Career Award!

Our proposal “Toward Lifelong Safety of Autonomous Systems in Uncertain and Interactive Environments” gets selected by NSF Career Award.

This Faculty Early Career Development Program (CAREER) grant will fund research that enables autonomous systems to operate safely in close interaction with humans, as required, for example, in next generation manufacturing infrastructure, thereby promoting the progress of science, and advancing the national prosperity. Until recently, humans were physically separated from robots to prevent injuries and fatalities. Modern robotics focuses on humans and collaborative robotic systems working together on the same tasks. A safety hazard in such interactive environments is the occurrence of human errors. It is critical that safety conscious responses be programmed into collaborative robotic systems to guarantee safe behavior even when tasks or environments change. This project will develop a new algorithmic framework for safety assurance of autonomous robotic systems that aims for optimal performance when safety can be managed, anticipates and compensates for inevitable failures when it cannot, and learns from past mistakes. This framework will increase trustworthiness of autonomous systems while minimizing human efforts in deployment and maintenance, critical steps toward granting full autonomy to intelligent robots in uncertain and interactive environments, including such application domains as industrial robotics and autonomous driving. Through close integration of research and education, this project will contribute to new interdisciplinary training in robotics and autonomy, accessible dissemination of research in robot safety to the public, and opportunities for interactive learning through a remotely operated robotic platform. Partnerships with the Advanced Robotics for Manufacturing Institute, the Girls of Steel Robotics program, and the Choate Rosemary Hall college-preparatory school will be leveraged to provide opportunities for graduate student internships with small manufacturers and broaden participation in research of individuals from currently underrepresented groups.

This research aims to make fundamental contributions to a theory of cross-task safe guardians that augment existing hardware platforms without manual tuning, monitor and optimally modify their nominal task-oriented control actions to satisfy constraints representing safety requirements, and accomplish these objectives under time-varying uncertainty. It achieves this aim by investigating data-efficient model learning algorithms that accurately track the dynamics of an interactive environment, as well as by designing adaptive controllers that safely adjust the control strategy according to newly learned dynamic models. A responsibility-based evolutionary adversarial learning approach is developed to enable the adaptive safe control algorithm to achieve optimal performance given limits on available resources. Evaluation of the safe guardian and intelligent optimizer approaches is achieved in simulation and experimentally using autonomous vehicles interacting with human-operated vehicles in different traffic conditions, as well as in space-sharing applications involving robot arm manipulators and other human or robotic agents.

More details can be found on the NSF Webpage.

January 26, 2022

ATI's Article on Our Work

ATI shared the story of our project on robotic weld bead removal project. Read the post here.

December 9, 2021

Congratulations Tianhao for Passing Qualification Exam!

Congratulations Tianhao!

November 18, 2021

Weiye Presented Our Paper in CoRL 2021

Check our paper

  1. Model-free Safe Control for Zero-Violation Reinforcement Learning.

November 7, 2021

ICL presence at MECC 2021

  1. System Identification of Safe Controllers Using Inverse Optimization
  2. Safe Adaptation with Multiplicative Uncertainties Using Robust Safe Set Algorithm (preprint)

October 25, 2021

ICL Mini Workshop on Recent Projects

Learn more of our recent work through the mini workshop.

0:00 Tianhao Wei: Robotic safety with Neural Network Components

11:50 Ruixuan Liu: Towards safe human-robot collaboration

25:40 Rui Chen and Alvin Shek: Real-time Collaborative Robot Handling with Recurrent Encoder-Decoder Attentive Neural Process

39:06 Letian Wang: Hierarchical, Adaptable and Transferable Networks (HATN) forDriving Behavior Generation

August 14, 2021

Ruixuan's Master Thesis Talk on Data-Efficient Behavior Prediction for Safe Human-Robot Collaboration

Ruixuan finishes his master study on Data-Efficient Behavior Prediction for Safe Human-Robot Collaboration. His thesis can be found here.

August 14, 2021

Keynote Talk in the RSS R4P Workshop

July 19, 2021

2nd International Verification of Neural Networks Competition (VNN-COMP'21)

The organizers reported the results of VNN-COMP’21 on 4th Workshop on Formal Methods for ML-Enabled Autonomous Systems (FoMLAS) affiliated with CAV 2021. The benchmarks used in the competition can be found here. The final slides can be found here. The final report will be out soon.

July 18, 2021

Presentation Video in ECC, 2021

Flexible MPC-based Conflict Resolution Using Online Adaptive ADMM

Feasible Region-based Identification Using Duality

June 7, 2021

Presentation Video in ICRA, 2021

Distributed Motion Coordination Using Convex Feasible Set Based Model Predictive Control

May 31, 2021

Congratulations Jaskaran and Weiye for Passing Qualification Exam!

Congratulations Jaskaran and Weiye!

May 19, 2021

Presentation Video in ACC, 2021

Human Motion Prediction Using Adaptable Recurrent Neural Networks and Inverse Kinematics

April 19, 2021

Talk in Lorentz Workshop "Robust Artificial Intelligence"

January 31, 2021

Chase gets the Uber PhD Fellowship!

Chase Noren, 2nd year PhD student in the lab, gets the prestigious Uber PhD Fellowship.

December 7, 2020

Presentation Videos in CoRL, 2020

Tolerance-guided Policy Learning for Adaptable and Transferrable Delicate Industrial Insertion

November 13, 2020

Presentation Videos in DSCC, 2020

Contact-Rich Trajectory Generation in Confined Environment using Iterative Convex Optimization

Multi-car Convex Feasible Set Algorithm in Trajectory Planning

October 14, 2020

Weiye gets the DSCC 2020 Best Student Paper Finalist!

Weiye’s paper Contact-Rich Trajectory Generation in Confined Environments Using Iterative Convex Optimization gets the Best Student Paper Finalist in DSCC 2020.

October 3, 2020

Presentation Video in ACC, 2020

Experimental Evaluation of Human Motion Prediction Toward Safe and Efficient Human Robot Collaboration

June 23, 2020

Presentation Videos in L4DC, IFAC World Congress, ACC

Robust online model adaptation by EKF with exponential moving average and dynamic multi-epoch strategy (L4DC)

Why does symmetry cause deadlocks? (IFAC World Congress)

June 19, 2020

We received Amazon Research Reward!

The project “Hierarchical Motion Planning for Efficient and Provably Safe Human-Robot Interactions” received 2019 Amazon Research Reward.

June 19, 2020

50% Demo for Automatic Onsite Polishing

March 31, 2020

Tianhao presented safe control in CDC

Tianhao presented the paper Safe control algorithms using energy functions: A unified framework, benchmark, and new directions in the 58th IEEE Conference on Decision and Control. The code is available here.

December 10, 2019

Siyan presented adaptable human prediction in AI for HRI

Siyan presented the paper Adaptable Human Intention and Trajectory Prediction for Human-Robot Collaboration in the AAAI Fall Symposium Series - AI for HRI. This work was done by Abulikemu and Siyan during their internships in ICL. More information can be found in the poster and the presentation slides.

November 7, 2019

We published a new book with CRC Press!

The book “Designing Robot Behavior in Human-Robot Interactions” is available here.

alt book

October 10, 2019

Jaskaran gets the Uber Presidential Fellowship!

Jaskaran Grover, 2nd year PhD student in the lab, gets the prestigious Uber Presidential Fellowship.

September 1, 2019

Tutorial on NeuralVerification.jl available on YouTube

August 4, 2019

The Difference Between Technology and Fantasy

The Robotiq Blog posted an interview with Changliu (link). In this interview, Changliu talks cobot safety, the importance of having realistic expectations of what cobots can and can’t do and her vision of a manufacturing world after cobots.

July 13, 2019

CARS Workshop on NeuralVerification.jl

We will organize a workshop at Stanford on June 7 and June 10 introducing the toolbox NeuralVerification.jl. Agenda can be found here.

May 16, 2019

Oral presentation in the 2019 IEEE Intelligent Vechicle Symposium

We will present our paper “AGen - Adaptable generative prediction networks for autonomous driving” in IV2019 in June. Preprints will be available soon. This work is supported by Holomatic. We would like to thank our sponsor for their generous support.

May 15, 2019

Visitors from Choate Rosemary Hall

A group of high school students and teachers from Choate visited the Intelligent Control Lab.

May 10, 2019

Our ICLR workshop paper received the Best Applied Paper Award

Our paper “NeuralVerification.jl: Algorithms for Verifying Deep Neural Networks” has won the Best Applied Paper award at the ICLR 2019 Workshop on Debugging Machine Learning Models. This paper will be presented at the workshop on May 6.

April 28, 2019

We received a generous donation from FANUC!

FANUC generously donated an industrial robot arm LR Mate 200iD/7L to the Intelligent Control Lab to support our research. We look forward to developing more pratical methodologies and industrial applications with the robot arm.

April 15, 2019

Webinar "Algorithms for Verifying Deep Neural Networks" available on YouTube

April 2, 2019

Preprint "Algorithms for Verifying Deep Neural Networks" available on ArXiv

This paper surveys emerging algorithms to verify whether a deep neural network satisfies certain input-output properties. We will present the work at AAAI 2019 Spring Symposium Verification of Neural Networks (VNN19) on March 25, and ICLR 2019 Workshop on Debugging Machine Learning Models (DebugML-19) on May 6.

March 17, 2019