16-883 Special Topics: Provably Safe Robotics

Time: Tuesday and Thursday 9:30 am — 10:50 am
Location: NSH 1305
Instructor: Changliu Liu (cliu6@andrew.cmu.edu)
Office Hours: By appointment
Canvas: https://canvas.cmu.edu/courses/39520
Piazza: https://piazza.com/class/lr9ct0wql4l2n8/

Course Material

Slides and Notes

Selected Student Projects

• Jessie Fan. A Survey: Jailbreak Attacks and Their Evolution Against Defenses
• Kai Yun. Deloying Neural Safety Index for Safe Reactive Control
• Bo Ying Su. Safety Preference Learning from User Feedback using Tactile Sensors

Course Description

Safe autonomy is critical in many application domains, ensuring the safety of both the ego robot and other interacting agents. This course covers:

• Designing safe robotic systems through planning, prediction, learning, and control.
• Verifying safety using methods like neural network verification, safety/reachability analysis, and multi-agent system analysis.

Learning Objectives

• Familiarize with tools and methodologies for designing and verifying safe robotic systems.
• Use these tools to enhance or verify existing designs.
• Develop new tools and methodologies for robot safety.

Assessment Structure

• Participation: 10 points
• Paper Reading: 55 points
  • Presentations: 30 points (15 points each)
  • Summaries: 25 points (2 points each)
• Project: 35 points
  • Proposal: 5 points
  • Presentation: 10 points
  • Final Report: 20 points

Policies

Class Attendance and Participation

Participation and attendance are critical. Notify the instructor beforehand if you must miss a class to arrange alternative ways to catch up.

Paper Presentation

• Students will present two papers.
• Each presentation: 25 minutes with a 15-minute discussion.
• Presentations should cover:
  • Major contributions
  • Problem formulation
  • Solution methodology
  • Results
  • Limitations and future work

Project

• Proposal (5 points): A 1-page abstract in IEEE format introducing the problem and outlining the plan.

• Final Report (20 points): A 6-page conference paper in IEEE format, presenting:
  • Introduction and motivation
  • Literature review
  • Problem formulation and solution approach
  • Results and analysis
  • Discussion on learnings and future directions
• Presentation (10 points): 10-minute presentation per team with the same structure as paper presentations.

Weekly Schedule

Week 1

• 1/16: Lecture 1 - Overview of the Course
• 1/18: Lecture 2 - Robot Safety in the Real World
• 1/19: Paper Presentation Sign-up

Week 2

• 1/23: Lecture 3 - Safe Control (Overview)
• 1/25: Lecture 4 - Safe Control (Dynamic Limits)

Week 3

• 1/30: Lecture 5 - Safe Control (Synthesis)
• 2/1: Paper Reading 1: Safe Control

Week 4

• 2/6: Lecture 6 - Safe Planning
• 2/8: Paper Reading 2: Safe Planning

Week 5

• 2/13: Lecture 7 - Safe Learning (Offline)
• 2/15: Lecture 8 - Safe Learning (Safety Monitor)

Week 6

• 2/20: Lecture 9 - Safe Learning (Constrained MDP)
• 2/22: Paper Reading 3: Safe Learning

Week 7

• 2/27: Lecture 10 - Safety in Foundation Models (Ziwei)
• 2/29: Paper Reading 4: Safety in Foundation Models
• 3/1: Project Proposal Due

Week 8: Spring Break

• 3/5: No Class
• 3/7: No Class

Week 9

• 3/12: Lecture 11 - Formal Methods in Robotics (Xusheng)
• 3/14: Paper Reading 5: Formal Methods in Robotics

Week 10

• 3/19: Lecture 12 - Safety in Multi-Agent Systems
• 3/21: Paper Reading 6: Safety in Multi-Agent Systems

Week 11

• 3/26: Office Hours
• 3/28: Lecture 13 - Overview of Neural Network Verification

Week 12

• 4/2: Lecture 14 - Symbolic Reachability
• 4/4: Lecture 16 - SOTA Tools (Tianhao)

Week 13

• 4/9: Lecture 15 - Optimization and Search for NN Verification
• 4/11: No Class

Week 14

• 4/16: Paper Reading 7: Neural Network Verification
• 4/18: Lecture 17 - Application of NN Verification

Week 15

• 4/23: Paper Reading 8: Application of NN Verification
• 4/25: Project Presentations

Paper Reading Topics

Paper Reading 1: Safe Control

• BarrierNet: Differentiable Control Barrier Functions for Learning of Safe Robot Control
• Neural Differentiable Integral Control Barrier Functions for Unknown Nonlinear Systems with Input Constraints

Paper Reading 2: Safe Planning

• Shortest Paths in Graphs of Convex Sets
• Fail-Safe Motion Planning for Online Verification of Autonomous Vehicles Using Convex Optimization

Paper Reading 3: Safe Learning

• Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning
• Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms

Paper Reading 4: Safety in Foundation Models

• Jailbroken: How Does LLM Safety Training Fail?
• Universal and Transferable Adversarial Attacks on Aligned Language Models

Paper Reading 5: Formal Methods in Robotics

• Simultaneous Task Allocation and Planning for Temporal Logic Goals in Heterogeneous Multi-Robot Systems
• Formal Methods for Control Synthesis: An Optimization Perspective

Paper Reading 6: Safety in Multi-Agent Systems

• Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates
• Deception Game: Closing the Safety-Learning Loop in Interactive Robot Autonomy

Paper Reading 7: Neural Network Verification

• Complete Verification via Multi-Neuron Relaxation Guided Branch-and-Bound (MN-BaB)
• A DPLL(T) Framework for Verifying Deep Neural Networks (NeuralSAT)

Paper Reading 8: Application of Neural Network Verification

• System-Level Safety Guard: Safe Tracking Control through Uncertain Neural Network Dynamics Models
• Simultaneous Synthesis and Verification of Neural Control Barrier Functions through Branch-and-Bound Verification-in-the-Loop Training

Additional Policies and Resources

Academic Integrity

Plagiarism and cheating are strictly prohibited. Refer to CMU’s academic integrity policy for details. Cite all sources appropriately.

Accommodations for Students with Disabilities

Contact the Office of Disability Resources at access@andrew.cmu.edu for support and accommodations.

Student Support

If you experience stress or difficult life events, reach out to Counseling and Psychological Services (CaPS) or call 412-268-2922.