Koopman Theory for Robot Learning and Control

We develop Koopman-based methods that turn nonlinear robot dynamics into useful linear representations for scalable learning, prediction, control, and safety.

2024–2026

Overview

Robots are governed by nonlinear dynamics, especially when they walk, make contact, manipulate objects, or operate near obstacles. These dynamics are difficult to model accurately and can make real-time optimization expensive. Koopman operator theory offers a different perspective: instead of directly linearizing a nonlinear system around one operating point, it lifts the system into a richer space of observable features, where its evolution can be approximated by linear dynamics. Our research asks how this idea can become a practical foundation for robot learning and control. We develop data-driven Koopman representations that can be learned from vision and robot trajectories, refined continually as new data arrive, scaled to high-dimensional systems, and combined with safety constraints. This creates a bridge between expressive learned models and efficient, well-understood linear control tools such as model predictive control and quadratic programming.

Koopman Overview

Research Directions

Learning Koopman Representations from Vision

Dexterous manipulation involves complex interactions between a robot hand and the objects in a scene. Classical Koopman methods often assume access to ground-truth object states, but those states may be unavailable outside carefully instrumented environments. KOROL learns visually interpretable, task-relevant object features directly from images and uses a Koopman operator to roll those features and robot states forward in time. The learned representation supports trajectory propagation without requiring ground-truth object states at runtime, extending Koopman-based manipulation from simulation to real-world visual observations.

Publications:

[C84] KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation
Hongyi Chen, Abulikemu Abuduweili, Aviral Agrawal, Yunhai Han, Harish Ravichandar, Changliu Liu and Jeffrey Ichnowski
Conference on Robot Learning, 2024
Citation Formats:
```
    
```
Abstract:

Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between objects and multi-fingered hands. Koopman operators have emerged as a robust method for modeling such nonlinear dynamics within a linear framework. However, current methods rely on runtime access to ground-truth (GT) object states, making them unsuitable for vision-based practical applications. Unlike image-to-action policies that implicitly learn visual features for control, we use a dynamics model, specifically the Koopman operator, to learn visually interpretable object features critical for robotic manipulation within a scene. We construct a Koopman operator using object features predicted by a feature extractor and utilize it to auto-regressively advance system states. We train the feature extractor to embed scene information into object features, thereby enabling the accurate propagation of robot trajectories. We evaluate our approach on simulated and real-world robot tasks, with results showing that it outperformed the model-based imitation learning NDP by 1.08 and the image-to-action Diffusion Policy by 1.16. The results suggest that our method maintains task success rates with learned features and extends applicability to real-world manipulation without GT object states. Project video and code are available at: https://github.com/hychen-naza/KOROL.

Continual Koopman Learning for Legged Robots

Koopman Learning and Control for High-Dimensional Robots Image description

Humanoids and quadrupeds have high-dimensional nonlinear dynamics that change across robots, motions, and terrains. A fixed lifted model may therefore be too limited or become inaccurate under a domain shift. Our continual learning approach progressively expands both the training dataset and the latent representation, refining the Koopman dynamics as new experience becomes available. The method provides monotonic convergence of the linear approximation error and enables simple linear model predictive controllers to control Unitree G1, H1, A1, and Go2 robots as well as ANYmal D across diverse terrains.

Publications:

[C95] Continual Learning and Lifting of Koopman Dynamics for Linear Control of Legged Robots
Feihan Li, Abulikemu Abuduweili, Yifan Sun, Rui Chen, Weiye Zhao and Changliu Liu
Learning for Dynamics and Control Conference, 2025
Citation Formats:
```
    
```
Abstract:

The control of legged robots, particularly humanoid and quadruped robots, presents significant challenges due to their high-dimensional and nonlinear dynamics. While linear systems can be effectively controlled using methods like Model Predictive Control (MPC), the control of nonlinear systems remains complex. One promising solution is the Koopman Operator, which approximates nonlinear dynamics with a linear model, enabling the use of proven linear control techniques. However, achieving accurate linearization through data-driven methods is difficult due to issues like approximation error, domain shifts, and the limitations of fixed linear state-space representations. These challenges restrict the scalability of Koopman-based approaches. This paper addresses these challenges by proposing a continual learning algorithm designed to iteratively refine Koopman dynamics for high-dimensional legged robots. The key idea is to progressively expand the dataset and latent space dimension, enabling the learned Koopman dynamics to converge towards accurate approximations of the true system dynamics. Theoretical analysis shows that the linear approximation error of our method converges monotonically. Experimental results demonstrate that our method achieves high control performance on robots like Unitree G1/H1/A1/Go2 and ANYmal D, across various terrains using simple linear MPC controllers. This work is the first to successfully apply linearized Koopman dynamics for locomotion control of high-dimensional legged robots, enabling a scalable model-based control solution.

Video:

Safe Whole-Body Control

Learning a useful dynamics model is only part of the control problem: a robot must also track its task while respecting safety constraints in real time. We present KMPC, which integrates the learned Koopman neural dynamics with the Safe Set Algorithm so that tracking and safety are handled together in a single quadratic program. The lifted linear representation keeps the online optimization efficient, while the unified formulation avoids adding a separate safety filter after the task controller. Experiments on a Kinova Gen3 manipulator and a Unitree Go2 quadruped demonstrate accurate tracking and obstacle avoidance for strongly nonlinear systems.

Application to Robotic Arms

Application to Legged Robots

The lifted linear structure also makes hardware deployment inexpensive. Rather than retraining the neural embedding, we collect a small amount of hardware data and fine-tune only the Koopman matrices A and B, since the dominant simulation-to-hardware mismatch, arising from actuation differences and unmodeled dynamics, is largely captured by the linear dynamics in the lifted space. This keeps the learned latent coordinates intact while substantially reducing prediction error on the real robot. The same unified safe controller therefore transfers directly to the actual robot without retraining the embedding.

Sim-to-Real Transfer

Publications:

[C109] Whole-Body Safe Control of Robotic Systems with Koopman Neural Dynamics
Sebin Jung, Abulikemu Abuduweili, Jiaxing Li and Changliu Liu
IEEE International Conference on Robotics and Automation, 2026
Citation Formats:
```
    
```
Abstract:

Controlling robots with strongly nonlinear, high-dimensional dynamics remains challenging, as direct nonlinear optimization with safety constraints is often intractable in real time. The Koopman operator offers a way to represent nonlinear systems linearly in a lifted space, enabling the use of efficient linear control. We propose a data-driven framework that learns a Koopman embedding and operator from data, and integrates the resulting linear model with the Safe Set Algorithm (SSA). This allows the tracking and safety constraints to be solved in a single quadratic program (QP), ensuring feasibility and optimality without a separate safety filter. We validate the method on a Kinova Gen3 manipulator and a Go2 quadruped, showing accurate tracking and obstacle avoidance.

Scaling Laws for Neural Koopman Operators

Data-driven Koopman models depend on two coupled resources: the amount of training data and the dimension of the lifted space. We study how these quantities should scale together by decomposing approximation error into sampling error and projection error. The resulting theory connects dataset size and latent dimension to model accuracy and downstream control quality. We also introduce covariance and inverse-control regularizers that stabilize learned features and align the representation with physical actuation. Experiments across six robotic environments turn these results into practical guidance for allocating data and model capacity.

Publications:

[U] Scaling Law of Neural Koopman Operators
Abulikemu Abuduweili, Yuyang Pang, Feihan Li and Changliu Liu
arXiv:2602.19943, 2026
Citation Formats:
```
    
```

Unifying Goal

Together, these projects develop a Koopman pipeline that spans perception, continual model learning, control, and safety. The long-term goal is to give robots models that retain the computational advantages of linear systems while remaining expressive and adaptable enough for nonlinear, high-dimensional, real-world operation.

Abstract:

Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between objects and multi-fingered hands. Koopman operators have emerged as a robust method for modeling such nonlinear dynamics within a linear framework. However, current methods rely on runtime access to ground-truth (GT) object states, making them unsuitable for vision-based practical applications. Unlike image-to-action policies that implicitly learn visual features for control, we use a dynamics model, specifically the Koopman operator, to learn visually interpretable object features critical for robotic manipulation within a scene. We construct a Koopman operator using object features predicted by a feature extractor and utilize it to auto-regressively advance system states. We train the feature extractor to embed scene information into object features, thereby enabling the accurate propagation of robot trajectories. We evaluate our approach on simulated and real-world robot tasks, with results showing that it outperformed the model-based imitation learning NDP by 1.08 and the image-to-action Diffusion Policy by 1.16. The results suggest that our method maintains task success rates with learned features and extends applicability to real-world manipulation without GT object states. Project video and code are available at: https://github.com/hychen-naza/KOROL.

Abstract:

The control of legged robots, particularly humanoid and quadruped robots, presents significant challenges due to their high-dimensional and nonlinear dynamics. While linear systems can be effectively controlled using methods like Model Predictive Control (MPC), the control of nonlinear systems remains complex. One promising solution is the Koopman Operator, which approximates nonlinear dynamics with a linear model, enabling the use of proven linear control techniques. However, achieving accurate linearization through data-driven methods is difficult due to issues like approximation error, domain shifts, and the limitations of fixed linear state-space representations. These challenges restrict the scalability of Koopman-based approaches. This paper addresses these challenges by proposing a continual learning algorithm designed to iteratively refine Koopman dynamics for high-dimensional legged robots. The key idea is to progressively expand the dataset and latent space dimension, enabling the learned Koopman dynamics to converge towards accurate approximations of the true system dynamics. Theoretical analysis shows that the linear approximation error of our method converges monotonically. Experimental results demonstrate that our method achieves high control performance on robots like Unitree G1/H1/A1/Go2 and ANYmal D, across various terrains using simple linear MPC controllers. This work is the first to successfully apply linearized Koopman dynamics for locomotion control of high-dimensional legged robots, enabling a scalable model-based control solution.

Abstract:

Controlling robots with strongly nonlinear, high-dimensional dynamics remains challenging, as direct nonlinear optimization with safety constraints is often intractable in real time. The Koopman operator offers a way to represent nonlinear systems linearly in a lifted space, enabling the use of efficient linear control. We propose a data-driven framework that learns a Koopman embedding and operator from data, and integrates the resulting linear model with the Safe Set Algorithm (SSA). This allows the tracking and safety constraints to be solved in a single quadratic program (QP), ensuring feasibility and optimality without a separate safety filter. We validate the method on a Kinova Gen3 manipulator and a Go2 quadruped, showing accurate tracking and obstacle avoidance.