Talk@WashU - Scaling Up Robot Reasoning Using Hierarchical Representations

Changliu gave a talk at WashU, titled Scaling Up Robot Reasoning Using Hierarchical Representations.

Hierarchical reasoning is an efficient symbolic reasoning approach that breaks complex problems into smaller, more manageable pieces, leading to faster and more optimal solutions. In this talk, I will demonstrate how introducing hierarchies enhance robot reasoning. For safety critical control under uncertainty, there is always a trade-off between safety (requiring faster response) and performance (requiring long horizon reasoning). We introduce a hierarchical long-short term safety framework that allows two safety control mechanisms to operate at different frequencies, thus advancing the safety-performance tradeoff. In data-driven prediction of other agents’ behaviors, incorporating hierarchy into neural network design improves generalizability and transferability across complex scenarios, such as vehicle trajectory prediction at intersections. For complex long-sequence planning, large linear temporal logic (LTL) specifications are challenging for users to specify and interpret, as well as for planners to reason over. We propose a new specification called hierarchical linear temporal logic (HLTL), which is not only more interpretable, but also easier to specify. Moreover, planning over HLTL is more efficient and scalable for multi-robot long-sequence planning. I will conclude the talk with future perspectives on leveraging hierarchy to enhance reasoning using large language models.

This talk highlights the following work from ICL:

[J25] Decomposition-based Hierarchical Task Allocation and Planning for Multi-Robots under Hierarchical Temporal Logic Specifications
Xusheng Luo, Shaojun Xu, Ruixuan Liu and Changliu Liu
IEEE Robotics and Automation Letters, 2024
Citation Formats:
```
    
```
Abstract:

Past research into robotic planning with temporal logic specifications, notably Linear Temporal Logic (LTL), was largely based on a single formula for individual or groups of robots. But with increasing task complexity, LTL formulas unavoidably grow lengthy, complicating interpretation and specification generation, and straining the computational capacities of the planners. A recent development has been the hierarchical representation of LTL (Luo et al., 2024) that contains multiple temporal logic specifications, providing a more interpretable framework. However, the proposed planning algorithm assumes the independence of robots within each specification, limiting their application to multi-robot coordination with complex temporal constraints. In this work, we formulated a decomposition-based hierarchical framework. At the high level, each specification is first decomposed into a set of atomic sub-tasks. We further infer the temporal relations among the sub-tasks of different specifications to construct a task network. Subsequently, a Mixed Integer Linear Program is used to assign sub-tasks to various robots. At the lower level, domain-specific controllers are employed to execute sub-tasks. Our approach was experimentally applied to domains of navigation and manipulation. The simulation demonstrated that our approach can find better solutions using less runtimes.

Video:

[U] Simultaneous Task Allocation and Planning for Multi-Robots under Hierarchical Temporal Logic Specifications
Xusheng Luo and Changliu Liu
arXiv:2401.04003, 2024
Citation Formats:
```
    
```
Abstract:

Research in robotic planning with temporal logic specifications, such as syntactically co-safe Linear Temporal Logic (sc-LTL), has relied on single formulas. However, as task complexity increases, sc-LTL formulas become lengthy, making them difficult to interpret and generate, and straining the computational capacities of planners. To address this, we introduce a hierarchical structure to sc-LTL specifications with both syntax and semantics, proving it to be more expressive than flat counterparts. We conducted a user study that compared the flat sc-LTL with our hierarchical version and found that users could more easily comprehend complex tasks using the hierarchical structure. We develop a search-based approach to synthesize plans for multi-robot systems, achieving simultaneous task allocation and planning. This method approximates the search space by loosely interconnected sub-spaces, each corresponding to an sc-LTL specification. The search primarily focuses on a single sub-space, transitioning to another under conditions determined by the decomposition of automatons. We develop multiple heuristics to significantly expedite the search. Our theoretical analysis, conducted under mild assumptions, addresses completeness and optimality. Compared to existing methods used in various simulators for service tasks, our approach improves planning times while maintaining comparable solution quality.

[W] Obtaining hierarchy from human instructions: an llms-based approach
Xusheng Luo, Shaojun Xu and Changliu Liu
CoRL 2023 Workshop on Learning Effective Abstractions for Planning (LEAP), 2023
Citation Formats:
```
    
```

[J16] A hierarchical long short term safety framework for efficient robot manipulation under uncertainty
Suqin He, Weiye Zhao, Chuxiong Hu, Yu Zhu and Changliu Liu
Robotics and Computer-Integrated Manufacturing, 2023
Citation Formats:
```
    
```

[W] Hierarchical Adaptable and Transferable Networks (HATN) for Driving Behavior Prediction
Letian Wang, Yeping Hu, Liting Sun, Wei Zhan, Masayoshi Tomizuka and Changliu Liu
NeurIPS workshop on Machine Learning for Autonomous Driving, 2021
Citation Formats:
```
    
```
Abstract:

When autonomous vehicles still struggle to solve challenging situations during on-road driving, humans have long mastered the essence of driving with efficient transferable and adaptable driving capability. By mimicking humans’ cognition model and semantic understanding during driving, we present HATN, a hierarchical framework to generate high-quality driving behaviors in multi-agent dense-traffic environments. Our method hierarchically consists of a high-level intention identification and low-level action generation policy. With the semantic sub-task definition and generic state representation, the hierarchical framework is transferable across different driving scenarios. Besides, our model is also able to capture variations of driving behaviors among individuals and scenarios by an online adaptation module. We demonstrate our algorithms in the task of trajectory prediction for real traffic data at intersections and roundabouts, where we conducted extensive studies of the proposed method and demonstrated how our method outperformed other methods in terms of prediction accuracy and transferability.

[W] Online Adaptation of Neural Network Models by Modified Extended Kalman Filter for Customizable and Transferable Driving Behavior Prediction
Letian Wang, Yeping Hu and Changliu Liu
AAAI Workshop HCSSL, 2021
Citation Formats:
```
    
```
Abstract:

High fidelity behavior prediction of human drivers is crucial for efficient and safe deployment of autonomous vehicles, which is challenging due to the stochasticity, heterogeneity, and time-varying nature of human behaviors. On one hand, the trained prediction model can only capture the motion pattern in an average sense, while the nuances among individuals can hardly be reflected. On the other hand, the prediction model trained on the training set may not generalize to the testing set which may be in a different scenario or data distribution, resulting in low transferability and generalizability. In this paper, we applied a τ-step modified Extended Kalman Filter parameter adaptation algorithm (MEKFλ) to the driving behavior prediction task, which has not been studied before in literature. With the feedback of the observed trajectory, the algorithm is applied to neural-network-based models to improve the performance of driving behavior predictions across different human subjects and scenarios. A new set of metrics is proposed for systematic evaluation of online adaptation performance in reducing the prediction error for different individuals and scenarios. Empirical studies on the best layer in the model and steps of observation to adapt are also provided.