Neural-Symbolic Robotics

Robots must turn human intent into reliable behavior in a physical world. This requires more than language understanding or end-to-end policy learning alone: robots need task structures that express what should happen, learned controllers that determine how to act, and mechanisms to adapt both at deployment time.

Our work develops a neural-symbolic stack for this problem. We represent long-horizon tasks through hierarchical temporal logic and task trees; use those structures to plan, allocate, and execute work across robots; learn symbolic automata directly from visuomotor trajectories; and reconfigure learned task structure at inference time to steer behavior toward new, physically feasible preferences.

Research Themes

1. Specify complex tasks through hierarchy

Flat temporal-logic formulas become lengthy, difficult to interpret, and expensive to solve as tasks grow. We introduced hierarchical temporal logic specifications with explicit syntax and semantics for composing task objectives across levels of abstraction. This representation is more expressive than flat specifications, easier for people to understand in user studies, and supports scalable decision making for multi-robot systems.

We developed complementary planning methods that integrate task allocation with planning: search over loosely coupled specification subspaces, decomposition into atomic subtasks and task networks, and hierarchical task-and-motion planning that connects symbolic objectives to feasible robot motion.

$$ \begin{aligned} L_1:\quad & \phi_{1}^{1} = \Diamond\!\left(\phi_{2}^{1} \wedge \Diamond\!\left(\phi_{2}^{2} \wedge \Diamond \phi_{2}^{3}\right)\right) \\ L_2:\quad & \phi_{2}^{1} = \Diamond\!\left(s_{\mathsf{cc}} \wedge \Diamond t_{\mathsf{cc}}\right) \\ & \phi_{2}^{2} = \Diamond\!\left(s_{\mathsf{dm}} \wedge \Diamond t_{\mathsf{dm}}\right) \wedge \Diamond\!\left(s_{\mathsf{st}} \wedge \Diamond t_{\mathsf{st}}\right) \\ & \phi_{2}^{3} = \Diamond\!\left(s_{\mathsf{ht}} \wedge \Diamond t_{\mathsf{ht}}\right) \end{aligned} $$

Publications

1. [J33] Simultaneous Task Allocation and Planning for Multi-Robots under Hierarchical Temporal Logic Specifications
   Xusheng Luo and Changliu Liu
   IEEE Transaction on Robotics, 2025
   

   Preprint Page Code Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Research in robotic planning with temporal logic specifications, such as syntactically co-safe Linear Temporal Logic (sc-LTL), has relied on single formulas. However, as task complexity increases, sc-LTL formulas become lengthy, making them difficult to interpret and generate, and straining the computational capacities of planners. To address this, we introduce a hierarchical structure to sc-LTL specifications with both syntax and semantics, proving it to be more expressive than flat counterparts. We conducted a user study that compared the flat sc-LTL with our hierarchical version and found that users could more easily comprehend complex tasks using the hierarchical structure. We develop a search-based approach to synthesize plans for multi-robot systems, achieving simultaneous task allocation and planning. This method approximates the search space by loosely interconnected sub-spaces, each corresponding to an sc-LTL specification. The search primarily focuses on a single sub-space, transitioning to another under conditions determined by the decomposition of automatons. We develop multiple heuristics to significantly expedite the search. Our theoretical analysis, conducted under mild assumptions, addresses completeness and optimality. Compared to existing methods used in various simulators for service tasks, our approach improves planning times while maintaining comparable solution quality.

   Video:
   

1. [J25] Decomposition-based Hierarchical Task Allocation and Planning for Multi-Robots under Hierarchical Temporal Logic Specifications
   Xusheng Luo, Shaojun Xu, Ruixuan Liu and Changliu Liu
   IEEE Robotics and Automation Letters, 2024
   

   Link Page Code Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Past research into robotic planning with temporal logic specifications, notably Linear Temporal Logic (LTL), was largely based on a single formula for individual or groups of robots. But with increasing task complexity, LTL formulas unavoidably grow lengthy, complicating interpretation and specification generation, and straining the computational capacities of the planners. A recent development has been the hierarchical representation of LTL (Luo et al., 2024) that contains multiple temporal logic specifications, providing a more interpretable framework. However, the proposed planning algorithm assumes the independence of robots within each specification, limiting their application to multi-robot coordination with complex temporal constraints. In this work, we formulated a decomposition-based hierarchical framework. At the high level, each specification is first decomposed into a set of atomic sub-tasks. We further infer the temporal relations among the sub-tasks of different specifications to construct a task network. Subsequently, a Mixed Integer Linear Program is used to assign sub-tasks to various robots. At the lower level, domain-specific controllers are employed to execute sub-tasks. Our approach was experimentally applied to domains of navigation and manipulation. The simulation demonstrated that our approach can find better solutions using less runtimes.

   Video:
   

1. [C99] Hierarchical Temporal Logic Task and Motion Planning for Multi-Robot Systems
   Zhongqi Wei, Xusheng Luo and Changliu Liu
   Robotics: Science and Systems, 2025
   

   Preprint Code Video Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   Task and motion planning (TAMP) for multi-robot systems, which integrates discrete task planning with continuous motion planning, remains a challenging problem in robotics. Existing TAMP approaches often struggle to scale effectively for multi-robot systems with complex specifications, leading to infeasible solutions and prolonged computation times. This work addresses the TAMP problem in multi-robot settings where tasks are specified using expressive hierarchical temporal logic and task assignments are not pre-determined. Our approach leverages the efficiency of hierarchical temporal logic specifications for task-level planning and the optimization-based graph of convex sets method for motion-level planning, integrating them within a product graph framework. At the task level, we convert hierarchi- cal temporal logic specifications into a single graph, embedding task allocation within its edges. At the motion level, we represent the feasible motions of multiple robots through convex sets in the configuration space, guided by a sampling-based motion planner. This formulation allows us to define the TAMP problem as a shortest path search within the product graph, where efficient convex optimization techniques can be applied. We prove that our approach is both sound and complete under mild assumptions. To enhance scalability, we introduce a pruning heuristic that reduces the product graph size, enabling efficient planning for high-dimensional multi-robot systems. Additionally, we extend our framework to cooperative pick-and-place tasks involving object handovers between robots. We evaluate our method across various high-dimensional multi-robot scenarios, including simulated and real-world environments with quadrupeds, robotic arms, and automated conveyor systems. Our results show that our approach outperforms existing methods in execution time and solution optimality while effectively scaling with task complexity.

   Video:
   

2. Translate human instructions into executable structure

Natural-language instructions often leave temporal order, logical alternatives, and multi-robot responsibilities implicit. We bridge this gap by translating instructions into hierarchical task structures and then into hierarchical Linear Temporal Logic (LTL). The hierarchy retains intent while exposing ordered robot actions that standard planners can execute.

NL2HLTL2Plan uses language models in two stages: it first constructs a hierarchical task tree, then converts subtasks into LTL formulas that form an executable hierarchy. The resulting specifications support more complex instructions while improving task-allocation and planning success and cost. Earlier work established the foundation for obtaining hierarchy directly from human instructions with LLMs.

Publications

1. [J35] NL2HLTL2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Specifications
   Shaojun Xu, Xusheng Luo, Yutong Huang, Letian Leng, Ruixuan Liu and Changliu Liu
   IEEE Robotics and Automation Letters, 2025
   

   Preprint Page Code Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   To enable non-experts to specify long-horizon, multi-robot collaborative tasks, language models are increasingly used to translate natural language commands into formal specifications. However, because translation can occur in multiple ways, such translations may lack accuracy or lead to inefficient multi-robot planning. Our key insight is that concise hierarchical specifications can simplify planning while remaining straightforward to derive from human instructions. We propose Nl2Hltl2Plan, a framework that translates natural language commands into hierarchical Linear Temporal Logic (LTL) and solves the corresponding planning problem. The translation involves two steps leveraging Large Language Models (LLMs). First, an LLM transforms instructions into a Hierarchical Task Tree, capturing logical and temporal relations. Next, a fine-tuned LLM converts sub-tasks into flat LTL formulas, which are aggregated into hierarchical specifications, with the lowest level corresponding to ordered robot actions. These specifications are then used with off-the-shelf planners. Our Nl2Hltl2Plan demonstrates the potential of LLMs in hierarchical reasoning for multi-robot task planning. Evaluations in simulation and real-world experiments with human participants show that Nl2Hltl2Plan outperforms existing methods, handling more complex instructions while achieving higher success rates and lower costs in task allocation and planning.

1. [W] Obtaining hierarchy from human instructions: an llms-based approach
   Xusheng Luo, Shaojun Xu and Changliu Liu
   CoRL 2023 Workshop on Learning Effective Abstractions for Planning (LEAP), 2023
   

   Cite

   Citation Formats: BibTeX Plain Text

3. Learn symbolic structure from visuomotor experience

Hand-designed symbols can limit robot learning. We instead learn an emergent neural automaton policy from visuomotor trajectories. The model discovers a high-level state-machine skeleton that captures task progress, together with a continuous residual controller that realizes each stage in the physical world. This makes latent task structure explicit without requiring manually predefined symbolic states.

Publication

1. [C112] Emergent Neural Automaton Policies: Learning Symbolic Structure from Visuomotor Trajectories
   Yiyuan Pan, Xusheng Luo, Hanjiang Hu, Peiqi Yu and Changliu Liu
   Robotics: Science and Systems, 2026
   

   Preprint Page Cite

   Citation Formats: BibTeX Plain Text

4. Reconfigure task structure to steer learned behavior

Learned policies should follow new user preferences at test time without costly retraining. ReStruct steers a frozen neural automaton policy by modifying its symbolic task skeleton and updating its action priors, while preserving the learned low-level controller. This allows robots to satisfy object-centric preferences and temporal-logic constraints through physically aware task-structure reconfiguration.

Publication

1. [U] Inference-Time Robot Behavior Steering through Physically-Aware Reconfiguration of Task-Structure
   Yiyuan Pan, Hanjiang Hu, Shangtao Li, Xusheng Luo and Changliu Liu
   arXiv:2606.26588, 2026
   

   Preprint Abstract Cite

   Citation Formats: BibTeX Plain Text

   Abstract:
   

   A central challenge in deploying learned robot policies is inference-time behavior steering: redirecting a policy at test time to satisfy user preferences not anticipated during training, without retraining. Existing methods fail in two modes: end-to-end methods require fine-tuning or expert-level guidance, while neuro-symbolic methods rely on predefined symbols whose edits can result in logically reasonable but physically infeasible plans. To address this challenge, we propose ReStruct, which builds upon a neural automaton policy that decomposes a visuomotor policy into a high-level state-machine skeleton capturing task structure and a low-level continuous controller represented as a residual policy. Specifically, ReStruct adopts the automaton to represent the preference and incorporates it into the skeleton through a synchronous product, thereby reconfiguring the task structure. With the controller kept frozen, the action priors provided by the skeleton are updated accordingly to enable physically-aware control under a modified task structure. Extensive experiments from simulation and real-world show that ReStruct steers a wide range of preferences, from object-centric specifications to temporal-logic constraints, and after steering surpasses existing methods, exceeding VLA models in both task success and preference-following by up to 25%.

From Language to Physical Behavior

Human instruction

Describe desired outcome, alternatives, order, and collaboration.

Hierarchical task structure

Infer temporal and logical relations; compile intent into executable specifications.

Planning and control

Allocate subtasks, synthesize task-and-motion plans, and execute with robot controllers.

Learned automata and steering

Discover structure from trajectories, then reconfigure it at inference time to meet new preferences while respecting physical feasibility.

Together, these directions make symbolic structure a living interface between people, planners, and learned robot policies—not a fixed description written once before deployment.