Continually Evolving Skill Knowledge in Vision Language Action Model
* Corresponding author: Hesheng Wang
Abstract
Vision-language-action (VLA) models show promising knowledge accumulation ability through pretraining, yet continual learning remains challenging. Existing continual imitation learning (CIL) methods often require additional parameters or modules, limiting scalability for large VLA models. We propose Stellar VLA, a knowledge-driven CIL framework without parameter growth, featuring two variants: T-Stellar for task-centric modeling and \modify{TS-Stellar for hierarchical modeling of reusable skills across tasks}. Stellar VLA jointly optimizes task representations and a self-evolving knowledge space, with \modify{knowledge-posterior-guided} expert routing \modify{using} knowledge relations and Top-K semantic embeddings for task specialization. Experiments on LIBERO demonstrate strong performance against VLA and CIL baselines with only 1% replay. Dual-arm real-world experiments validate knowledge transfer across embodiments and scenes. TS-Stellar excels in hierarchical manipulation, while visualizations demonstrate knowledge retention and task discovery.
Motivation
Lifelong VLA agents must learn new tasks without forgetting old ones. Existing CIL techniques help in smaller policies, but scaling them to large VLAs often incurs heavy training and storage costs from growing task-specific modules. Replay-based methods are promising, yet they leave a key question open: can continual learning improve through better task-skill knowledge modeling, not larger model growth? Our Stellar VLA answers this question by learning a task/task-skill knowledge space and using knowledge-guided expert routing for adaptive specialization, enabling strong continual learning with lower replay dependence in both simulation and real-world manipulation.
Method
Figure 1. Overall architecture of Stellar VLA.
Stellar VLA learns task-centric latent representations and an evolving knowledge space to capture task and skill relationships for continual imitation learning. The learned knowledge guides expert routing in the MoE action head, balancing parameter sharing and specialization without increasing model size. We instantiate this framework with two knowledge models: T-Stellar for task-level modeling and TS-Stellar for hierarchical task-skill modeling that captures reusable skills across tasks.
Real-World Experiments
To further evaluate real-world effectiveness, we conduct dual-arm manipulation experiments across seven representative tasks: (1) Pick up Stick, (2) Handover Stick, (3) Pick up Bag, (4) Handover Toy, (5) Pick Toy Place Plate, (6) Pick Both Object, and (7) Pull Stick from Bag. These tasks cover grasping, handover, coordinated placement, and contact-rich interaction, providing a comprehensive benchmark for real-world generalization and policy robustness. Below videos present the performance of our method and π0.5 after completing continual learning across seven tasks, evaluated on all the tasks learned before. All videos are shown at 10x speed.
1. Pick up Stick
2. Handover Stick
3. Pick up Bag
4. Handover Toy
5. Pick Toy Place Plate
6. Pick Both Object
7. Pull Stick from Bag
Figure 2. Success rate curves on real-world manipulation tasks.
Table 1. Quantitative success-rate comparison in real-world experiments.
Simulation Results
We evaluate our models on the LIBERO benchmark, including LIBERO-goal, LIBERO-long, and LIBERO-30*, a 30-task subset of LIBERO-90 (see Appendix D.1 for construction), to assess performance under diverse goals, long-horizon reasoning, and multi-task settings. To comprehensively evaluate continual learning capability, we compare our method against both advanced VLA baselines and classical robotic continual imitation learning approaches.
Table 2. Success-rate curves across LIBERO simulation benchmarks.
To illustrate continual skill evolution in Stellar VLA, we visualize the learned knowledge space on LIBERO-long. The t-SNE results show that T-Stellar forms clear task-level clusters, while TS-Stellar reveals shared structure through subskill-level connectivity. For example, deep and light green clusters partially overlap because both contain the shared subgoal "put moka on stove." In contrast, VLA baselines tend to either mix latent representations or fail to preserve meaningful task relationships in feature space.
Figure 3. t-SNE visualization of learned representations in simulation tasks.
BibTeX
@article{wu2025continually,
title={Continually evolving skill knowledge in vision language action model},
author={Wu, Yuxuan and Wang, Guangming and Yang, Zhiheng and Deng, Tianchen and Yao, Maoqing and Sheil, Brian and Wang, Hesheng},
journal={arXiv preprint arXiv:2511.18085},
year={2025}
}