Researchers have introduced ContinualSkillBench, a dynamic evaluation framework designed to assess the ability of Large Language Model (LLM) agents to evolve their capabilities through in-context continual skill learning. The benchmark comprises five representative domains, each consisting of 100 interconnected subtasks that increase in difficulty and provide opportunities for cross-task skill reuse. The study investigates whether sequential task execution and explicit skill maintenance actually enhance task-solving performance. Experimental results indicate that while sequential execution generally improves performance, the degree of improvement varies significantly across different models and domains. Interestingly, the data suggests that in-context learning performs comparably to explicit skill maintenance on average, implying that performance gains often stem from adapting to prior context and feedback rather than the abstraction of reusable skills. While explicit skills offer selective advantages for tasks requiring precise outputs or repeatable procedures, less capable models frequently suffer from accumulating large, fragmented collections of task-specific skills. Ultimately, the findings reveal that while current in-context evolution mechanisms facilitate continual adaptation, LLM agents still struggle to consolidate experiences into robust, transferable, and consolidated skills.

Read original