GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent

ACL 2025

Bin Xie1, Rui Shao1✉, Gongwei Chen1✉, Kaiwen Zhou2,
Yinchuan Li2, Jie Liu1, Min Zhang1, Liqiang Nie1
1Harbin Institute of Technology, Shenzhen    2Huawei Noah’s Ark Lab
✉ Corresponding author  
arXiv PDF Code

Abstract

GUI automation faces critical challenges in dynamic environments. MLLMs suffer from two key issues: misinterpreting UI components and outdated knowledge. Traditional fine-tuning methods are costly for app-specific knowledge updates. We propose GUI-explorer, a training-free GUI agent that incorporates two fundamental mechanisms: (1) Autonomous Exploration of Function-aware Trajectory. To comprehensively cover all application functionalities, we design a Function-aware Task Goal Generator that automatically constructs exploration goals by analyzing GUI structural information (e.g., screenshots and activity hierarchies). This enables systematic exploration to collect diverse trajectories. (2) Unsupervised Mining of Transition-aware Knowledge. To establish precise screen-operation logic, we develop a Transition-aware Knowledge Extractor that extracts effective screen-operation logic through unsupervised analysis the state transition of structured interaction triples (observation, action, outcome). This eliminates the need for human involvement in knowledge extraction. With a task success rate of 53.7% on SPA-Bench and 47.4% on AndroidWorld, GUI-explorer shows significant improvements over SOTA agents. It requires no parameter updates for new apps. GUI-explorer is open-sourced and publicly available at https://github.com/JiuTian-VL/GUI-explorer.

Multi-Apps Task

Open Google Chrome and search for today's weather in Shenzhen. Carefully observe the screen and record the current weather conditions. Then, in Markor, create a note named "today.md" and write the temperature read from the webpage into it.

Single-App Task

Get the search results for stay tonight near 'wembley stadium' for 1 adult. Add one result to wishlist. Confirm that this item is in the wishlist.

Overview framework of our GUI-explorer


We propose Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent (GUI-explorer). It synergizes two key components: (1) Autonomous Exploration of Function-aware Trajectory. To cover all potential functions of target applications, we design a Function-aware Task Goal Generator. This module automatically constructs function-aware exploration goals by analyzing structural information of the environment, including screenshots and activity lists from APK files. Through systematic exploration, we obtain diverse function-aware trajectories. (2) Unsupervised Mining of Transition-aware Knowledge. To establish precise operation logic, we develop a Transition-aware Knowledge Extractor. This component extracts effective operation logic through unsupervised analysis of state transitions from structured interaction triples (observation, action, outcome). This eliminates human involvement. Through multimodal state modeling incorporating visual patterns and semantic patterns, the extractor captures operation constraints and outcome dependencies, generating transition-aware knowledge with explicit action-effect correlations. Finally, by performing visual-semantic retrieval between current screen visuals and the knowledge vector store to construct Dynamic Guidance, it achieves two goals: suppressing the misinterpretation of UI components, and ensuring action proposals align with actual UI states. This approach facilitates precise, goal-oriented prompt generation. These prompts guide the agent in effectively understanding and interacting with GUI elements.

Experiment

Table 1: Main Result of GUI-explorer on SPA-Bench single-app English Level 3 tasks.


Table 2: Main Result of GUI-explorer on AndroidWorld tasks.


Table 3: Main Result of GUI-explorer on GUI-KRB.


Conclusion

We present GUI-explorer, a GUI agent designed to address two key challenges: misinterpretation of UI components and knowledge obsolescence. Our approach achieves this through autonomous exploration and transition-aware knowledge mining. Experimental results demonstrate our SOTA performance across major benchmarks. We introduce the GUI-KRB benchmark, which reveals fundamental limitations in current MLLMs' interface understanding capabilities. Our dynamic guidance mechanism effectively mitigates these limitations.

BibTeX

@inproceedings{xie2025gui,
    title={GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent}, 
    author={Bin Xie and Rui Shao and Gongwei Chen and Kaiwen Zhou and Yinchuan Li and Jie Liu and Min Zhang and Liqiang Nie},
    booktitle={Annual Meeting of the Association for Computational Linguistics (ACL)},
    year={2025}
}