Verified project record

TencentQQGYLab/AppAgent

AppAgent is a multimodal LLM-based framework that operates Android smartphone apps through human-like interactions such as tapping and swiping, bypassing the need for system back-end access. It relies on a vision-language model and an Android device or emulator, making it suited for developers building app automation agents through autonomous exploration or human demonstration.

Project overview

Project type: AI Agent
Use cases: Automation
Deployment: Refer to project documentation
License: MIT

Best for

Developers, AI engineers, and researchers who have an Android device or emulator and want to build agents that interact with smartphone apps via front-end UI actions.

Key capabilities

Allows the agent to explore an app by attempting a given task without human intervention, reflecting on previous actions and generating documentation for elements explored.
Enables users to demonstrate a task by manually interacting with labeled UI elements on a screenshot, allowing AppAgent to learn and generate documentations for those UI elements.
Allows the agent to execute user-given tasks automatically using a previously generated documentation base.
Provides an optional grid overlay on the screen allowing the agent to tap or swipe anywhere on the screen, helping locate UI elements not labeled with a numeric tag.
Allows users to test AppAgent using their own models by writing a new model class in scripts/model.py.

Limitations and risks

GPT-4V is not free. Each request and response pair involved in this project costs approximately $0.03.
Running the agent without any documentation does not guarantee task success rate.
Alternative free models such as qwen-vl-max have relatively poorer performance compared with GPT-4V.

Getting started

Install dependencies using pip, configure an OpenAI API key in config.yaml, connect an Android device with USB debugging enabled, run learn.py, and follow the prompts. Setting up requires installing Python dependencies, configuring API keys, and connecting an Android device or setting up an Android Studio emulator.

Evidence and sources

GitHub project description: AppAgent: Multimodal Agents as Smartphone Users, an LLM-based multimodal agent framework designed to operate smartphone apps.
README: [2024.1.2]: Added an optional method for the agent to bring up a grid overlay on the screen to **tap/swipe anywhere** on the screen.
README: We introduce a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a…
README: The demo video shows the process of using AppAgent to follow a user on X (Twitter) in the deployment phase.
README: This section will guide you on how to quickly use `gpt-4-vision-preview` (or `qwen-vl-max`) as an agent to complete specific tasks for you on your Android app.

Reviewed at: 2026-07-15T14:53:57Z