This is the official repository of the paper "Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System"

UI-NEXUS provides a comprehensive benchmark for compositional mobile device operation tasks. UI-NEXUS supports interactive evaluation in 20 fully controllable local utility app environments, as well as 30 online Chinese and English service apps. It comprises 100 interactive task templates with an average optimal step count of 14.05.

See introduction videos on our website.

It's recommended to use conda, which you can download here.

```
conda create -n ui-nexus python=3.11.8
conda activate ui-nexus
```

Then, you can install all the required packages by:

```
pip install -r requirements.txt
```

The subset on local utility app environments, named as UI-NEXUS-Anchor, is developed by enhancing the task difficulty based on AndroidWorld benchmark. We adopt the same 20 apps as AndroidWorld and design more sophisticated task instruction templates with comprehensive coverage of task dependency structures.

The infrastructure is mainly adapted from AndroidWorld and its adaption in SPA-Bench.

1. Set up the Android Emulator

   1. Download Android Studio here
   2. Create an Android Virtual Device (AVD) by following these instructions. For hardware select Pixel 6, for System Image select Tiramisu, API Level 33, and choose AVD name. Watch the setup video.

2. Launch the Android Emulator from the command line

   Launch the emulator from the command line, not using the Android Studio UI, with the -grpc 8554 flag which is needed communication with accessibility forwarding app.

   # Typically it's located in ~/Android/Sdk/emulator/emulator or
   # ~/Library/Android/sdk/emulator/emulator
   EMULATOR_NAME=YOUR_AVD_NAME 
   ~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554

3. Set up the adapted AndroidEnv:

   cd android_env
   
   python setup.py install

4. Install AndroidWorld.

   cd android_world
   python setup.py install

5. Install ffmpeg, if not already installed.

   # Linux (Ubuntu/Debian)
   # sudo apt update && sudo apt install ffmpeg
   
   # macOS
   brew install ffmpeg

6. Run the setup script of AndroidWorld to automatically install the local utility apps

   python run.py \
       --suite_family=android_world \
       --perform_emulator_setup 

7. Desktop Setup

   To minimize the impact of the app launching process on the test results, we placed all the app icons on the home screen during the tests. We install lawnchair on the emulator, use lawnchair as the default launcher, and changed the desktop grid to 5x4 to exactly fit in the 20 local utility apps. The layout in our experiments is shown in the screenshot

We have developed a user-friendly script for Android emulator environment setup. You can configure the states of 20 local utility apps by simply writing json files.

More details are elaborated in emulator_init/README.md

The configurations files of UI-NEXUS-Anchor are in android_world/task_config. Currently some representative cinfiguration files are provided. Complete configuration files will be released soon.

We are actively refactoring the codebase to unify the interface for all agents. This feature is a high priority and will be released soon.