DroidTask

This repository contains introduction to our open source data. You can download it from Google Drive.

There are 2 files in the Drive; they are similar on size and content: user_tasks contains several .yaml files as tasks for each app.

• user_tasks.tar.gz
• utgs.tar.gz

user_tasks

Inside the user_tasks, there are several sub-folders named as each app; inside them there are several task files written in .yaml.

user_tasks
├── applauncher
│   ├── task1.yaml
│   ├── task2.yaml
│   ├── task3.yaml
│   ├── task4.yaml
│   └── task5.yaml

Every task is organized by a list of records. Each record contains UI state, current choice, possible input, and hashed state string. Below is one record for example.

records:
- Choice: 0
  Input: Clock
  State: "<input id=0 >Search</input>\n <button id=1 text='Sort by'></button>\n <button\
    \ id=2 text='Toggle app name visibility'></button>\n <button id=3 text='More options'></button>\n\
    \ <button id=4>Calendar</button>\n <button id=5>Camera</button>\n <button id=6>Clock</button>\n\
    \ <button id=7>go back</button>"
  new_state_str: 3f9adbf098ff69152b801042222f25ddd612674644e5d20b4c3617b4c9e33b3f
  state_str: ca84887fa50fbe8cb13d9c9fbcd7f1e568612b5e7448bbc9ac0425944993c94c

To understand this, the record does a step that "Click on the Search button, and input 'Clock' into it". What's more, if there is no input, the input will be set to null.

utgs

Besides the task files, we also provide all the raw files generated by DroidBot-GPT.

utgs
├── applauncher
│   ├── events
│   ├── states
│   ├── states_view_ids
│   ├── stylesheets
│   ├── views
│   └── utg.yaml

States & UI Tree Matching

If you want to use a new form instead of the provided HTML in our .yaml, you can use new_state_str to find a corresponding source state file in the states folder.

In the states folder, there are pairs of .json(states) and .png(screenshot), named in the form of timestamps. Below is an example of the .json file:

{
  "tag": "2023-07-08_212539",
  "state_str": "f8ad670e9a731699680252f02c59a057647e07cd5290c9cc4cb69554d0285bd9",
  "views":[
    ...
  ],
  ...
}

Here lists several important keys only. Notice that, the new_state_str can be mapped to the only state by the key state_str in the .json file. Another important key is views. Inside this there is a complete raw UI tree of the current screen, through which you can process to the form you want.

More details

For more details, please get access to our homepage and paper.