diff --git a/.assets/right_turn_hard_fov.gif b/.assets/right_turn_hard_fov.gif new file mode 100644 index 0000000..01e4db7 Binary files /dev/null and b/.assets/right_turn_hard_fov.gif differ diff --git a/.assets/visualization.png b/.assets/visualization.png new file mode 100644 index 0000000..62b14ee Binary files /dev/null and b/.assets/visualization.png differ diff --git a/README.md b/README.md index 22306f0..11b90c5 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ## **Can world models imagine traffic dynamics for training autonomous driving agents? The answer is YES!** -Using the high-fidelity CARLA simulator, we are able to train a world model that not only learns complex environment dynamics but also have an agent interact with the neural network "simulator" to learn to drive. +Integrating the high-fidelity CARLA simulator with world models, we are able to train a world model that not only learns complex environment dynamics but also have an agent interact with the neural network "simulator" to learn to drive. Simply put, in CarDreamer the agent can learn to drive in a "dream world" from scratch, mastering maneuvers like overtaking and right turns, and avoiding collisions in heavy traffic—all within an imagined world! @@ -19,6 +19,13 @@ Dive into our demos to see the agent skillfully navigating challenges and ensuri **Documentation:** [CarDreamer API Documents](https://car-dreamer.readthedocs.io/en/latest/). +## :sun_with_face: Built-in Task Demos + +> [!TIP] +> A world model is learnt to model traffic dynamics; then a driving agent is trained on world model's imagination! The driving agent masters diverse driving skills including lane merge, left turn, and right turn, to random roadming purely **from scratch**. + +We train DreamerV3 agents on our built-in tasks with a single 4090. Depending on the observation spaces, the memory overhead ranges from 10GB-20GB alongwith 3GB reserved for CARLA. + @@ -53,11 +60,27 @@ Dive into our demos to see the agent skillfully navigating challenges and ensuri
Right turn hard
-![CarDreamer](.assets/architecture.png) +## :blossom: The Power of Intention Sharing + +> [!TIP] +> **Human drivers use turn signals to inform their intentions** of turning left or right. **Autonomous vehicles can do the same!** + +Let's see how CarDreamer agents communicate and leverage intentions. Our experiment have demonstrated that through sharing intention, the policy learning is much easier! Specifically, a policy without knowing other agents' intentions can be conservative in our crossroad tasks; while intention sharing allows the agents to find the proper timing to cut in the traffic flow. + + + + + + + + + + +
Sharing waypoints vs. Without sharing waypointsSharing waypoints vs. Without sharing waypoints
          
-# 📋 Prerequisites +## 📋 Prerequisites Clone the repository: @@ -80,11 +103,30 @@ pip install flit flit install --symlink ``` -# Training +## :mechanical_arm: Training Find ``README.md`` in the corresponding directory of the algorithm you want to use and follow the instructions. -# Citation +## :computer: Visualization Tool + +We stream observations, rewards, terminal conditions, and custom metrics to a web brwoser for each training session in real-time, making it easier to engineer rewards and debug. + + + + + + + + +
Visualization Server
+ +## :hammer: System + +To easily customize your own driving tasks, and observation spaces, etc., please refer to our [CarDreamer API Documents](https://car-dreamer.readthedocs.io/en/latest/). + +![CarDreamer](.assets/architecture.png) + +# :star2: Citation If you find this repository useful, please cite this paper: ``` @@ -135,13 +177,6 @@ If you find this repository useful, please cite this paper: Shuangyu Cai - - - ustcmike -
- ucdmike -
- gaodechen @@ -162,10 +197,17 @@ If you find this repository useful, please cite this paper:
Hanchu Zhou
+ + + + ustcmike +
+ ucdmike +
-TODO: test monitor save \ No newline at end of file +TODO: test monitor save