Skip to content

Latest commit

 

History

History
303 lines (256 loc) · 19.3 KB

README_EN.md

File metadata and controls

303 lines (256 loc) · 19.3 KB

CARLARain

A Rainy Autonomous Driving Simulator based on CARLA

CARLARain is an autonomous driving simulator framework that supports the simulation of rainy scenes in complex lighting environments. Based on CARLA, it realizes the environmental simulation, vehicle simulation, and pedestrian simulation of driving scenarios. In combination with HRIGNet and CRIGNet, it introduces controllable and realistic rainy scene simulation in complex lighting environments. This framework can build a rich rainy scene simulation training environment for autonomous driving visual perception algorithms, covering diverse time periods and lighting conditions, and meeting the requirements of multiple tasks such as semantic segmentation, instance segmentation, depth estimation, and object detection in autonomous driving scenarios.

Structure

  • configs：Configuration files
  • CarRain：CARLA client code, used to obtain the background RGB images, semantic segmentation maps, instance segmentation maps, depth maps, and object bounding boxes of CARLA simulations
  • CRIGNet：CRIGNet code, used to generate low-resolution rain streak images
  • RainControlNet：RainControlNet code, used to expand low-resolution rain streak images into high-resolution rain streak images
  • HRIGNet：HRIGNet code, used to generate rainy scene images based on background RGB images and rain streak images
  • data：Output data path

How to run

Environment Setup

Prepare Model Weights

  • All model weight files can be obtained here: BaiduCloud (Extraction code: i4zi)
  • CRIGNet model weights: Download CRIGNet/00000-crig_fastgan_g1w0.5_rainTrainL_256-FastGAN and extract it to CRIGNet/log_raintrainl_256
  • RainControlNet model weights: Download RainControlNet/crig_hint256-128_out512_4xgrad_1e-4-2024-10-05-T07-21-55 and extract it to RainControlNet/logs
  • HRIGNet model weights: Download HRIGNet/2023-10-19T19-31-52_blender-gdm-rainlayer-hw512-f4-em3 and 2023-10-21T21-50-57_blender-hrig-rainlayer+masked-gdm512-hw512-hybrid-unet128-em3 and extract them to HRIGNet/logs

Prepare YAML Configuration Files

  • Refer to configs/seqTest.yaml, and for detailed explanations of the content, see Docs/config.md

Run

1. Obtain Background Images and Label Data

  • Run the CARLA server:

    • In the CARLA directory, execute ./CarlaUE4.sh -RenderOffScreen -quality-level=Epic
    • Configure serverIP and serverPort in CarRain/config.py

  • Activate the conda environment: conda activate carla

  • Run the CARLA client to obtain background RGB images, semantic segmentation maps, instance segmentation maps, depth maps, and object bounding boxes: python CarRain/main.py -c configs/seqTest.yaml

  • Obtain object detection data: python CarRain/isToDetect.py -c configs/seqTest.yaml

  • Obtain videos: python CarRain/seqToVideo.py -c configs/seqTest.yaml

2. Obtain Rain Streak Images

You can choose to obtain rain streak images as rain layer masks using existing methods, or generate rain streak images using CRIGNet + RainControlNet provided here.

  1. Existing methods:

  2. CRIGNet + RainControlNet:

    • Use CRIGNet to obtain low-resolution rain streak images:

      • conda activate hrig
      • cd CRIGNet&&python gen_rain_mask.py -c../configs/seqTest.yaml&&cd..

    • Use RainControlNet to obtain high-resolution rain streak images:

      • conda activate control
      • cd RainControlNet&&python upscale_rain_mask.py -c../configs/seqTest.yaml&&cd..

3. Obtain Rainy Images

  • Use HRIGNet to obtain rainy images:
    • conda activate hrig
    • cd HRIGNet&&python predict_car.py -c../configs/seqTownsCombineTest.yaml

CARLARain Dataset

Based on CARLARain, we constructed an autonomous driving rainy - scene image dataset. We utilized eight different built-in scenarios provided by CARLA and set the time to three periods: daytime, evening, and night, to simulate driving scenarios under different lighting conditions. In terms of vehicle and pedestrian simulation, 100 vehicles and 500 pedestrians were randomly placed in each scene. The resolution of the rendered images is 2048×1024.

This dataset includes eight different scenarios. Each scenario contains three time periods, and each time period consists of 1000 frames of samples. Each sample includes RGB images of clear autonomous driving scenes, semantic segmentation images, instance segmentation images, depth images, rain-streak images, rainy RGB images, and object bounding-box data. The dataset is divided into a training set and a test set according to the scenarios. The training set includes seven scenarios, and the test set includes one scenario.

The dataset can be downloaded from CARLARain-Dataset (Extraction code: i4zi)

Dataset Type Scene Time Frame Sample Count Image Type
Trainset 7 3 1000 21000 Scene RGB image, semantic segmentation image, instance segmentation image, depth image, rain streak image, rainy RGB image, object bounding box
Testset 1 3 1000 3000
Background Rainy Depth Semantic Segmentation Instance Segmentation

License

The CARLARain code is distributed under the MIT License.

Reference