Intel-OneAPI

Team Name - Momentum

Team Members - GVS Mothish, Akash Agrawal, Sudarshan Bandyopadhyay, Swapnil Mane

Challenge Name - Object Detection For Autonomous Vehicles

📜 Prototype Brief:

Description: This project is part of the Intel OneAPI Hackathon 2023, under the object detection and Segmentation for Autonomous Vehicle theme by Team Momentum.We embarked on a project to develop a robust object detection and segmentation model capable of handling complex scenarios like bad weather and low light conditions. For the task at hand we have implemented Hybridnets model, which is an end2end perception network for multi-tasks. Our work focused on traffic object detection, drivable area segmentation and lane detection. HybridNets can run real-time on embedded systems, and obtains SOTA Object Detection, Lane Detection on BDD100K Dataset. The model is trained on image input which will be temporal in nature. Further training is done using HybridNet neural network Architecture with the intel's oneDNN pytorch optimization to perform faster inferencing. Finally the real time inference will be achieved using the intel's oneDNN libraries which will provide mainly three outputs that are, object bounding box, object class and lane detection.

🚩 Medium Article

Article Link

🚩 PPT and Demonstration

PPT and Demonstration Link

🍞 Tech Stack:

Technologies used to Build the prototype Intel® AI Analytics Toolkits, and it's libraries

🍞 Project Structure

HybridNets
│   backbone.py                     # Model configuration
│   export.py                       # UPDATED 10/2022: onnx weight with accompanying .npy anchors
│   hubconf.py                      # Pytorch Hub entrypoint
│   hybridnets_test_images_old.py   # Image inference
│   hybridnets_test_images.py       # Modified hybridnets_test to get inference time for a no of images
│   hybridnets_test_videos_old.py   # Video inference
│   hybridnets_test_videos.py       # Modified hybridnets_test_videos to get inference for different length of videos
│   speedup_test.py                 # Calculate 
│   train.py                        # Train script
│   train_ddp.py                    # DistributedDataParallel training (Multi GPUs)
│   val.py                          # Validate script
│   val_ddp.py                      # DistributedDataParralel validating (Multi GPUs)
│   frameCount_vs_time_plot.py      # Plot framecount vs time taken to infer
│   imageCount_vs_time_plot.py      # Plot imagecount vs time taken to infer
│   speedup_test.py                 # Speedup test for comparison between with and without optimization
│
├───demo                            # Images and videos for testing inference
│
├───demo_results                    # Post processing results on test images and videos to validate the inference
│
├───data                            # Time record for inferences on different conditions
│
├───plots                           # Comparison plots between inferences on different conditions
│
├───encoders                        # https://github.com/qubvel/segmentation_models.pytorch/tree/master/segmentation_models_pytorch/encoders
│       ...
│
├───hybridnets
│       autoanchor.py               # Generate new anchors by k-means
│       dataset.py                  # BDD100K dataset
│       loss.py                     # Focal, tversky (dice)
│       model.py                    # Model blocks
│
├───projects
│       bdd100k.yml                 # Project configuration
│
└───utils
│   constants.py
│   plot.py                         # Draw bounding box
│   smp_metrics.py                  # https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/metrics/functional.py
│   utils.py                        # Various helper functions (preprocess, postprocess, eval...)

🍞 Installation

The project was developed with Python>=3.7 and Pytorch>=1.10.

# Creating Anaconda Virtual Environment Inside Project Folder
conda create -p venv python==3.7.2 -y

# Activating the created Virtual Environment
conda activate venv/

#Installing Dependencies
pip install -r requirements.txt

#Installing Pytorch for CPU
pip install torch==1.13.1+cpu torchvision==0.14.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

#Installing Intel Pytorch Optimisation Ipex Dependency
pip install intel_extension_for_pytorch==1.13.100 -f https://developer.intel.com/ipex-whl-stable-cpu

🚩 Project Demo - Step-by-Step Code Execution Instructions:

# Download end-to-end weights
curl --create-dirs -L -o weights/hybridnets.pth https://github.com/datvuthanh/HybridNets/releases/download/v1.0/hybridnets.pth

# Image inference with Intel Optimisation
python3 hybridnets_test_images.py --source demo/images --output demo_result/images --use_optimization True --enable_postprocessing True

# Image inference without Intel Optimisation
python3 hybridnets_test_images.py --source demo/images --output demo_result/images --use_optimization False --enable_postprocessing True

# Video inference with Intel Optimisation
python3 hybridnets_test_videos.py --source demo/video --output demo_result/video --use_optimization True --enable_postprocessing True

# Video inference without Intel Optimisation
python3 hybridnets_test_videos.py --source demo/video --output demo_result/video --use_optimization False --enable_postprocessing True

# Result is saved in a new folder called demo_result

🚩 Usage

Dataset Structure:

HybridNets
└───datasets
    ├───imgs
    │   ├───train
    │   └───val
    ├───det_annot
    │   ├───train
    │   └───val
    ├───da_seg_annot
    │   ├───train
    │   └───val
    └───ll_seg_annot
        ├───train
        └───val

For BDD100K: (DataSets Used)

• imgs
• det_annot
• da_seg_annot
• ll_seg_annot

🚩 Benchmarking Results

*inference time is measured in seconds.

📜 References

HybridNets: End-to-End Perception Network Paper Link

📜 What We Learned:

1. Expansion of domain knowledge in deep learning based computer vision techniques like object detection and segmentation.
   
2. Usage of End2End Perception Network Hybridnet to do image and video inferencing for simultaneous object detection and segmentation.
   
3. Importance of Robustness in Autonomous Driving: Development of a robust algorithm that could handle all weather conditions, night time conditions as evident by the results.
   
4. Incorporation of Intel oneAPI libraries oneDNN libraries.
   
5. Learnt about Optimisation techniques for faster inferencing, specifically the libraries developed by Intel.