Adversarial Simulator allows one to simulate adversarial attacks in the virtual environment within the Unreal Engine and inference with Nvidia DRIVE SDK. This repository contains the code, assets, example datasets, models, test results, and a video demo for the paper :
A Simulated Environment Approach to Investigate the effect of Adversarial Perturbations on Traffic Signs
Anonymous
Abstract -To study the effect of adversarial attack environment must be controlled. Autonomous driving includes mainly 5 phases sense, perceive, map, plan, and Drive. Autonomous Vehicle ‘sense’ the surrounding with the help of Cameras and Lidars. Deep Learning techniques are considered Blackbox and found to be vulnerable to adversarial attacks. In this research, we study the effect of the various known adversarial attack in the Unreal Engine-based high-fidelity real-time raytraced simulated environment. This experiment seeks an answer to questions such as if adversarial attacks work in moving vehicle scenarios and can an unknown network be targeted. We found that the existing Blackbox and Whitebox affects different traffic signs differently. Attack found to be affecting classification in static scenes are not similarly affecting in the moving vehicle scenarios. However, some attacks with barely noticeable perturbations were found to completely block the identification of certain traffic signs. By simulating the interaction of light on Traffic sign have also observed that daylight condition has a significant impact on the performance of the model. Our findings are found to be closely simulating scenarios as found in the real world.
For details related to setting up environments, Building Nvidia Drive SDK, Recording sessions, and Addons refer to the documentation: https://sunil.gitbook.io/adversarial-simulator/
An Adversarial simulator requires two components to be installed including an Unreal Engine and Nvidia Drive SDK. An Unreal Engine required to generate a real-like scenario.
Post-installation we require to set up the environment so that different scenarios can be generated. In Unreal Engine, such scenarios are organized in Maps. adversarial Simulator provides two maps.
2.1 MAP - 1
It has basic components for controlled testing, tracks are organized in parallel for a testing plurality of scenarios in parallel, keeping the overall environment the same.
2.2 MAP - 2
It is consisting of vast open land with various objects like buildings, water, over bridges, traffic lights, traffic signals, roadside objects, various kinds of animals, etc. This map can be used for a variety of testing.
Drive net sample comes up with video inference capabilities by default. additionally, if you want to predict Images then you need to modify samples. This section covers all the basic procedure for compilation and modifying the samples to r
4. Add ons
This section suggests various objects packs that can be added to the provided maps to create more complex scenarios.
- Animals
- Cars and mopeds
- Buildings
- Road builder
- Rocks
- Trees
- Shrubs and various kind of Grass
- car material packs
- Water simulation
- Weather simulation assets
- Dashboard for experimentation management
├── Config # Unreal configuration file
├── Content
│ ├── AnimalVarietyPack # Animals
│ ├── AutomotiveMats424 # Automotive materials like glass, metals
│ ├── cinematics # Camera and recording related dashboards
│ ├── City # Buildings and Homes
│ ├── City_Pack # Buildings and Homes
│ ├── EssentialTreeKit # Variety of trees
│ ├── Experiment # Roadsigns, patchs, postrs,
│ ├── Foliage # Flower shrub
│ ├── Geometry # Landscape
│ ├── GT_Free2020 # car esseentials
│ ├── Maps # MAP-1 and MAP-2 are located here
│ ├── MegascansMeadowPack # Gradss, shrubs etc
│ ├── MRT # modular road tool for constructing custom road
│ ├── rocks # rock and boulders
│ ├── Vehicles # bikes and other vehicles
│ ├── VehicleVarietyPack # various 4 wheeler vehicles
│ ├── WaterPlane # water simulation assets
│ └── Weather # wather effect such as Wind, leaf fall and lightning
├── docs # Documentation
├── scripts # C++ scripts for video and frame based inference
└── tools # Swiss army knife toolbox for windows file managementThe experimentation included many high-resolution assets and some-time single file span above 50 MB. 50 Mb is the size limitation imposed by Github for any single fine. Hence to manage larger files this project usesDVC (Data Version Control)an Open-source Version Control System for Machine Learning Projects. DVC allow us to share the large files with git easily
git clone
identify large texture files
dvc pull
If you're interested in contributing to this project, refer the following guidelines:
Identify a larger file than 50MB, these files will be sent to DVC. Following is the procedure to identify larger files recursively in windows and Linux:
{% tabs %} {% tab title="Windows" %} Use the Swiss army knife tool provided with the repository:
.\tools\sfk197.exe list -big -mbytes \path\to\content\folder
If you have not downloaded then download and setup DVC for windows
Commit larger files to DVC,
dvc add \path\to\the\large\file
Exclude files to git which are committed to DVC. This command will be provided by DVC prompt post committing files to DVC
Push files to DVC
dvc commit -m "the most imp file added"
dvc push
Push remaining files to Github {% endtab %}
{% tab title="Linux" %} Use bash command to find larger files
sudo find ./path/to/AdversarialSiumator/dir -type f -printf "%s\t%p\n" | sort -n | tail -1
Commit larger files to DVC
dvc add \path\to\the\large\file
Exclude files to git which are committed to DVC. This command will be provided by DVC prompt post committing files to DVC
Push files to DVC
dvc commit -m "the most imp file added"
dvc push
Push remaining files to Github {% endtab %} {% endtabs %}
Post-installation of Drive SDK, samples can be found at /usr/local/driveworks-2.2/samples/build/src/
cmake -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-9.2 \
-DSDK_NO_GITVERSION_BUILD=true -DCMAKE_BUILD_TYPE=Release \
-DSDK_BUILD_EXPERIMENTAL=false -DSDK_USE_CCACHE=true \
-DSDK_BUILD_{DEMOS,BENCHMARKS,PACKAGE}=false \
-DSDK_BUILD_DOCUMENTATION=false -DSDK_BUILD_BENCHMARKS=false \
-DSDK_BUILD_SAMPLES=true -DCMAKE_C_COMPILER=gcc-6 \
-DCMAKE_CXX_COMPILER=g++-6 -DSDK_USE_CCACHE:BOOL=ON ..
cmake --build .For more details on the installation procedure: https://sunil.gitbook.io/adversarial-simulator/docs/installation-procedures
To run the sample on PNG image, one must change the SignNet file to accept PNG files instead of the video feeds, The example file is provided at :
./sample_sign_classifier --model=EU_V2 --offscreen=2 --path=/media/sunil/Shots/directional_light_20_lux/null_real_test.0220.png --input-type=imagefile To run the sample on png image with Visualization
./sample_sign_classifier --path=/path/image.png --input-type=imagefile --viz trueFiles are organized as given below. The repository has the following pre-compiled assets :
In addition to this, the repository has all the codes require to infer on the images as well as video.
Following is the output of the command tree -d -L 2. It shows the directory structure up to level 2.
This project provides all the necessary assets for adversarial simulation. That includes the followings:
- Modular Roads
- Road signs with different effects like patches, perturbation.
- Any road sign can be modified easily
- Animal Pack
- It has animals like Fox, Pig, Crow etc..
- 3D People Pack
- It has people with limited motion to check algorithm with people identification
- Vehicle Variety Pack
- Various kinds of vehicles including cars, Mopeds, Trucks etc..
- City - Object Pack
- has various objects like house, dustbin, street light, roadsigns etc..
- Standalone HTML API based dashboard
- More vehicles
- Vehicle Spawning
- Central Weather control