Table of Contents

1. Advanced Lane Lines
   • Pipeline
   • Pipeline Images
2. Files
   • Usage
3. Challenges
   • Lack of Intuition
   • Building Intuition with Visual Augmentation
   • Road Textures
4. Shortcomings & Future Enhancements
   • Enhancements for future
5. Acknowledgements & References

---

Advanced Lane Lines

This video contains results and illustration of challenges encountered during this project:

---

Pipeline

1. Camera Calibration
   • RGB2Gray using cv2.cvtColor
   • Finding and Drawing Corners using cv2.findChessboardCorners and cv2.drawChessboardCorners
   • Identifying Camera Matrix and Distortion Coefficients using cv2.calibrateCamera
   • Undistort
     • Cropped using cv2.undistort
     • Uncropped additionally using cv2.getOptimalNewCameraMatrix
   • Perspective Transform in corners_unwarp
2. Filters using filtering_pipeline
   • RGB to HSL
   • H & L Color Threshold Filters
   • Gradient, Magnitude and Direction Filters
   • Careful Combination of the above
   • Guassian Blur to eliminate noise K=31
3. Lane Detection pipeline
   • undistort
   • perspective_transform
     • crop_to_region_of_interest
   • filtering_pipeline
   • fit_lane_lines
     • left_fitx
     • right_fitx using histogram[:midpoint] and sliding window to capture points forming a lane line along with 2nd Order Polynomial curve fitting, identifies
   • overlay_and_unwarp
     • Car's Trajectory car_fitx
     • Lane Center mid_fitx
     • fill_lane_polys
   • calculate_curvature
     • left_curve_radius
     • right_curve_radius
     • off_centre_m
   • put_metrics_on_image
   • finally returning an undistorted image
4. Save as Video.mp4

Pipeline Images

Camera Calibration > Undistort

Camera Calibration > Perspective Transform

Lane Detection > Undistort

Lane Detection > ROI Mask Overlay

Lane Detection > Perspective Transform

Lane Detection > Filtering Pipeline

Lane Detection > Offset and Curvature Identified

Minor: Note the position from center is represented as a positive 0.2(m). Compare with images below.

Files

The project was designed to be modular and reusable. The significant independent domains get their own Class and an individual file:

1. camera.py - Camera Calibration
2. lanes.py - Lane Detection
3. main.py - Main test runner with test_road_unwarp, test_calibrate_and_transform and test_filters
4. utils.py - Handy utils like imcompare, warper, debug shared across modules
5. settings.py - Settings shared across module
6. vision_filters.py - Gradient, magnitude, direction, Sobel filters and related
7. README.md - description of the development process (this file)

All files contain detailed comments to explain how the code works. Refer Udacity Repository CarND-Advanced-Lane-Lines - for calibration images, test images and test videos

Usage

Repository includes all required files and can be used to rerun advanced lane line detection on a given video. Set configuration values in settings.py and run the main.py python script.

$ grep 'INPUT\|OUTPUT' -Hn settings.py
settings.py:9:      INPUT_VIDEOFILE = 'project_video.mp4'
settings.py:11:     OUTPUT_DIR = 'output_images/'

$ python main.py
[load_or_calibrate_camera:77] File found: camera_cal/camera_calib.p
[load_or_calibrate_camera:78] Loading: camera calib params
[test_road_unwarp:112] Processing Video:  project_video.mp4
[MoviePy] >>>> Building video output_images/project_video_output.mp4
[MoviePy] Writing video output_images/project_video_output.mp4  
100%|██████████████████████████████████████████████████████████████▉| 1260/1261 [11:28<00:00,  1.99it/s]
[MoviePy] Done.
[MoviePy] >>>> Video ready: output_images/project_video_output.mp4

$ open output_images/project_video_output.mp4

Challenges

Lack of Intuition

There was no visual indication for developing an intuition to identify Car's Lane-Center offset by looking at the video. For example in this image the car is quit off the center towards the left side of the lane. But it doesn't show:

Figure: Frame with No Intuitive Indication of Off-Center Distance

Building Intuition with Visual Augmentation

To get an intuitive feel, I decided to identify, approximate and visualize the position of the car in the lane with respect to lane-center. In order to achieve this, I identified four different lane lines:

1. Left Lane Marker Line
2. Approximate Car's Trajectory Line
3. Lane Center Line, and
4. Right Lane Marker Line

Figure: Frame with Intuitive Off-Center Highlights

Road Textures

Figure: Frame with Bridge

Figure: Frame with Shadows (Sidenote: Smaller Off-Center Position)

A simple and elegant solution to approximate the trajectory of the car was to use the existing lane end markers and interpolate between them. Using the center of the image as car's position and identifying its relative position between the lane ends, I came up with an approximate car trajectory, as follows:

# Identify/Highlight car's offset position in lane
ratio = (float(image.shape[1])/2 - left_fitx[-1]) / (right_fitx[-1] - left_fitx[-1])
mid_fitx = (left_fitx + right_fitx)*0.5
car_fitx = (left_fitx + right_fitx)*ratio

Then it was just a matter of coloring them with a generic fill_lane_polys()lanes.py#L222-L236 function to draw polygons on lanes given a lane fit line. The off-center distance was thus displayed the lane itself in RED.

self.fill_lane_poly(image, car_fitx, ploty, mid_fitx, mid_color)

Shortcomings & Future Enhancements

Figure: Example of a frame where the current implementation falls apart

Enhancements for future

1. Use ∞ for straight lanes beyond say ~10km radius
2. Add Intermediate Processing Frames to Video
3. Smoothen Radius Metric using Moving Average (lowpass filter)
   • Use a Line class to keep track of left and right lane lines
   • Consider Weighted Averaging (on line-length, for example)
4. Reuse lane markers to eliminate full frame search for subsequent frames
5. Use sliders to tune thresholds (Original idea courtesy Sagar Bhokre)
6. Work on harder challenge videos

Acknowledgements & References

• Sagar Bhokre - for project skeleton & constant support
• Caleb Kirksey - for motivation and the idea of using camera bias
• CarND-Advanced-Lane-Lines - Udacity Repository containing calibration images, test images and test videos