Read me for Self-Driving-Cars-Term1-Project3 - Behavioral Cloning

• A submission by Jai Lad

Table of contents

1. Objective(s)
2. Result Videos
   1. Drivers' perspective
   2. Behind the car perspective
   3. Remix of the above!
3. Pre-Requisites / Setup / Dependencies
4. A note on using GPUs for training
5. Packages used to build the pipeline ( model.ipynb )
6. Key File(s)
7. Other File(s)
8. High Level Approach
   1. Pick five different CNN architectures for evaluation.
   2. Implement the architectures.
   3. Quick end-to-end validation of pipeline.
   4. Evaluate different CNN architectures.
   5. Select a CNN architecture.
   6. Data analysis and visualization.
   7. Data summary.
   8. Data balancing strategy.
   9. Data preprocessing.
   10. Data loading.
   11. Training, validation and testing.
   12. Off-center recovery practice.
9. Conclusion
10. Future Work and Learning

---

Objective :

• Use the simulator to collect data of good driving behavior.
• Design, train and validate a model that predicts a steering angle from image data - model.h5 .
• Use the model to drive the vehicle autonomously around the first track in the simulator. The vehicle should remain on the road for an entire loop around the track.
• Summarize the results with a written report - writeup.md

---

Result videos :

• The following videos are as captured during autonomous driving by my trained model. ( model.h5 )

- Drivers' perspective :

- A behind the car perspective :

- A mandatory Remixed version of the above video ! :

---

Pre-Requisites / Setup / Dependencies :

• Term1 - starter kit

  • I followed the instructions for Docker image ( Mac ) because I preferred to have the same code easily deployable to my local workstation ( with CPU ) versus GPU ( Amazon ), and Docker allows for that flexibility. This was also done to gain more familiarity with the Docker environment, and also because the setup was fairly straightforward.

  • Other than the above, I also setup the starter kit as the MiniConda environment which allowed me to easily interact with the generated model from the Command line. This also gave an opportunity to practise setup with the Anaconda environment.
    
    

---

A note on using GPUs for training :

• With regards to Amazon P2 GPU instances, I was able to get good results with my local workstation because of which I did not setup the Amazon P2 instances for this project, though I have used it in the past for model training. There are quite a pre-built Amazon AMIs which contain TensorFlow, Keras and related libraries installed which can be used for experimentation on Amazon. This is an example of such an AMI. Here is another useful resource on how to setup AWS P2 GPU instances.
  
  

---

Packages used to build the pipeline ( model.ipynb ) :

• Keras - 1.2.1 ( with Tensorflow backend )
• OpenCV - 3.1.0
• Numpy - 1.12.1
• Pandas - 0.19.2
• MatPlotLib - 2.0.0
• SciKitLearn - 0.18.1
• Python 3.6 - resource, csv, time, datetime
  
  

---

Key File(s) :

• model.ipynb - Jupyter notebook used to create and train the model.
• drive.py - Script to drive the car.
• model.h5 - A trained Keras model.
• writeup.md - A report writeup file
• video.mp4 - Video of the model.h5 driving successfully on track 1.
  
  

---

Other File(s) :

• Term1Project3BehaviorCloning.ipynb - the main Jupyter notebook which was used to experiment with various model architectures, and which was also used to perform data distribution analysis and balancing. This is the main 'parent' notebook which was used to complete the project, and from this, the model.ipynb was derived.

• SideExperiments.ipynb - another Jupyter notebook which was used to conduct small side experiments for this project.
  
  

---

High level overview of approach :

• For each of the steps below, the details are provided within the writeup.md file.

- Pick five different CNN architectures for evaluation.

• I selected five initial architectures for implementing this problem.

- Implementing the architectures

• The implementation platform chosen was Keras ( w/ Tensorflow backend ).

- Do quick end-to-end validation - training, validation, testing, driving.

• This was done from a risk mitigation perspective, to prove out the end-to-end pipeline up front.

- Evaluate different CNN architectures

• The above selected models were competetively evaluated against each other.
• Evaluation parameter(s):
  • Training accuracy.
  • Validation accuracy.
  • Training loss.
  • Validation loss.
  • Training time.
  • File Size of a trained model.

- Select a CNN architecture for the project

• In this step, the final CNN architecture was selected for this project.

- Data Analysis and Visualization

• In this step, descriptive statistics for sample data was generated, and the data set was visualized.
• Generated Box and Whisker plots for the data sets.
• Generated Histograms for the data sets.
• I visualized a few images from the CSV file.

- Data Summary

• From the above analysis, a summary was prepared, which was the foundation for the next step, which is coming up with a strategy for data balancing.

- Data balancing strategy

• In this step, the data balancing strategy was finalized. Over represented data was rejected, where as under represented data was additionally captured or synthesized.

- Data preprocessing

• Data Normalization
• Region of interest selection

- Data Loading

• In this step, the challenges of loading the data were resolved with a generator pattern.

- Training and evaluation

• In this step, the model training and evaluation was performed.

- Center driving and Off-center recovery practice.

• In this step, the model was trained specifically for off-center recovery.

Conclusion :

• At the end of the process, the vehicle is able to drive autonomously around the track without leaving the road.
• The model was trained at 9mph, however, it is able to drive itself upto 25 mph.

Future Work and Learning :

• Ensuring that the model can drive at the top speed of 30 mph.

• Ensuring that the model is able to drive well, on the much more challenging track 2.

• Experimenting with 'off-track' recovery. That is, when a model has actually left the road, is it able to use the Camera to recover back to the road.

• Extending this project by doing multi-class predictions, for adaptive throttle control implementation. The current model uses PID control to stay as close as possible to a fixed speed. However, by extending the model, we should be able to implement adaptive speed control, in which for a given scene, in addition to predicting the steering angle, we also predict the throttle value. For example, on a straight road, we should be able to zoom at the max speed, whereas on a curve, we can reduce the throttle to an appropriate value.

• Generate activation maps of different Keras layers for a given image. This would be helpful in seeing, 'how' the Keras layer sees a given image, i.e. what road features stand out for the network and help it in making the decisions.

• Currently, I have used a modified Nvidia architecture to drive the car. However, as an optimization exercise, I can try to shave off a few layer(s) from this architecture, and then evaluate it, until I can come up with a smaller architecture which drives 'well enough' on the track. The advantage of this would be smaller training time, and smaller memory footprint of a saved model.

• Given a pre-trained network I need to learn how to insert a pre-processing Lambda layer at the very start of the network.

---