Experimental setup to train an autonomous shuttle vehicle in a real physical world will be extremely troublesome. Since there are a lot of scie.jpgic parameters ought to be considered to make a training and test situation^[1]. Moreover, the expense and the strategic troubles in case of the real-world experimental training is also a hindrance^[2]. To able to learn properly, a machine needs to collect and train a large number of parameters in the model. In that case, the requirements of large data arise. Using a single vehicle, it is impossible to perform all the event needed to train a vehicle perfectly and gather data for those situations. Thus, alternate planning is badly needed where a virtual world can perform the continuous experiment of training a vehicle for autonomous car research. The simulator also gives an inherently safe environment for driving research^[3]. There is no endangerment to the driver or other vehicle users or pedestrians in case of the simulator. However, training autonomous driving vehicle with reinforcement learning in the real environment involves non-affordable trial-and-error. It is more desirable to train in a virtual environment initially and then transfers to the real environment^[4]. A brief survey about the existing open-source vehicle simulators available online was presented in^[5] where an efficient virtual simulating environment, CAIAS simulator for the self-driving car by utilizing the Unity real-engine was introduced, but there is no detailed description about specific deep-learning based training methods.

This paper extends the previous work done in a previous paper^[5] by adding more hostile and realistic environments and implementing an end-to-end deep learning method to teach a self-driving car for controlling the motion of the vehicle.

In this paper, three major contributions will be highlighted. First, we discuss open-source simulators and their features. All the features in those simulators are explained in details and key factors that are necessary for training autonomous vehicle with application to deep learning methods are discussed. Second, we propose a new effective approach to make a virtual simulation environment suitable for teaching deep learning based agents which control the motion of a self-driving car. Specially, we developed a synthetic virtual simulator by using the real-engine from Unity which could reflect various realistic situations and weather conditions, making the autonomous vehicles to learn and understand more realistic situations and some unexpected events. Finally, we verify the effectiveness of the developed simulator by implementing an end-to-end CNN algorithm^[6] for training a self-driving shuttle.

The remainder of the section is organized as follows. Section 2 provides an overview of the current open-source simulators used for self-driving car researches. In Section 3, an autonomous vehicle simulating environment for training the self-driving car is presented. Section 4 describes the detailed implementation of an end-to-end CNN based decision-making agent. Section 5 shows the verification of the performance of the proposed method.

There are many open-source vehicle simulators available these days, and each was built for a specific purpose and test. An overview of the simulators is represented below.

Unity is used to build a virtual simulating environment for training autonomous vehicles. This vehicle simulator is modified version of Udacity simulator^[13]. Since Unity has a good user interface and powerful interactive design module, this integrated platform is best for creating 3D simulator or other interactive contents such as virtual reconstructions or 3D animations in real time^[14]. To design the simulation environment and for training purpose, we used a hardware system which consists of Intel Core i7 CPU, 64 GB RAM and GTX 1080 GPU.

3.1 The process of importing Shuttle cart in Unity

An autonomous vehicle for training purpose is designed using Both SolidWorks and Blender software as a prototype of the real shuttle vehicle shown in [Fig. 1] and then is converted to fbx file and later imported environment. The server-client connection is initiated in the simulation environment using Socket. This SocketIO initiates the communication between the agent and the algorithm. Agent vehicle and environment work as a server and the model algorithm for training which works as a client. The simulator is made compatible for common operating systems like Windows, Linux and Mac OS with both 32-bit and 64-bit.

[Fig. 1] 

Blender and SolidWorks software both are used to design the prototype vehicle. (a) Top View (b) Isometric View (c) Front View (d) Side View

3.2 Environments Numbers

Two distinctive worlds with two different tracks, situations and weather are provided. One of the environment track is a forest environment. Here, the road is used in a loop, because in real life most of the time, a shuttle car is used in campus premises in a loop path. The same environment but different path track is used to verify the performance after training the model in the training path. Different path scenarios is a way to authentic that the simulator result can easily be transferred to the real physical world since the test ground path is totally different for the training path. Forest environment contains critical path turning, step slop, bumpy road condition, broken roads, sunny and gloomy weather. Sunny weather for training shown in [Fig. 2(a)] and gloomy or raining weather for the test shown in [Fig. 2(b)].

[Fig. 2] 

The forest environment for Train (a) and Test (b)

The user can choose preferable tracks to train the model. For training, path design, color is different [Fig. 2(a)] and the weather condition is sunny. For the test, road color and weather will be gloomy like [Fig. 2(b)]. The reason for converting one real-life scenario to convert to a virtual environment is to check and test some algorithm which can imitate the behavior from the image. So, having one scenario as like real life will give the advantage to evaluate those algorithms as well. Another track is randomly designed just to check all the steering angles and speeds of the agent car produced during the simulation. This environment contains the countryside road, trees, random obstacles, grasses and driving lanes.

The environment is constructed using different critical conditions and scenarios to make the model from training more well-efficient. The simulator contains another campus rural map shown in [Fig. 3] & [Fig. 4] in which we tried to develop one environment mimics with the campus environment. Skybox used to make the environment look more realistic which is a wrapper around your entire scene that shows what the world looks like beyond your geometry^[15].

[Fig. 3] 

The second virtual environment on the simulator is the depiction of shuttle rail on a campus area

[Fig. 4] 

The map of the second track of campus rural area

3.3. Weather Conditions and Different Road Layout

The tracks both contain rainy, hailing, fog, snowy, dry weather with both day and night time feature. To check further performance road layout and the color is changed.

To make it more realistic closely related to the physical world, some feature like a road bump, broken road, road slop, sun reflection in the road are included shown in [Fig. 5] and [Fig 6]. [Table 1] show the list of all simulation condition in the simulator.

[Fig. 5] 

Rainy Weather in the Environments (a) Wet road Forest Environment, (b) Campus rural area

[Fig. 6] 

Hailing Weather Condition in the Environments (Marked yellow particles are Hail Stone) in (a), Snowy Weather Condition in the Environments (b), Foggy Weather Condition with less visibility in the Environments (c), Sun Reflection in the wet road in (d)

[Table 1] 

All tracks and Weathers

3.4 Types of Sensors

Three types of sensors configuration are provided in the agent vehicle. The user can choose one or multiple sensors to generate results according to their model.

3.4.1 RGB Camera

A built-in script in Unity is used for the camera image. A camera is a device through which the player views the world like a simple camera. Here specifying pixel value is one of the important things. The single-camera is shown in [Fig. 7(b)] and multiple cameras showed in [Fig. 7(a)] is used as a sensor input. The idea of three multiple cameras is from behavior learning using convolutional network^[6].

[Fig. 7] 

Multiple camera (Left-Center-Right) train and receive data from the environment. (a) And a Single camera to take image input for the test (b)

3.4.2 LiDAR Sensor

LiDAR sensor imitation and obtain the LiDAR data as a form of an image by casting each individual laser in the simulator. This implementation is intuitive and accurate. For each laser in the simulated LiDAR, a raycast is used to detect the distance. In the update loop, if the timer exceeds the limit, a list of raycast will be a trigger to gather distance information^[5,16]. The result of all raycast is stored in a depth map image which is shown in [Fig. 8(b)]. The green image below the picture of the environment is the distance matrix correspond to the positions of all the obstacle in the environment, depicted in [Fig. 8(a)].

[Fig. 8] 

LiDAR sensor casting ray over different obstacles (a) and a depth map of the distance matrix corresponding to the positions of all the obstacle in the environment (b)

3.4.3 Depth Camera Sensor

A depth or motion vector texture can be generated from camera function in unity. This is a minimalistic G-buffer Texture that can be used for post-processing effects or to implement custom lighting models. Hence, it is usable as an output image like [Fig. 9] from the depth camera which produces the layer of the depth sensor completely based on the experiment layout^[5,16].

[Fig. 9] 

These figures show the output of the RGB camera and depth camera simultaneously

In this section, we will discuss the algorithms we used to verify the performance of the simulator. Also, the procedures for training a vehicle will be thoroughly described for algorithms like behavior learning^[6,17] in this section. The methodology displays an end-to-end convolutional neural network (CNN)^[6], trained with labeled data (steering angle, obstacles, roads, lanes) on different street conditions to avoid obstacles and autonomous navigation in an obscure condition. It includes image input from the simulation environment, image processing and transformation, color modification for lanes and array representation of images (NumPy array^[18]). The labeled data and images are fed into the convolutional neural network, which generates a calculated steering angle and speed values as output. [Fig. 10] presents a flowchart for the steps related to autonomous driving in the methodology.

[Fig. 10] 

Step by Step Process of training and testing of Behavioral Cloning

4.2 Architecture of CNN Training

The depicted technique structure is a fully-connected convolutional neural network which can learn representations from input camera images and use it for autonomous driving. The shown [Fig. 11] display the end-to-end convolutional neural network used for autonomous driving in this work. This architecture is similar to the architecture of Nvidia CNN architecture model^[6]. The network consists of 5 convolutional layers and 3 fullyconnected layers. The convolutional layers are designed for features extraction. First 3 convolutional layers use 2x2 strides and next 2 convolutional layers use 1x1 strides. All the convolutional layers use 5x5 kernels. Fully-connected layers are designed to function as a steering controller which generate the steering control value output. [Table 2] presents a summary for the network with layers, kernel size, strides, number of filters (convolutional layers) and the number of neurons (fully-connected layers).

[Fig. 11] 

Fully Connected CNN Architecture

[Table 2] 

Behavioral learning configurations for Layers, Filters, Kernels, Strides and Neurons

The CNN is trained with the weights to minimize the mean square error between predicted steering command and either steering commands from the human driver or the adjusted steering for off-center and rotated images.

The reordered data from the simulation environment (record image) is labeled with road-type and driver's behavior (turning, lane switching, lane following, brake, etc.). Data is preprocessed and fed into the network for training with their corresponding labels. [Fig.12] illustrates the data collection and training for autonomous driving using end-to-end CNN.

[Fig. 12] 

Training Strategy using behavior learning

Convolutional Neural Network training is dependent on how it sees the images. Keras helps to visualize the CNN layers output^[21] and to understand how an input image is processed inside the network. Following output in [Fig. 13] is generated on an image input by the CNN network. Keras feature map generates all the plots for the convolutional layer. From convolutional layer 1 & 2, the model can get features like edges, curves and lines. The features in convolutional layer 3 & 4 are a higher level of complex feature which is learned from previous output.

[Fig. 13] 

CNN Layer Output Visualization (a) Conv. 1 (b) Conv.2 (c) Conv.3 (d) Conv.4

The network is trained with 10 epochs and 20000 samples per epochs. The learning rate for the network is in exponential form, α = 0.0001 and, a batch size of 40 is applied. The training loss and validation loss is calculated and visualized in run-time using Keras. History can be tracked in Keras while training is going on. Later, it is possible to use those data to plot the graph using matplotlib python library. [Fig. 14] presents the training accuracy and training loss and validation loss respectively. Since the loss of both training and validation reduced systematically, it can be said that the model is not over-fitted. [Fig. 15] plot shows the model loss after 200000 batches. As the training proceed, the model loss reduced. It indicates the model got better with more training data.

[Fig. 14] 

The training loss after 10^th Epoch

[Fig. 15] 

The training loss after 200000 batches

[Table 3] reports the number of times vehicle effectively finished test condition under two distinct scene and two different weather using the same model generated after training. The quantitative evaluation is performed in both tracks. One of them is Forest shuttle track in rainy weather another of them is campus track during the night and snowy weather. We have executed each of the conditions mentioned in [Table 3] 20 times using our trained model on the mentioned scenarios and collected the number of times it succeeded. Though the result may vary with a different model of the dataset. Since a good amount of dataset will help to generate a perfect model. In general, the execution of all strategies is working properly and the achievement rate is good. The test results from the performance verify the efficacy of the simulator. From the result, it was evident that end-to-end CNN works efficiently in a similar environment it was trained. Validating the model in a different environment gives less success rate in case of behavioral cloning method. So, it is worthwhile to implement behavioral cloning in shuttle car since there is less change in a feature of a shuttle car and its environment.

In this study, we discussed about various simulators and their supporting API. One of the main reason to choose this simulator and train especially for behavioral cloning is the data generation for various environment and weather is quite easier in this simulator. Recording data for training using three camera sensor is a key feature to train using Nvidia end-to-end CNN behavioral cloning model. Also, the socket interface made the communication easier between the code and simulator. We discussed the development process of a dynamic simulator which has two different worlds consisting of different weather condition and situations. The dataset from the synthetic environment made the model more robust for shuttle car environment. Since it is kind of supervised learning, the dataset from the diverse environment and weather can boost up the accuracy of the steering angle of the model. We learned about the training procedures and methods in the platform. We implemented behavioral learning algorithms and checked quantitative performance in the road. We found that in an unknown environment the prediction accuracy gets reduced. In case of shuttle car, this algorithm is beneficial, since the shuttle car has less change in feature in front of the camera sensor in its environment and very specific road lane for shuttle car.