Perception and Deep Reinforcement Learning-Based Local Path Planner for Autonomous Vehicles

2.1 System Overview

The system is outlined as [Fig. 1], which illustrates the main components of the pipeline. The agent has three different input blocks, each of them would generate semantic information accordingly. The observation is stated as follows:

[Fig. 1]

Overall system diagram

where $$ O_t\in \mathbb{R}$$ represents the raw observation signal; function e is latent parsing of driver view; Ψ is the ray casting algorithm for producing distance of surrounding environments; ⊕ is the concatenation operation.

2.2 Latent Encoding for Driver View

In this section, we describe the process of compressing images into a latent space for use as observations in an autonomous driving system. The driver’s view is critical, requiring analysis of semantic features for effective decision-making. We utilize the method from ^[9] to compress images into a latent state, reducing the computational load on the DRL model while providing it with more relevant, abstracted information. This approach improves efficiency by focusing the DRL model on higher-level tasks. The architecture is outlined in [Fig. 2], and two loss functions, a reconstruction loss and a feature loss, are used to ensure accurate latent space representation as:

[Fig. 2]

Variational Autoencoder for parsing driver view

where $$ Y_l\in \mathbb{R}$$ represents the label, while $$ Y_k\in \mathbb{R}$$ denotes its reconstructed output through the autoencoder network. The total number of samples is indicated by n. Within the latent space representation, $$ {\mu }_j\in \mathbb{R}$$ corresponds to the mean of the j^th dimension, and $$ {\sigma }_j\in \mathbb{R}$$ represents its associated standard deviation. The complete dimensionality of the latent space is denoted by J. This dual-objective optimization ensures both accurate reconstruction of the input state and a well-structured latent space. The encoder network employs convolutional layers to transform the input image into two vectors: a mean vector and a standard deviation vector, from which the latent vector is sampled.

2.3 Ray Casting Algorithm

While the front view or driver view observation supplies the local infomation of the agent in road, the bird’s-eye view (BEV) or top camera captures agnotic nature of surrounding information. The BEV segmented view can be achieved by using ^[10]. However, inputing it directly to DRL model would lead to be difficult for convergence and increase the computational burden. Therefore, we propose an efficient ray casting algorithm to extract essential spatial information by parsing distances to environmental boundaries.

To begin with, the process is generally demonstrated in [Fig. 3], which inputs a BEV segmented view and using image processing techniques to parse this image. Particularly, we convert a BEV segmented view into a binary image B_t with the drivable area being 255 and the non-drivable area is zero. Subsequently, we extract the binary contour C(α), which represents the boundaries between drivable and non-drivable regions. The continuous contour function C(α) maps the curve parameter α to two-dimensional spatial coordinate.

[Fig. 3]

The diagram of prepocessing BEV image

The contour results are presented in [Fig. 4], with the output of the ray-casting algorithm displayed in the leftmost image. With the environmental boundaries defined, we proceed to implement the ray casting mechanism. The algorithm employs an angular resolution Δρ, which determines the granularity of environmental sampling. Given this resolution, we distribute N rays uniformly around the vehicle’s position, where each ray Rᵢ is cast at angle ρ_i = iㆍΔρ, for i ∈ {0, 1, ..., N-1}. This uniform sampling ensures comprehensive coverage of the surrounding environment. The main idea of our algorithm lies in computing the intersections between these rays and the environmental boundaries. For each ray R_i, we determine its intersection with the contour C(α) by solving the equation C(α) = R_i(λ) shown in [Fig. 5], where $$ \lambda \in {\mathbb{R}}^{+}$$ represents the ray parameter. This intersection calculation yields crucial information about the distance to obstacles or boundaries in each direction. To obtain the final distance measurements, we compute a distance vector D = [d₀, d₁, ..., d_n-1]. Each component d_i is determined by finding points (x_i, y_i) = C(α_min) that minimize the Euclidean distance to the corresponding ray within its angular sector [ρ_i, ρ_i + Δρ]. Mathematically, this is expressed as:

[Fig. 4]

The processed images

[Fig. 5]

The principle of raycasting algorithm

where [α_l, α_u] represents the contour parameter range corresponding to the angular sector [ρ_i, ρ_i + Δρ]. Distance vector D serves as a compact yet informative representation of the environment’s spatial structure. This representation significantly reduces input dimensionality while preserving the essential spatial information necessary for autonomous navigation. By transforming complex visual data into a more manageable form, our ray casting algorithm enables efficient processing within the DRL framework while maintaining the fidelity of environmental perception.

2.4 Soft Actor Critic Network

Soft Actor-Critic (SAC) ^[11] is an off-policy maximum entropy deep reinforcement learning algorithm that provides a framework for energy-efficient and robust learning. The key principle of SAC is to maximize both the expected return and the entropy of the policy, which encourages exploration while maintaining stable learning. This is formulated as:

where β is the temperature parameter determines the relative importance of entropy according to reward; π* is optimal policy; τ series of state-action pairs, γ^t is discount factor ∈ [0,1]; s_t and a_t are state and action at time step t, respectively; R(s_t, a_t) is expected reward; and H(π(⋅|s_t))represents the entropy of the policy at state s_t. SAC employs two Q-functions to mitigate positive bias in policy improvement, and a state value function. These are defined as equation (6), (7), respectively.

The policy network in SAC is parameterized as a Gaussian distribution with state-dependent mean and standard deviation:

Then, it is updated to minimize the Kullback-Leibler divergence as:

where θ is policy network parameters; σ_θ is standard deviation of the policy distribution; and μ_θ is mean of the policy distribution. To further lift the exploration nature, the temperature parameter β is automatically adjusted during training to achieve a target entropy $$ {\mathcal{H}}_{target}$$:

The final loss comprises 3 different losses from equation (9), (10) and a Q-function loss expressed as:

The training process would include four basic steps: sample the batch from replay buffer, update Q-networks using L_Q and policy utilizing L_π, adjust temperature param β, and update the target network.

2.5 Reward Function

Given that the optimal policy is highly influenced by reward signals, the agent’s behavior is significantly shaped by the design of the reward function. In this case, the reward function is structured around five key factors, with the first being the vehicle’s speed, which is regulated to ensure appropriate speed control. It is defined as:

where ϵ is the current speed of car; ϵ_l and ϵ_u is the lower and upper speed limit respectively; ϵ_t is speed target that the car needs to achieve. The centering factor is defined as:

where $$ {\mathrm{\varrho }}_{\mathrm{c}}$$ is the distance of the vehicle from the center of the lane and $$ {\mathrm{\varrho }}_{\mathrm{m}\mathrm{a}\mathrm{x}}$$ is the maximum allowable distance for full reward. The standard deviation reward is calculate using the standard deviation of history distance $$ {\sigma }_{{\mathrm{\varrho }}_c}$$ from agent to the center path. This is formulated as:

The angle factor demonstrates the deviation from the center road ϕ. Its reward is defined as:

where safe distance maintenance from obstacles or roadside is defined by raycasting algorithm. Its reward is stated as:

where κ is a scaling factor that normalizes the standard deviation penalty. Additionally, R_T represent the following traffic light rule is motivated by ^[4]. Finally, the reward function is for total term is a weighted multiplication of all terms.

3.1 Environment and Setup

In order to validate our method, we conducted our experiments on CARLAR simulator, which allows us to collect comprehensive types of inputs. The simulation is reset whenever the robot experiences a collision within the virtual environment. We trained the deep reinforcement learning (RL) algorithm within the framework using a service robot model for approximately 1,200 episodes. The entire training process trained by SAC took nearly 8 hours, significantly reducing the time.

[Table 1] shows the training configuration of our experiment setup. In detail, the experiments were conducted on a system equipped with an NVIDIA-GeForce RTX 2060 GPU and an Intel Core i7-8700 processor running at 3.20GHz, supported by 16GB of RAM. The software environment consisted of Ubuntu 20.02 as the operating system and Carla version 0.9.13 for simulation purposes. The proposed method is implemented within Pytorch framework.

[Table 1]

Training parameters

3.2 Evaluation

First, to prove our effectiveness of our methods compared to other method, we refered the testing metric from the paper ^[12]. The Town 1 and Town 2 are set up with traffic light state. Our success rate achieve a robust state for all cases and working smoothly for all cases. Moreover, we have improve the safety term by using ray casting algorithm. This both reduces the ration to colide with obstacle and with the footpath. Because in SAC the exploration nature is enhanced, which somehow make the agent find the shortest path to go. This characteristic is not optimal during turns, since it may jostle the roadside. That is an unexpected situation in autonomous driving. Moreover, in our work, we still optimize the traffic state for the in DRL network. The raycasting algorithm also support this process by evaluation the measured surrounding distane. When reaching the intersection, the fronview and side view distance may be significantly large and change predictably. [Fig. 6] showcases the autonomous car have an ability to realize the red-light traffic and intersection to stop moving.

[Fig. 6]

Follow the red light situation. The red box indicates vehicle speed

The algorithm is compared with two state of the art algorithms DDPG ^[15] and PPO ^[16]. The [Fig. 7] and [Fig. 8] illustrate the comparative performance of different trajectory planning methods, including our implemented SAC approach, against the ground truth path. The plot shows trajectories in a 2D coordinate system, with all methods starting from the marked initial point (approximately at x=0, y=0). SAC demonstrates remarkable alignment with both the ground truth and alternative methods (DDPG and PPO), particularly in handling the sharp vertical transition near x=-80. This indicates that our SAC implementation successfully captures the optimal path planning behavior, matching the ground truth’s characteristics in both the steep ascent phase and the subsequent horizontal stabilization at approximately y=30. PPO, otherwise, fail in the testing conduction at the sharp turn. Now, we can see that, DDPG and SAC almost aligned with the system but SAC can travel with a shortest path. As demonstrated in [Table 2], SAC achieves shorter path lengths due to its enhanced exploration capabilities, which are attributed to its entropy-based optimization. While this approach results in slightly larger center-line deviations, the algorithm maintains collision-free navigation and operational safety throughout all test scenarios. This prove that the whole pipeline of our framework working well in street environment. To validate our framework, we compare against state-of the-art methodology on CARLA benchmarks. The results illustrated in [Fig. 9] demonstrate that our proposed approach not only performs well in generalized tasks but also exhibits robust performance during sharp turns while maintaining acceptable velocities.

[Fig. 7]

The trajectory of comperative methods on Town 1

[Fig. 8]

The trajectory of comperative methods on Town 2

[Table 2]

Quantitative comparison of performance from different types of policies

[Fig. 9]

The comparative success rate compared with other existing pipelines: MP, IL, RL, CIL, CIRL, CAL [7,13,14]