Mapless sensor-level navigation is described as the policy trained in the simulation to drive a mobile robot in the real world. We formulate the autonomous navigation problem by using a Partial Markov Decision Process (POMDP), which consists of a tuple (S, A, P, R, O), where S presents state space (s_t∈S), A is action space (a_t∈A), P presents actions space, R present reward function, O is sensor observation.

State space: The state of the agent s_t = [L_t, d_t, α], where L_t is normalized laser readings, d_t is the displacement of the robot in polar coordinates, α is the deviation angle between the robot and the goal direction.

Action is two-dimensional vector, which includes linear velocity v and angular velocity w of that robot mapped with the input of model a_t = (v, w).

Reward function: Since the optimal policy is affected intensively by reward signals, distinct behaviors of the agent are reshaped as the reward function. Based on the previous work^[3], we restrengthen the reward function of the policy for collision-free navigation tasks, comprise of r_g, r_s, r_rel and r_col:

P^t and P^t-1 are the robot at the current and position and previous timestamp. r_g is reward of reaching the goal. The relative reward r_rel and safety reward r_s are defined as:

Where c, k are coefficients, r₀ is robot radius. The resparse reward when reaching to the target is r_g = 500 and r_col = -600 is given if the collisions happen.

The fundamental concept of Distributional SAC framework is to use random information to estimate value function within continuous action space. The advantage of DSAC is to take advantage of SAC while keep exploration based on random domain of value function. Existing distributional RL^[5] algorithm rely on distributional Bellman equation. The value distribution can be defined as:

Where A=DB denotes equal probability laws between those two random variables A and B, γ∈[0,1) is discount factor, Z^π (S', A') is random return given the next state-action (S', A'). Z^π (s, a) is random distribution return and r(s, a) is random reward. With maximum entropy RL of SAC^[2], the soft action-value function of a DSAC policy is:

Inheriting from the Actor-Critic network, the critic utilizes quantile fraction τ₁, ... τ_N and τ1',⋯τN', N' =32 is the number of quantile sampled separately, the loss of the critic:

The quantile regress loss and the temporal difference are defined as equation (7), (8) respectively.

Where (s_t, a_t, r_t, s_t+1) is a transition of buffer. Z^τst,at is critic output, which estimate of the τ - quantile of distributional value function. The objective of policy is formulated as

In order measure risk sensitive decision, CVaR^[8] is applied to approximate quantile value under uncertainty with actions and rewards.

Where β∈(0,1) is the risk distortion coefficient, β = 1 give a risk-neutral policy.

The robot states are generated from laser readings from LiDAR and combined with relative location from odometry data. Based on^[4], the architecture of the network comprises of 108 inputs of Actor networks including 105 laser scans from the YDLIDAR G6 sensor, deviation angles, and minimum range distance. After using a re-parameterized trick, the Actor network produces distribution of the bound angular and linear velocity. The Actor neural network is constructed by decreasing the network size over each layer, leading to optimize computation and keep the simple network and rich presentations The Critic network is present in [Fig. 2], which is incooperated with quantile^[8].

[Fig. 2] 
The structure of actor and critic network

Both physical simulation and real-world environments are particularly utilized for training and testing Deep RL model in term of accelerating simulation time (shown in [Fig. 3]). The whole training process is conducted on a Gazebo Simulation with a 10Hz control frequency. The Simulation will reset (after 0.5s) when the robot’s collision happens in the virtual environment The virtual environment is created by designing an environment with natural elements that closely resemble those found in the real world. Various obstacles of different sizes ensure the diversity and complexity of the test environment. To generalize the algorithm’s performance, we trained the deep RL algorithm with the framework over service robot model for approximately 1200 episodes. With the acceleration time in simulation. The entire training process trained by DSAC tooks nearly 5 hours, significantly reducing the time^[9].

[Fig. 3] 
The Gazebo simulation during training

To evaluate mapless navigation algorithms, the learning curve over training progress is depicted in [Fig. 4]. The maples navigation algorithm based on DSAC algorithm is compared with two state of the art algorithm SAC^[4] and PPO^[10]. It can be seen that the speed of learning curve using DSAC get faster and also more stabilize than SAC and PPO. [Fig. 4] illustrates the highest average score of Distributional SAC among other SOTA algorithms. The hyper parameters for training and testing in the simulation are shown in [Table 1]. The deep RL model will find arbitrary goals which is set randomly in the virtual environment avoiding different shapes of object (shown in [Fig. 5]). The [Fig. 6] shows the success rate in three virtual environments environments. The risk-sensitive policy is changed slightly depending on confidence level β, the risk neutral policy performs better than others shown in [Fig. 7].

[Fig. 4] 
Training curve of SAC, PPO, DSAC algorithms according to hyper parameters in Table 1

[Fig. 5] 
Testing in simulation environment including environment (a), environment (b), environment (c)

[Fig. 6] 
The success rate under test evaluation in the simulation

[Fig. 7] 
Comparison of risk averse policy with CVaR

The mapless navigation using DSAC is performed in two real world scenarios. First, we evaluate the DSAC algorithm in the clutter environment illustrated in [Fig. 8], with two consecutive target positions. Then, corridor environment with human motions is performed to evaluate the collision avoidance ability of the DSAC algorithm. The YDLIDAR is equipped on the service robot to sensing the environment from -60° to 60°. The on-board computer was a mini-PC with CPU AMD Ryzen™ 7 5700U Processor. The trajectory result of clutter environment and corridor environment are shown in [Fig. 9]. The map information is installed to only assist localization visually in the real-world scenarios’ evaluation^[11]. The experiment video can be shown in https://youtu.be/CxFutv_RlkU.

[Fig. 8] 
Sim-to-real transfer in real-world scenario: (a) clutter environment, (b) corridor environment

[Fig. 9] 
Trajectory of the robot in real-world scenarios: (a) clutter environment, (b) corridor environment