Deep Reinforcement Learning-Based QoS-Aware Routing Protocol for Space–Air–Ground Integrated Networks

I.INTRODUCTION

With the development of beyond 5G and novel 6G communications, a new Space–Air–Ground Integrated Network (SAGIN) has been proposed, consisting of satellites, high-altitude platforms (HAPs), unmanned aerial vehicles (UAVs), and terrestrial communication networks. SAGIN has the potential to offer global coverage, ultra-reliable low-latency communication, and high-capacity connectivity to remote areas, oceanic regions, aerial regions, and disaster areas. By integrating space, aerial, and ground networks, SAGIN can effectively overcome the coverage, scalability, and robustness limitations of traditional terrestrial networks [1]. But some routing problems arise due to the dynamic and heterogeneous nature of the SAGIN. Unlike ground or space networks alone, the network topology of SAGIN is highly dynamic due to the movement of satellites and UAVs [2]. The network features heterogeneous links, including radio frequency (RF) and free-space optical links, variable propagation delays, intermittent connectivity, and fluctuating traffic demands. These characteristics make it challenging to apply traditional routing protocols, such as shortest-path or static link-state algorithms, to ensure quality of service (QoS) guarantees. To maintain end-to-end QoS requirements, such as latency, throughput, packet delivery ratio, and reliability, intelligent routing mechanisms that can adapt and be aware of the context are necessary. Recent advancements in artificial intelligence, particularly deep reinforcement learning (DRL), have shown strong potential for addressing complex decision-making problems in dynamic network environments. DRL combines reinforcement learning (RL)’s trial-and-error policy optimization with deep neural networks’ powerful function approximation capability, enabling scalable solutions in high-dimensional state spaces. In the context of SAGIN, DRL enables routing agents to learn optimal forwarding policies directly from network interactions without relying on predefined traffic models or static assumptions. Some recent papers apply DRL for SAGIN routing and traffic control. Adaptive routing in space–air–ground environment using GAN-enhanced DRL has been studied in IEEE Transactions on Cognitive Communications and Networking to improve load balance and dynamic routing [3]. Traffic optimization for integrated satellite–terrestrial systems using DRL has been investigated in the IEEE Journal on Selected Areas in Communications. AI-based routing and QoS optimization for future integrated networks have been surveyed, and emerging research has been considered in a paper on arXiv [4].

However, existing routing approaches in SAGIN still face several limitations. Many traditional and AI-based routing methods focus on a single performance metric, such as shortest path or delay minimization, without jointly considering throughput, latency, and energy efficiency. In highly dynamic SAGIN environments, rapid topology variations, intermittent connectivity, and heterogeneous communication links can significantly degrade routing performance. Moreover, several existing approaches rely on predefined traffic assumptions and static optimization strategies, which reduce adaptability under real-time network conditions. Therefore, there is a strong need for intelligent and adaptive routing mechanisms capable of learning optimal routing policies dynamically while satisfying multiple QoS requirements simultaneously.

To address these challenges, this paper proposes a DRL-based adaptive routing framework for SAGIN. The proposed framework dynamically selects routing paths by continuously interacting with the network environment and learning optimal routing policies based on changing network conditions. Unlike conventional shortest-path routing approaches, the proposed DRL model considers multiple QoS parameters simultaneously, including throughput, end-to-end delay, and energy consumption. A reward-driven learning mechanism enables the routing agent to make adaptive and intelligent decisions under varying traffic loads and heterogeneous network conditions.

The proposed work introduces several important innovations. First, a dynamic SAGIN topology model integrating space, aerial, and terrestrial communication nodes is developed for realistic routing evaluation. Second, a DRL-based adaptive routing agent is designed to optimize routing decisions in dynamic and heterogeneous network environments. Third, a multi-objective reward function is introduced to jointly maximize throughput while minimizing delay and energy consumption. Finally, the proposed framework is evaluated against conventional routing approaches using extensive simulation analysis to demonstrate improvements in network performance and convergence characteristics.

The major contributions of this paper are summarized as follows:

• 1.Development of a dynamic SAGIN network model considering delay, bandwidth, and energy constraints.
• 2.Design of a DRL-based adaptive routing mechanism for intelligent path selection.
• 3.Introduction of a multi-objective reward computation strategy for QoS-aware routing optimization.
• 4.Comparative performance evaluation with traditional routing approaches under varying traffic conditions.
• 5.Detailed analysis of throughput, delay, energy efficiency, and convergence performance.

The remainder of this paper is organized as follows. Section II presents the literature review related to SAGIN routing and DRL-based optimization techniques. Section III describes the proposed DRL-based adaptive routing framework and system model. Section IV explains the simulation setup and performance metrics. Section V presents the experimental results and comparative analysis. Finally, Section VI concludes the paper and discusses future research directions.

II.LITERATURE REVIEW

A DRL-based QoS-aware routing protocol for SAGIN aims to address these challenges by formulating routing as a Markov decision process (MDP), where the state space describes network conditions, for example, link quality, queue occupancy, residual bandwidth, node mobility, or delay [5]. The action space denotes next-hop or path selection across the heterogeneous segments [6]. The reward function can incorporate multiple QoS objectives such as minimizing latency, maximizing throughput, and minimizing packet loss [7]. By continuously interacting with the environment and optimizing cumulative rewards, the DRL agent learns adaptive routing strategies that dynamically balance traffic loads and satisfy QoS requirements under rapidly changing network conditions [8]. Hence, the fusion of DRL with SAGIN routing could open avenues for intelligent, scalable, and quality-of-service-oriented communications in future 6G networks [9]. Although heuristic-based enhancements such as AC-AODV improve energy-aware routing, they lack adaptive decision-making capabilities under dynamic topology variations [10]. DRL provides a promising alternative by enabling real-time policy learning through environmental interaction [11]. In the context of non-terrestrial networks (NTNs), recent studies have explored intelligent optimization strategies for service orchestration and routing [12]. A transformer-assisted ant colony optimization (ACO) framework was proposed for service function chaining (SFC) deployment in Low Earth Orbit (LEO) satellite networks, where routing and virtual network function (VNF) embedding are jointly optimized to enhance cost efficiency and communication success rate [13]. By leveraging attention-based encoding of SFC requests and neural-assisted path selection, the approach demonstrates improved delay and deployment cost performance [14]. However, the study primarily addresses SFC embedding within satellite constellations and does not consider energy-aware routing or multi-layer space–air–ground integration [15]. Furthermore, adaptive RL mechanisms for dynamic routing decisions remain unexplored [16]. The rapid evolution of SAGINs and NTNs has introduced new challenges in routing, resource allocation, and service orchestration due to highly dynamic topologies, heterogeneous architectures, and limited onboard resources [17]. Recent studies have explored intelligent optimization techniques to address these complexities. To enhance service provisioning in satellite-based systems, transformer-assisted ACO frameworks have been proposed for SFC deployment in LEO NTNs [18]. By jointly addressing routing and VNF embedding, these approaches leverage attention mechanisms to encode SFC requests and neural-enhanced ACO algorithms to identify cost-efficient deployment paths. Simulation results demonstrate improvements in cost efficiency, communication success rate, and delay performance [19]. However, these solutions primarily focus on SFC embedding within satellite layers and do not explicitly address energy-aware routing or multi-layer SAGIN optimization under highly dynamic conditions [20]. Within the broader SAGIN context, intelligent resource orchestration has been investigated through the integration of graph pointer neural networks (GPNNs) and RL. Under an SDN-based SAGIN architecture, joint optimization of virtual node and link deployment has been formulated to maximize resource utilization while minimizing deployment latency [21]. By exploiting graph-based representations and adaptive learning, such frameworks outperform conventional RL-based schemes in terms of long-term reward and acceptance ratio. Nevertheless, the emphasis remains on SFC mapping and resource allocation rather than dynamic routing optimization, and energy-efficient path selection strategies are not comprehensively considered [22]. Routing-specific research in SAGIN has also explored cross-layer optimization mechanisms. A vector-weighted topology model incorporating signal-to-noise ratio (SNR), delay variation, and queuing length has been proposed to characterize heterogeneous network conditions [23]. Combined with Wiener channel prediction and M/M/1 queuing models, an ACO-based cross-layer routing algorithm has been developed to improve packet delivery performance in time-varying environments. Although the decentralized nature of ACO enhances adaptability, the approach relies on heuristic optimization and lacks learning-based policy evolution, limiting its capability to achieve long-term optimal performance in large-scale dynamic SAGIN scenarios [24]. Energy-efficient routing has further been studied through mathematical optimization frameworks. A two-scale time-expanded graph (TTEG) model has been introduced to capture topology evolution across large timescales while modeling mission-driven data transmission at smaller timescales. The energy minimization problem has been formulated as a mixed-integer nonlinear program (MINLP), jointly optimizing link assignment, storage allocation, power control, and routing decisions. While iterative decomposition techniques reduce computational burden, the centralized and computationally intensive nature of MINLP formulations restricts real-time scalability and adaptability in highly dynamic environments [25]. Similarly, heuristic energy-aware routing protocols such as ant colony-based AODV variants have been developed to improve link lifetime and network stability in integrated space–air–ground systems [26]. These approaches enhance energy utilization compared to conventional AODV by incorporating pheromone-based path selection. However, their performance heavily depends on parameter tuning and lacks adaptive learning mechanisms for handling rapidly evolving topologies.

Research Gap: Although considerable efforts have been devoted to routing optimization within SAGIN, most existing approaches are based on heuristic algorithms such as ACO or algorithms. While some recent studies have applied AI-based techniques, there has been limited research on employing DRL to improve throughput, latency, and energy consumption simultaneously in dynamic multi-layer SAGIN. Additionally, the DRL-based approaches that do exist tend to focus on resource allocation or task offloading, rather than on end-to-end routing decisions. There is a gap in developing an adaptive, energy-aware DRL-based routing framework capable of managing heterogeneous network conditions and improving QoS.

III.SYSTEM MODEL AND PROBLEM FORMULATION

The proposed study models the SAGIN as a connected weighted directed graph with nodes representing satellites, UAVs, and ground stations, and edges representing communication links characterized by their delay and bandwidth. The routing problem is a multi-objective optimization problem consisting of minimizing end-to-end delay, increasing throughput, and minimizing packet loss under dynamic and heterogenous network conditions. To perform these operations, we model the routing process as an MDP with state space recording current network parameters such as delay, bandwidth available, queue length, and action space representing the next-hop node on the network. A reward function jointly optimizes QoS metrics by rewarding higher bandwidth and penalizing excessive delay and packet loss. We train our model using greedy exploration and experience replay to ensure stability. To measure the performance improvement, we compare our method with the traditional delay-based shortest-path routing algorithm in identical conditions. In our simulation environment, we simulate a heterogeneous SAGIN topology where there are ground, aerial, and space nodes where the link parameters are dynamically varying to simulate the actual network behavior. We test the algorithm over several independent runs to ensure its statistical reliability in terms of throughput, end-to-end delay, packet loss ratio, energy consumption, and scalability.

IV.PROPOSED METHODOLOGY

To enhance routing adaptability and performance, a DRL-based routing framework is introduced. In this framework, the routing process is modeled as a sequential decision-making problem, where the agent interacts with the network environment to determine optimal next-hop selections. At each step, the agent observes the network state, which includes relevant link characteristics such as delay and bandwidth, and selects an action corresponding to the next-hop node among available neighbors. A multi-objective reward function is designed to encourage higher throughput while penalizing excessive delay, enabling balanced optimization of performance metrics. Through iterative interaction and reward-driven learning, the agent learns a routing policy that maximizes cumulative discounted rewards, allowing adaptive and intelligent routing decisions under dynamic network conditions.

V.EXPERIMENTAL RESULTS AND DISCUSSION

VI.CONCLUSION

In this work, a DRL-based QoS-aware routing framework was proposed for SAGIN. The framework addressed the challenges of heterogeneous architecture, dynamic topology changes, and varying QoS requirements. The SAGIN environment was represented as a weighted graph. The routing problem was formulated as a constrained optimization problem. End-to-end delay, bandwidth availability, and link dynamics across space, air, and ground segments were considered. Unlike traditional routing protocols, the proposed DRL-based routing agent was not dependent on fixed metrics or predefined rules. It learned adaptive routing decisions through continuous interaction with the network environment. The routing task was modeled as an MDP. The agent learned an optimal routing policy by maximizing cumulative rewards. This helped balance multiple QoS objectives, including latency reduction, throughput improvement, and network stability. The mathematical system model included realistic delay and bandwidth constraints. It also considered path-based QoS metrics in the integrated satellite network. This provided a practical analytical basis for studying SAGIN deployment scenarios in future 6G systems. The results demonstrated that DRL improved routing performance in dynamic SAGIN environments. The framework showed potential for autonomous, scalable, and self-optimizing network operations. This study was limited to simulation-based evaluation. Real-world deployment issues such as satellite mobility, intermittent connectivity, and UAV energy limitations were not fully addressed.

Future work may focus on multi-agent DRL architectures. Additional studies may also investigate energy-aware routing, secure policy learning, and real-time deployment over large-scale satellite–UAV–terrestrial testbeds.