{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T18:15:54Z","timestamp":1772907354170,"version":"3.50.1"},"reference-count":52,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2023,12,17]],"date-time":"2023-12-17T00:00:00Z","timestamp":1702771200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2020YFB1808003"],"award-info":[{"award-number":["2020YFB1808003"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["61801379"],"award-info":[{"award-number":["61801379"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2020JQ-647"],"award-info":[{"award-number":["2020JQ-647"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["2020YFB1808003"],"award-info":[{"award-number":["2020YFB1808003"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61801379"],"award-info":[{"award-number":["61801379"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["2020JQ-647"],"award-info":[{"award-number":["2020JQ-647"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Natural Science Foundation of Shaanxi Province of China","award":["2020YFB1808003"],"award-info":[{"award-number":["2020YFB1808003"]}]},{"name":"Natural Science Foundation of Shaanxi Province of China","award":["61801379"],"award-info":[{"award-number":["61801379"]}]},{"name":"Natural Science Foundation of Shaanxi Province of China","award":["2020JQ-647"],"award-info":[{"award-number":["2020JQ-647"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Traditional low earth orbit (LEO) satellite networks are typically independent of terrestrial networks, which develop relatively slowly due to the on-board capacity limitation. By integrating emerging mobile edge computing (MEC) with LEO satellite networks to form the business-oriented \u201cend-edge-cloud\u201d multi-level computing architecture, some computing-sensitive tasks can be offloaded by ground terminals to satellites, thereby satisfying more tasks in the network. How to make computation offloading and resource allocation decisions in LEO satellite edge networks, nevertheless, indeed poses challenges in tracking network dynamics and handling sophisticated actions. For the discrete-continuous hybrid action space and time-varying networks, this work aims to use the parameterized deep Q-network (P-DQN) for the joint computation offloading and resource allocation. First, the characteristics of time-varying channels are modeled, and then both communication and computation models under three different offloading decisions are constructed. Second, the constraints on task offloading decisions, on remaining available computing resources, and on the power control of LEO satellites as well as the cloud server are formulated, followed by the maximization problem of satisfied task number over the long run. Third, using the parameterized action Markov decision process (PAMDP) and P-DQN, the joint computing offloading, resource allocation, and power control are made in real time, to accommodate dynamics in LEO satellite edge networks and dispose of the discrete-continuous hybrid action space. Simulation results show that the proposed P-DQN method could approach the optimal control, and outperforms other reinforcement learning (RL) methods for merely either discrete or continuous action space, in terms of the long-term rate of satisfied tasks.<\/jats:p>","DOI":"10.3390\/s23249885","type":"journal-article","created":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T11:28:07Z","timestamp":1702898887000},"page":"9885","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Computation Offloading and Resource Allocation Based on P-DQN in LEO Satellite Edge Networks"],"prefix":"10.3390","volume":"23","author":[{"given":"Xu","family":"Yang","sequence":"first","affiliation":[{"name":"Xi\u2019an Institute of Space Radio Technology, Xi\u2019an 710100, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-7306-3803","authenticated-orcid":false,"given":"Hai","family":"Fang","sequence":"additional","affiliation":[{"name":"Xi\u2019an Institute of Space Radio Technology, Xi\u2019an 710100, China"}]},{"given":"Yuan","family":"Gao","sequence":"additional","affiliation":[{"name":"Xi\u2019an Institute of Space Radio Technology, Xi\u2019an 710100, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3986-8060","authenticated-orcid":false,"given":"Xingjie","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Xi\u2019an University of Technology, Xi\u2019an 710048, China"}]},{"given":"Kan","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Xi\u2019an University of Technology, Xi\u2019an 710048, China"}]},{"given":"Zheng","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Xi\u2019an University of Technology, Xi\u2019an 710048, China"}]}],"member":"1968","published-online":{"date-parts":[[2023,12,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"18391","DOI":"10.1109\/ACCESS.2017.2735988","article-title":"LEO satellite constellation for Internet of Things","volume":"5","author":"Qu","year":"2017","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1109\/MNET.2018.1800182","article-title":"Heterogeneous space and terrestrial integrated networks for IoT: Architecture and challenges","volume":"33","author":"Chien","year":"2019","journal-title":"IEEE Netw."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"156","DOI":"10.23919\/JCC.2020.12.011","article-title":"System integration of terrestrial mobile communication and satellite communication\u2014The trends, challenges and key technologies in B5G and 6G","volume":"17","author":"Chen","year":"2020","journal-title":"China Commun."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1109\/MWC.001.1900333","article-title":"Vision, requirements, and technology trend of 6G: How to tackle the challenges of system coverage, capacity, user data-rate and movement speed","volume":"27","author":"Chen","year":"2020","journal-title":"IEEE Wirel. Commun."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1478","DOI":"10.1109\/JSAC.2020.2986851","article-title":"SFC-based service provisioning for reconfigurable space-air-ground integrated networks","volume":"38","author":"Wang","year":"2020","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"3256","DOI":"10.1109\/TVT.2020.2964659","article-title":"Integrated resource management for terrestrial-satellite systems","volume":"69","author":"Fu","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1109\/MNET.2018.1800164","article-title":"Cooperative transmission in integrated terrestrial-satellite networks","volume":"33","author":"Zhu","year":"2019","journal-title":"IEEE Netw."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/CC.2018.8438271","article-title":"Satellite communications integration with terrestrial networks","volume":"15","author":"Kapovits","year":"2018","journal-title":"China Commun."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Yang, Y., Ma, M., Wu, H., Yu, Q., Zhang, P., You, X., Wu, J., Peng, C., Yum, T.-S.P., and Shen, S. (2022). 6G network AI architecture for everyone-centric customized services. IEEE Netw., 1\u201310.","DOI":"10.1109\/MNET.124.2200241"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1109\/MNET.001.1900652","article-title":"Envisioning device-to-device communications in 6G","volume":"34","author":"Zhang","year":"2020","journal-title":"IEEE Netw."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Lakew, D.S., Tran, A.T., Masood, A., Dao, N.N., and Cho, S. (2023, January 11\u201314). A Review on Satellite-Terrestrial Integrated Wireless Networks: Challenges and Open Research Issues. Proceedings of the International Conference on Information Networking (ICOIN), Bangkok, Thailand.","DOI":"10.1109\/ICOIN56518.2023.10049009"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1109\/MNET.106.2100622","article-title":"Integrated satellite-terrestrial networks: Architectures, key techniques, and experimental progress","volume":"36","author":"Sun","year":"2022","journal-title":"IEEE Netw."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"974","DOI":"10.1007\/s11036-012-0368-0","article-title":"A survey of computation offloading for mobile systems","volume":"18","author":"Kumar","year":"2013","journal-title":"Mob. Netw. Appl."},{"key":"ref_14","first-page":"129","article-title":"Decentralized computation offloading game for mobile cloud computing","volume":"26","author":"Chen","year":"2014","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"637","DOI":"10.1109\/JIOT.2016.2579198","article-title":"Edge computing: Vision and challenges","volume":"3","author":"Shi","year":"2016","journal-title":"IEEE Internet Things J."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1594","DOI":"10.1109\/TCOMM.2017.2787700","article-title":"Computation offloading and resource allocation in mixed fog\/cloud computing systems with min-max fairness guarantee","volume":"66","author":"Du","year":"2017","journal-title":"IEEE Trans. Commun."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"7432","DOI":"10.1109\/TVT.2017.2672701","article-title":"Joint computation offloading and interference management in wireless cellular networks with mobile edge computing","volume":"66","author":"Wang","year":"2017","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Chang, Z., Zhou, Z., Ristaniemi, T., and Niu, Z. (2017, January 4\u20138). Energy efficient optimization for computation offloading in fog computing system. Proceedings of the IEEE Global Communications Conference (IEEE GLOBECOM), Singapore.","DOI":"10.1109\/GLOCOM.2017.8254207"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1109\/JIOT.2021.3126825","article-title":"Integrated satellite-terrestrial networks toward 6G: Architectures, applications, and challenges","volume":"9","author":"Zhu","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Brik, B., Frangoudis, P.A., and Ksentini, A. (2020, January 7\u201311). Service-oriented MEC applications placement in a federated edge cloud architecture. Proceedings of the IEEE International Conference on Communications (ICC), Dublin, Ireland.","DOI":"10.1109\/ICC40277.2020.9148814"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"166079","DOI":"10.1109\/ACCESS.2019.2953172","article-title":"Device-enhanced MEC: Multi-access edge computing (MEC) aided by end device computation and caching: A survey","volume":"7","author":"Mehrabi","year":"2019","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1109\/TII.2022.3177415","article-title":"Service Function Chaining in Industrial Internet of Things With Edge Intelligence: A Natural Actor-Critic Approach","volume":"19","author":"Li","year":"2023","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"11365","DOI":"10.1109\/ACCESS.2018.2805798","article-title":"Energy efficient task caching and offloading for mobile edge computing","volume":"6","author":"Hao","year":"2018","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Trakadas, P., Masip-Bruin, X., Facca, F.M., Spantideas, S.T., Giannopoulos, A.E., Kapsalis, N.C., Martins, R., Bosani, E., Ramon, J., and Prats, R.G. (2022). A reference architecture for cloud-edge meta-operating systems enabling cross-domain, data-intensive, ML-assisted applications: Architectural overview and key concepts. Sensors, 22.","DOI":"10.3390\/s22229003"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Militano, L., Arteaga, A., Toffetti, G., and Mitton, N. (2023). The cloud-to-edge-to-IoT continuum as an enabler for search and rescue operations. Future Internet, 15.","DOI":"10.3390\/fi15020055"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1109\/TWC.2019.2943563","article-title":"Optimal task offloading and resource allocation in mobile-edge computing with inter-user task dependency","volume":"19","author":"Yan","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"5288","DOI":"10.1109\/TWC.2021.3066650","article-title":"Joint service caching, computation offloading and resource allocation in mobile edge computing systems","volume":"20","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"4522","DOI":"10.1109\/TWC.2021.3130649","article-title":"Joint Power Control and Computation Offloading for Energy-Efficient Mobile Edge Networks","volume":"21","author":"Wu","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1960","DOI":"10.1109\/TWC.2021.3108641","article-title":"Energy-efficient joint task offloading and resource allocation in OFDMA-based collaborative edge computing","volume":"21","author":"Tan","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"4957","DOI":"10.1109\/TWC.2019.2931321","article-title":"Energy efficient power allocation for delay constrained cognitive satellite terrestrial networks under interference constraints","volume":"18","author":"Ruan","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Shi, S., Li, G., An, K., Gao, B., and Zheng, G. (2017). Energy-efficient optimal power allocation in integrated wireless sensor and cognitive satellite terrestrial networks. Sensors, 17.","DOI":"10.3390\/s17092025"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Spantideas, S.T., Giannopoulos, A.E., Kapsalis, N.C., Kalafatelis, A., Capsalis, C.N., and Trakadas, P. (2021, January 7\u201310). Joint energy-efficient and throughput-sufficient transmissions in 5G cells with deep Q-learning. Proceedings of the IEEE International Mediterranean Conference on Communications and Networking (MeditCom), Athens, Greece.","DOI":"10.1109\/MeditCom49071.2021.9647592"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Hsieh, C.K., Chan, K.L., and Chien, F.T. (2021). Energy-efficient power allocation and user association in heterogeneous networks with deep reinforcement learning. Appl. Sci., 11.","DOI":"10.3390\/app11094135"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2059","DOI":"10.1109\/JSAC.2018.2864373","article-title":"Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach","volume":"36","author":"Liu","year":"2018","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"10863","DOI":"10.1109\/TVT.2022.3182647","article-title":"Digital twin assisted task offloading for aerial edge computing and networks","volume":"71","author":"Li","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"5725","DOI":"10.1109\/TWC.2023.3235997","article-title":"Energy efficient computation offloading in aerial edge networks with multi-agent cooperation","volume":"22","author":"Liu","year":"2023","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"5871","DOI":"10.1109\/TVT.2019.2907682","article-title":"Deep Q-learning aided networking, caching, and computing resources allocation in software-defined satellite-terrestrial networks","volume":"68","author":"Qiu","year":"2019","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"211","DOI":"10.23919\/JCC.2020.05.016","article-title":"Deep reinforcement learning based joint edge resource management in maritime network","volume":"17","author":"Xu","year":"2020","journal-title":"China Commun."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1117","DOI":"10.1109\/JSAC.2019.2906789","article-title":"Space\/aerial-assisted computing offloading for IoT applications: A learning-based approach","volume":"37","author":"Cheng","year":"2019","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"55915","DOI":"10.1109\/ACCESS.2020.2982356","article-title":"Latency and energy optimization for MEC enhanced SAT-IoT networks","volume":"8","author":"Cui","year":"2020","journal-title":"IEEE Access"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wang, B., Xie, J., Huang, D., and Xie, X. (2022, January 10\u201312). A Computation Offloading Strategy for LEO Satellite Mobile Edge Computing System. Proceedings of the International Conference on Communication Software and Networks (ICCSN), Chongqing, China.","DOI":"10.1109\/ICCSN55126.2022.9817597"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"996","DOI":"10.1109\/TGCN.2022.3186792","article-title":"Optimal Computation Offloading in Collaborative LEO-IoT Enabled MEC: A Multiagent Deep Reinforcement Learning Approach","volume":"7","author":"Lyu","year":"2023","journal-title":"IEEE Trans. Green Commun. Netw."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Maattanen, H.L., Hofstrom, B., Euler, S., Sedin, J., Lin, X., Liberg, O., Masini, G., and Israelsson, M. (2019, January 9\u201313). 5G NR Communication over GEO or LEO Satellite Systems: 3GPP RAN Higher Layer Standardization Aspects. Proceedings of the IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA.","DOI":"10.1109\/GLOBECOM38437.2019.9014090"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"6606","DOI":"10.1109\/TWC.2021.3075289","article-title":"Machine learning-based resource allocation in satellite networks supporting internet of remote things","volume":"20","author":"Zhou","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"9582","DOI":"10.1109\/TWC.2022.3178171","article-title":"Adapting to dynamic LEO-B5G systems: Meta-critic learning based efficient resource scheduling","volume":"21","author":"Yuan","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Dong, Y., Wang, L., Wang, J., Hu, X., Zhang, H., Yu, F.R., and Leung, V.C.M. (2023). Accelerating Wireless Federated Learning via Nesterov\u2019s Momentum and Distributed Principle Component Analysis. IEEE Trans. Wirel. Commun.","DOI":"10.1109\/TWC.2023.3329375"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"7566","DOI":"10.1109\/TWC.2022.3159779","article-title":"An online zero-forcing Precoder for weighted sum-rate maximization in green CoMP systems","volume":"21","author":"Dong","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_48","unstructured":"Xiong, J., Wang, Q., Yang, Z., Sun, P., Han, L., Zheng, Y., Fu, H., Zhang, T., Liu, J., and Liu, H. (2018). Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Masson, W., Ranchod, P., and Konidaris, G. (2016, January 12\u201317). Reinforcement learning with parameterized actions. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10226"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Wang, Y., Zhang, J., Zhang, X., Wang, P., and Liu, L. (2018, January 19\u201321). A computation offloading strategy in satellite terrestrial networks with double edge computing. Proceedings of the IEEE International Conference on Communication Systems (ICCS), Chengdu, China.","DOI":"10.1109\/ICCS.2018.8689224"},{"key":"ref_52","first-page":"1949","article-title":"Joint computation offloading and resource allocation for D2D-Assisted mobile edge computing","volume":"16","author":"Jiang","year":"2023","journal-title":"IEEE Trans. Serv. Comput."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/24\/9885\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:40:22Z","timestamp":1760132422000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/24\/9885"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,17]]},"references-count":52,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["s23249885"],"URL":"https:\/\/doi.org\/10.3390\/s23249885","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,17]]}}}