Curriculum learning for adaptive multi-UAV swarm target search in obstacle environment

doi:10.23919/JSEE.2026.000009

Abstract

Abstract:

Previous studies have paid limited attention to the critical role of unmanned aerial vehicle (UAV) altitude variation in determining three-dimensional (3D) search efficiency. To address this gap, this paper introduces a deep reinforcement learning framework integrated with adaptive matrix optimization in multi-UAV target search. The proposed framework explicitply models altitude variations’ impacts on observation range and detection accuracy. To balance flight safety, coverage, and detection precision, deep Q-network (DQN)-altitude matrix optimization (AMO) discretizes 3D trajectories into horizontal paths and vertical altitude decisions, effectively reducing the problem dimensionality. Furthermore, a curriculum learning approach is employed to decompose the search process into phased sub-tasks, each with customized decision rules and reward mechanisms. This hierarchical strategy accelerates agent learning and enhances performance in complex scenarios. Comprehensive experimental evaluations in simulated environments demonstrate that the proposed DQN-AMO outperforms benchmark methods in both robustness and generalization.

Key words: multi-UAV swarm, adaptive optimization, target search, curriculum learning

Xin Cao, He Luo, Guoqiang Wang, Yingying Ma, Jue Zhang. Curriculum learning for adaptive multi-UAV swarm target search in obstacle environment[J]. Journal of Systems Engineering and Electronics, 2026, 37(4): 1297-1316.

Figures/Tables 16

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Table 1

Fig 6

Table 2

Table 3

Table 4

Fig 7

Table 5

Average search steps and search success rate between the proposed method and the baseline methods in four search scenes"

Scene	Size	DQN		MTS-ACO		Ours-CL		The proposed method		Target
Scene	Size	ASS	SSR/%	ASS	SSR/%	ASS	SSR/%	ASS	SSR/%	Target
I	10×10	6.68	100	6.41	100	6.22	100	6.09	100	Target 1
	20×20	9.99	98	8.65	100	9.08	100	8.04	100
	30×30	15.98	87	13.78	94	15.01	90	12.71	97
	40×40	29.46	72	25.31	82	27.22	77	23.88	89
	50×50	58.02	47	48.59	62	53.14	53	40.79	74
I	10×10	8.01	99	7.03	100	7.36	100	6.71	100	Target 2
	20×20	13.65	90	10.78	96	12.23	94	10.18	98
	30×30	23.77	76	18.25	86	20.61	81	16.99	91
	40×40	43.22	61	31.46	74	36.52	68	28.25	83
	50×50	71.04	30	64.29	49	68.14	41	55.55	63
	10×10	8.99	98	7.99	100	8.39	99	7.48	100	Target 3
	20×20	15.96	84	12.31	92	13.27	89	11.77	97
	30×30	32.93	67	24.71	80	27.97	75	21.06	86
	40×40	51.23	52	40.35	67	47.90	59	34.76	77
	50×50	84.64	17	71.65	37	77.27	29	65.33	54
II	10×10	7.05	100	6.30	100	6.30	100	6.18	100	Target 1
	20×20	10.87	88	9.73	93	9.81	92	8.77	100
	30×30	21.98	76	16.31	87	17.80	85	14.67	95
	40×40	42.91	60	30.98	72	35.39	69	26.14	83
	50×50	73.10	28	62.06	47	69.57	32	52.65	66
	10×10	8.18	96	7.42	98	7.72	97	7.14	100	Target 2
	20×20	15.10	77	12.81	86	14.39	86	11.97	96
	30×30	33.60	64	25.23	77	28.35	73	19.20	86
	40×40	53.00	49	37.21	64	46.25	58	32.12	76
	50×50	83.20	16	71.79	32	81.13	20	63.93	53
	10×10	9.71	94	8.43	98	8.81	96	8.15	98	Target 3
	20×20	19.27	70	15.38	81	16.60	80	13.97	91
	30×30	41.27	55	31.34	70	35.56	65	25.78	80
	40×40	60.09	40	48.52	58	54.41	49	39.23	70
	50×50	91.12	9	80.78	24	87.76	11	70.81	41
III	10×10	8.30	99	7.34	100	7.96	100	6.73	100	Target 1
	20×20	12.99	82	10.33	90	11.37	87	9.81	97
	30×30	26.96	74	21.01	84	23.44	79	16.36	91
	40×40	48.25	56	37.13	67	43.51	62	30.98	75
	50×50	89.07	13	75.33	28	79.48	22	67.13	40
	10×10	10.08	93	8.89	99	9.56	96	8.55	100	Target 2
	20×20	22.05	71	15.99	81	19.26	78	13.14	90
	30×30	41.02	61	31.71	73	38.24	66	25.34	84
	40×40	61.96	44	46.33	59	49.66	55	39.27	64
	50×50	94.98	8	81.24	20	88.84	14	73.02	31
	10×10	11.97	87	10.61	93	10.87	90	9.79	96	Target 3
	20×20	29.06	63	20.55	74	22.33	70	17.76	85
	30×30	49.79	50	35.22	67	43.26	57	31.80	77
	40×40	67.53	34	55.24	51	58.71	47	48.24	56
	50×50	98.04	5	89.30	14	92.80	10	77.62	25
IV	10×10	9.37	92	8.57	100	8.91	98	7.99	100	Target 1
	20×20	16.02	70	12.95	84	13.90	81	11.19	92
	30×30	42.40	62	25.62	78	34.36	70	20.97	87
	40×40	60.13	46	50.34	56	58.33	50	39.82	66
	50×50	97.77	7	84.96	18	87.55	14	72.31	32
	10×10	11.33	86	9.99	92	10.62	92	8.61	95	Target 2
	20×20	26.75	58	21.36	74	22.55	71	13.09	84
	30×30	50.34	47	39.19	65	42.74	57	29.80	76
	40×40	64.48	38	58.63	48	62.20	40	49.79	57
	50×50	99.02	2	87.61	14	96.82	6	80.55	22
IV	10×10	14.67	78	11.76	89	12.28	85	9.34	91	Target 3
	20×20	33.56	49	24.98	66	27.23	58	20.79	78
	30×30	59.01	36	46.76	52	53.36	44	39.88	64
	40×40	77.73	24	74.17	26	70.80	32	56.55	50
	50×50	100.00	0	97.24	8	98.10	4	83.40	18

Table 5

Fig 8

Fig 9

Fig 10

Table 6

References 38

1	Shakhatreh H, Sawalmeh A H, Al-Fuqaha A, et al Unmanned aerial vehicles (UAVs): a survey on civil applications and key research challenges. IEEE Access, 2019, 7, 48572. doi: 10.1109/ACCESS.2019.2909530
2	Xu Y, Li X B, Meng X P, et al An iterated greedy heuristic for collaborative human-UAV search of missing tourists. Knowledge-Based Systems, 2024, 286, 111409. doi: 10.1016/j.knosys.2024.111409
3	Liu D Q, Zhu X M, Bao W D, et al Smart: vision-based method of cooperative surveillance and tracking by multiple UAVs in the urban environment. IEEE Trans. on Intelligent Transportation Systems, 2022, 23, 24941. doi: 10.1109/TITS.2022.3203411
4	Zhen Z Y, Xing D J, Gao C Cooperative search-attack mission planning for multi-UAV based on intelligent self-organized algorithm. Aerospace Science and Technology, 2018, 76, 402. doi: 10.1016/j.ast.2018.01.035
5	Liu L X, Wang X, Yang X, et al Path planning techniques for mobile robots: review and prospect. Expert Systems with Applications, 2023, 227, 120254. doi: 10.1016/j.eswa.2023.120254
6	Xue Y T, Chen W A UAV navigation approach based on deep reinforcement learning in large cluttered 3D environments. IEEE Trans. on Vehicular Technology, 2023, 72, 3001. doi: 10.1109/TVT.2022.3218855
7	Otto A, Agatz N, Campbell J, et al Optimization approaches for civil applications of unmanned aerial vehicles (UAVs) or aerial drones: a survey. Networks, 2018, 72, 411. doi: 10.1002/net.21818
8	Xiao Y G, Li Y H, Liu H, et al Adaptive large neighbor- hood search algorithm with reinforcement search strategy for solving extended cooperative multi task assignment problem of UAVs. Information Sciences, 2024, 679, 121068. doi: 10.1016/j.ins.2024.121068
9	Zhao B C, Huo M Y, Li Z, et al Clustering-based hyper-heuristic algorithm for multi-region coverage path planning of heterogeneous UAVs. Neurocomputing, 2024, 610, 128528. doi: 10.1016/j.neucom.2024.128528
10	Jia Z, Du X, Dong Q C, et al Distributed collaborative complete coverage path planning based on hybrid strategy. Journal of Systems Engineering and Electronics, 2024, 35 (2): 463. doi: 10.23919/JSEE.2023.000118
11	Xu X J, Xie C Y, Luo Z F, et al A multi-objective evolu- tionary algorithm based on dimension exploration and discrepancy evolution for UAV path planning problem. Information Sciences, 2024, 657, 119977. doi: 10.1016/j.ins.2023.119977
12	Al-Hourani A, Kandeepan S, Lardner S Optimal lap altitude for maximum coverage. IEEE Wireless Communication Letter, 2014, 3, 569. doi: 10.1109/LWC.2014.2342736
13	Mnih V, Kavukcuoglu K, Silver D, et al Human-level control through deep reinforcement learning. Nature, 2015, 518, 529. doi: 10.1038/nature14236
14	Soviany P, Ionescu R T, Rota P, et al Curriculum learning: a survey. International Journal of Computer Vision, 2022, 130, 1526. doi: 10.1007/s11263-022-01611-x
15	Tang H W, Sun W, Yu H S, et al. A multirobot target searching method based on bat algorithm in unknown environments [J]. Expert Systems with Application, 2020, 141; 112945.
16	Hu J Q, Wu H S, Zhan R J, et al Self-organized search-attack mission planning for UAV swarm based on wolf pack hunting behavior. Journal of Systems Engineering and Electronics, 2021, 32 (6): 1463. doi: 10.23919/jsee.2021.000124
17	Sun L, Wang J S, Wang J, et al Efficient joint deployment of multi-UAVs for target tracking in traffic big data. IEEE Trans. on Intelligent Transportation Systems, 2024, 25 (7): 7780. doi: 10.1109/TITS.2024.3370250
18	Stamatopoulos M N, Banerjee A, Niko-Lakopoulos G Conflict-free optimal motion planning for parallel aerial 3D printing using multiple UAVs. Expert systems with applications, 2024, 246, 123201. doi: 10.1016/j.eswa.2024.123201
19	Li J X, Zhao Z S, Qin X H Adaptive sliding mode control using a novel fully feedback recurrent neural network for quad-rotor UAVs. Neurocomputing, 2024, 610, 128592. doi: 10.1016/j.neucom.2024.128592
20	Li J, Xiong Y H, She J H UAV path planning for target coverage task in dynamic environment. IEEE Internet of Things Journal, 2023, 10 (20): 17734. doi: 10.1109/JIOT.2023.3277850
21	Hentati A I, Fourati L C, Rezgui J Cooperative UAVs framework for mobile target search and tracking. Computers and Electrical Engineering, 2022, 101, 107992. doi: 10.1016/j.compeleceng.2022.107992
22	Yang Y F, Zhang C S, Liu Y, et al Deep reinforcement learning assisted novelty search in voronoi regions for con- strained multi-objective optimization. Swarm and Evolution- ary Computation, 2024, 91, 101732. doi: 10.1016/j.swevo.2024.101732
23	Huang C C, Du B, Chen M Multi-UAV cooperative online searching based on voronoi diagrams. IEEE Trans. on Aerospace and Electronic Systems, 2024, 60, 3038. doi: 10.1109/TAES.2024.3362320
24	Ma T X, Wang Y H, Li X T Convex combination multi- ple populations competitive swarm optimization for moving target search using UAVs. Information Sciences, 2023, 641, 119104. doi: 10.1016/j.ins.2023.119104
25	Guan W, Wang K Autonomous collision avoidance of unmanned surface vehicles based on improved a-star and dynamic window approach algorithms. IEEE Intelligent Transportation Systems Magazine, 2023, 15, 36. doi: 10.1109/mits.2022.3229109
26	Pehlivanoglu Y V, Pehlivanoglu P An enhanced genetic algorithm for path planning of autonomous UAV in target coverage problems. Applied Soft Computing, 2021, 112, 107796. doi: 10.1016/j.asoc.2021.107796
27	Phung M D, Ha Q P Motion-encoded particle swarm optimization for moving target search using UAVs. Applied Soft Computing, 2020, 97, 106705. doi: 10.1016/j.asoc.2020.106705
28	Alfeo A L, Cimino M G, Vaglini G Enhancing bio- logically inspired swarm behavior: metaheuristics to foster the optimization of UAVs coordination in target search. Computers & Operation Research, 2019, 110, 34. doi: 10.1016/j.cor.2019.05.021
29	Hu G, Cheng M, Houssein E H, et al: A novel co-evolutionary multigroup particle swarm optimization for multi-mission UAVs path planning. Swarm and Evolutionary Computation, 2025, 63, 102923.
30	Hu G, Huang F Y, Shu B, et al MAHACO: multi-algo- rithm hybrid ant colony optimizer for 3D path planning of a group of UAVs. Information Sciences, 2025, 694, 121714. doi: 10.1016/j.ins.2024.121714
31	Zheng Y J, Du Y C, Ling H F, et al Evolutionary collab- orative human-UAV search for escaped criminals. IEEE Trans. on Evolutionary Computation, 2019, 24, 217.
32	Schmidhuber J Deep learning in neural networks: an overview. Neural Networks, 2015, 61, 85. doi: 10.1016/j.neunet.2014.09.003
33	Ellis B, Cook J, Moalla S, et al SMACV2: an improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 2023, 36, 37567.
34	Vinyals O, Babuschkin I, Czarnecki W M, et al Grandmaster level in StarCraft II using multi-agent reinforce- ment learning. Nature, 2019, 575, 350. doi: 10.1038/s41586-019-1724-z
35	Salameh H B, Alhafnawi M, Masadeh A, et al Federated reinforcement learning approach for detecting uncertain deceptive target using autonomous dual UAV system. Information Processing & Management, 2023, 60, 103149. doi: 10.1016/j.ipm.2022.103149
36	Zhang T, Zhou Q Y, Zheng Y, et al Improved proximal policy optimization for UAV tracking in complex environments. Knowledge-Based Systems, 2025, 30, 113627. doi: 10.1016/j.knosys.2025.113627
37	Wei D X, Zhang L, Liu Q, et al UAV swarm coopera- tive dynamic target search: a MAPPO-based discrete optimal control method. Drones, 2024, 8 (6): 214. doi: 10.3390/drones8060214
38	Xiao J, Yuan G H, Xue Y X, et al A deep reinforcement learning based distributed multi-UAV dynamic area coverage algorithm for complex environment. Neurocomputing, 2024, 595, 127904. doi: 10.1016/j.neucom.2024.127904

Method	Parameter	Value
DQN	Gamma	0.99
	Learning rate	0.0004
	Batch size	128
	Epsilon	0.01
MTS-ACO	Population size N	8
	Evaporation rate	0.5
	Values of the pheromones	1
	Heuristic influence	1
Ours-CL	Gamma	0.99
	Learning rate	0.00033
	Batch size	128
	Epsilon	0.01
DQN-AMO	Gamma	0.99
	Learning rate	0.0001
	Batch size	32
	Epsilon	0.05

Metric	Description
ASS	The average search steps to search the target in all episodes (including failure episodes)
SSR	The percentage of successful episodes in all episodes
ACA	The proportion of grids that have been visited over all grids
DA	The accuracy of the agent’s observation of the ground target

Target	Scene	Mode	ASS/%	SSR/%	ACA%	DA%
1	I	Fixed	36.02	70	0.652	0.95
		Random	34.46	68	0.684	0.94
		The proposed mode	33.79	74	0.673	0.96
	II	Fixed	56.59	61	0.725	0.93
		Random	58.73	58	0.741	0.92
		The proposed mode	52.65	66	0.734	0.95
	III	Fixed	71.69	34	0.782	0.93
		Random	72.24	32	0.793	0.91
		The proposed mode	62.72	40	0.784	0.92
	IV	Fixed	78.56	27	0.795	0.91
		Random	80.72	25	0.799	0.90
		The proposed mode	72.31	32	0.801	0.91
2	I	Fixed	58.74	58	0.715	0.95
		Random	60.69	56	0.749	0.94
		The proposed mode	55.55	63	0.741	0.95
	II	Fixed	65.60	45	0.743	0.93
		Random	68.48	41	0.773	0.92
		The proposed mode	62.93	53	0.770	0.94
	III	Fixed	78.30	24	0.773	0.93
		Random	80.80	21	0.792	0.91
		The proposed mode	73.02	31	0.804	0.92
	IV	Fixed	85.48	17	0.780	0.91
		Random	86.29	15	0.797	0.89
		The proposed mode	80.55	22	0.799	0.91
3	I	Fixed	67.43	47	0.752	0.95
		Random	68.21	45	0.767	0.93
		The proposed mode	65.33	54	0.765	0.95
	II	Fixed	73.96	33	0.761	0.92
		Random	76.30	29	0.772	0.91
		The proposed mode	70.81	41	0.769	0.93
	III	Fixed	87.29	17	0.783	0.93
		Random	89.71	14	0.799	0.90
		The proposed mode	77.62	25	0.812	0.94
	IV	Fixed	90.21	12	0.794	0.91
		Random	94.36	9	0.817	0.88
		The proposed mode	83.40	18	0.806	0.91

[1]	Xiaoxuan Hu, Yanhong Liu, and Guoqiang Wang. Optimal search for moving targets with sensing capabilities using multiple UAVs [J]. Systems Engineering and Electronics, 2017, 28(3): 526-535.
[2]	Yang Chunhui, Liu Junxian, Chen Honghui & Luo Xueshan. Adaptive optimization of agile organization of command and control resource [J]. Journal of Systems Engineering and Electronics, 2009, 20(3): 558-564.