Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory

doi:10.23919/JSEE.2023.000012

Abstract

Abstract:

In real-time strategy (RTS) games, the ability of recognizing other players’ goals is important for creating artifical intelligence (AI) players. However, most current goal recognition methods do not take the player ’s deceptive behavior into account which often occurs in RTS game scenarios, resulting in poor recognition results. In order to solve this problem, this paper proposes goal recognition for deceptive agent, which is an extended goal recognition method applying the deductive reason method (from general to special) to model the deceptive agent’s behavioral strategy. First of all, the general deceptive behavior model is proposed to abstract features of deception, and then these features are applied to construct a behavior strategy that best matches the deceiver’s historical behavior data by the inverse reinforcement learning (IRL) method. Final, to interfere with the deceptive behavior implementation, we construct a game model to describe the confrontation scenario and the most effective interference measures.

Key words: deceptive path planning, inverse reinforcement learning (IRL), game theory, goal recognition

Yunxiu ZENG, Kai XU. Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory[J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 270-288.

Figures/Tables 17

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Fig 8

Table 1

Fig 9

Fig 10

Fig 11

Table 2

Fig 12

Table 3

Fig 13

Table 4

References 26

1	MIRSKY R, STERN R, GAL Y, et al Goal and plan recognition design using plan libraries. ACM Transactions on Intelligent Systems and Technology, 2019, 10 (2): 1- 23.
2	LORTHIOIR G, INOUE K Design adaptive AI for RTS game by learning player's build order. Proc. of the 29th International Joint Conference on Artificial Intelligence and the 16th Pacific Rim International Conference on Artificial Intelligence, 2020, 5194- 5195.
3	MASTERS P, SARDINA S Deceptive path-planning. Proc. of the 26th International Joint Conference on Artificial Intelligence, 2017, 4368- 4375.
4	XU K, ZENG Y X, QIN L, et al Single real goal, magnitude-based deceptive path-planning. Entropy, 2020, 22 (1): 88. doi: 10.3390/e22010088
5	PAUWELS E, DETON S W Hybrid emerging threats and information warfare: the story of the Cyber-AI deception machine. Proc. of the 21st Century Prometheus, 2020, 107- 124.
6	KATAKWAR H, AGGARWAL P, MAQBOOL Z, et al. Influence of probing action costs on adversarial decision-making in a deception game. https://www.researchgate.net/publication/351928087.
7	OLIVEIRA E D, DONADONI L, BORIERO S, et al Deceptive actions to improve the attribution of rationality to playing robotic agents. International Journal of Social Robotics, 2021, 13, 391- 405. doi: 10.1007/s12369-020-00647-8
8	BOWYER J B. Cheating: deception in war & magic, games & sports, sex & religion, business & con games, politics & espionage, art & science. New York: St. Martin ’s Press, 1982.
9	KOTT A, MCENEANEY W M. Adversarial reasoning: computational approaches to reading the opponents mind. Boca Raton: Taylor of Francis Group, 2007.
10	XU K, HU Y, ZENG Y X, et al Improving the scalability of the magnitude-based deceptive path-planning using subgoal graphs. Entropy, 2020, 22 (2): 162. doi: 10.3390/e22020162
11	BONTRAGER P, KHALIFA A, ANDERSON D, et al Superstition in the network: deep reinforcement learning plays deceptive games. Proc. of the 15th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2019, 10- 16.
12	CHAKRABORTI T, KULKARNI A, SREEDHARAN S, et al Explicability? Legibility? Predictability? Transparency? Privacy? Security? The emerging landscape of interpretable agent behavior. Proc. of the 29th International Conference on Automated Planning and Scheduling, 2019, 86- 96.
13	MASTERS P, SARDINA S Goal recognition for rational and irrational agents. Proc. of the 18th International Conference on Autonomous Agents and Multiagent Systems, 2019, 440- 448.
14	MASTERS P, KIRLEY M, SMITH W Extended goal recognition: a planning-based model for strategic deception. Proc. of the 20th International Conference on Autonomous Agents and Multiagent Systems, 2021, 871- 879.
15	YUE S G, XU K, QIN L et al A two-layer semi-Markov model for recognizing the destination of a moving agent. Proc. of the IEEE International Conference on Mechatronics and Automation, 2015, 918- 922.
16	SHAH S I H, PIETRO G D. An overview of inverse reinforcement learning techniques. Intelligent Environments, 2021. DOI: 10.3233/AISE210097.
17	ZIEBART B D, MAAS A L, BAGNELL J A, et al Maximum entropy inverse reinforcement learning. Proc. of the 23rd Association for the Advancement of Artificial Intelligence, 2008, 1433- 1438.
18	ZENG Y X, XU K, QIN L, et al A semi-Markov decision model with inverse reinforcement learning for recognizing the destination of a maneuvering agent in real time strategy games. IEEE Access, 2020, 8, 15392- 15409. doi: 10.1109/ACCESS.2020.2967642
19	AMADO L, AIRES J P, PEREIRA R F, et al. LSTM-based goal recognition in latent space. Proc. of the International Joint Conference on Neural Networks, 2018. DOI: 10.48550/arXiv.1808.05249.
20	MAYNARD M, DUHAMEL T, KABANZA F. Cost-based goal recognition meets deep learning. Proc. of the AAAI Workshops, 2019. DOI:arXiv:1911.10074.
21	BRAYNOV S Adversarial planning and plan recognition: two sides of the same coin. Proc. of the Secure Knowledge Management Workshop, 2006, 67- 70.
22	ANG S, CHAN H, JIANG A X, et al Game-theoretic goal recognition models with applications to security domains. Proc. of the International Conference on Decision and Game Theory for Security, 2017, 256- 272.
23	RAMIREZ M, GEFFNER H Plan recognition as planning. Proc. of the 21st International Joint Conference on Artifical Intelligence, 2009, 1778- 1783.
24	WANG X H, LIU X H, WANG Y C, et al Research on path planning of mobile robot based on improved A* algorithm. Proc. of the 3rd International Symposium on Autonomous Systems, 2019, 29- 31.
25	YIN Q J, YUE S G, ZHA Y B, et al A semi-Markov decision model for recognizing the destination of a maneuvering agent in real time strategy games. Mathematical Problems in Engineering, 2016, 2016, 1907971.
26	TASTAN B, CHANG Y, SUKTHANKAR G Learning to intercept opponents in first person shooter games. Proc. of the IEEE Conference on Computational Intelligence and Games, 2012, 100- 107.

Strategy	Goal	Training set sample	First feature	Second feature	Result/%
Dissimulation strategy	1	100	Feature (iii)	Feature (viii)	79.85
	2	100	Feature (iii)	Feature (viii)	98.26
	3	100	Feature (iii)	Feature (viii)	95.39
Simulation strategy	1	100	Feature (iii)	Feature (ix)	79.85
	2	100	Feature (iii)	Feature (ii)	98.26
	3	100	Feature (i)	Feature (ii)	95.39

Strategy	Goal	Training set samples	sim(j)	dissim(j)	Result/%
Dissimulation strategy	1	100	0.2	0.05	75.33
	2	100	0.1	0.3	60.87
	3	100	0.02	0.02	79.30
Simulation strategy	1	100	0.05	0.2	72.54
	2	100	0.3	0.01	78.75
	3	100	0.01	0.1	77.89

Strategy	Road network	Target 1		Target 2		Target 3
Strategy	Road network	Weighted path length	Path length	Weighted path length	Path length	Weighted path length	Path length
Simulation strategy	1	1110.9	1268	1085.2	1241	1555.8	1670
	2	1203.9	1370	1138.5	1303	1555.8	1670
	3	1203.9	1370	1138.5	1303	1555.8	1670
	4	1276.5	1460	1211	1393	1626	1769
	5	1276.5	1460	1211	1393	1555.8	1670
Dissimulation strategy	1	1260.7	1302	1172.9	1291	1762.2	1759
	2	1337.9	1399	1200.5	1332	1762.2	1759
	3	1337.9	1399	1200.5	1332	1762.2	1759
	4	1395.6	1459	1234.5	1353	1838	1849
	5	1366.7	1459	1197.5	1321	1787.7	1789

Deceptive strategy	Map	Goal recognition without deception/%	Goal recognition under deception/%	Goal recognition using IRL/%	Weighted length without deceptive modeling	Weighted length with deceptive modeling
Simulation strategy	Chicago sketch road network	85.5	33.7	82.4	1399.4	1347.7
	UT2004 map	84.6	42.5	74.6	16.2	18.1
	Moving-AI benchmarks	89.1	54.7	87.4	250.9	268.2
Dissimulation strategy	Chicago sketch road network	85.5	55.6	67.8	1433.5	1450.6
	UT2004 map	84.6	52.3	71.1	16.1	17.9
	Moving-AI benchmarks	89.1	67.5	79.1	178.2	197.1

[1]	Bing HUA, Rui NI, Mohong ZHENG, Yunhua WU, Zhiming CHEN. Cooperative game theory-based steering law design of a CMG system [J]. Journal of Systems Engineering and Electronics, 2023, 34(1): 185-196.
[2]	Shouyi LI, Mou CHEN, Qingxian WU, Yuhui WANG. Threat sequencing of multiple UCAVs with incomplete information based on game theory [J]. Journal of Systems Engineering and Electronics, 2022, 33(4): 986-996.
[3]	Ming ZHANG, Jianjun ZHU, Hehua WANG. Evolutionary game analysis of problem processing mechanism in new collaboration [J]. Journal of Systems Engineering and Electronics, 2021, 32(1): 136-150.
[4]	Jie ZHANG, Gang WANG, Shaohua YUE, Yafei SONG, Jiayi LIU, Xiaoqiang YAO. Multi-agent system application in accordance with game theory in bi-directional coordination network model [J]. Journal of Systems Engineering and Electronics, 2020, 31(2): 279-289.
[5]	Zhuoming LI, Xing CHEN, Yu ZHANG, Peng WANG, Wei QIANG, Ningqing LIU. Fuzzy mathematics and game theory based D2D multicast network construction [J]. Journal of Systems Engineering and Electronics, 2019, 30(1): 13-21.
[6]	Chao GENG, Shiyou QU, Yingying XIAO, Mei WANG, Guoqiang SHI, Tingyu LIN, Junjie XUE, Zhengxuan JIA. Diffusion mechanism simulation of cloud manufacturing complex network based on cooperative game theory [J]. Journal of Systems Engineering and Electronics, 2018, 29(2): 321-335.
[7]	Qilong SUN, Naiming QI, Longxu XIAO, Haiqi LIN. Differential game strategy in three-player evasion and pursuit scenarios [J]. Journal of Systems Engineering and Electronics, 2018, 29(2): 352-366.
[8]	Yufei Liu, Dechang Pi, and Qiyou Cheng. Ensemble kernel method: SVM classification based on game theory [J]. Systems Engineering and Electronics, 2016, 27(1): 251-.
[9]	Chen Chen, Xinbo Gao, Qingqi Pei, and Xiaoji Li. BNE-based concurrent transmission considering channel quality and its PSO searching strategy in Ad Hoc networks [J]. Journal of Systems Engineering and Electronics, 2012, 23(5): 649-660.
[10]	Guofang Nan, Zhifei Mao, and Minqiang Li. Game-theoretic approach to power and admission control in hierarchical wireless sensor networks [J]. Journal of Systems Engineering and Electronics, 2012, 23(2): 216-224.
[11]	Guopeng Zhang, Peng Liu, and Enjie Ding. Energy efficient resource allocation in non-cooperative multi-cell OFDMA systems [J]. Journal of Systems Engineering and Electronics, 2011, 22(1): 175-182.
[12]	Xiaohui Yu and Qiang Zhang. Fuzzy Nash equilibrium of fuzzy n-person non-cooperative game [J]. Journal of Systems Engineering and Electronics, 2010, 21(1): 47-56.
[13]	L¨u Na, Zhang Yuetong & Zhang Guopeng. Cooperative game theoretic load balancing scheme with admission control in IEEE 802.11 WLANs? [J]. Journal of Systems Engineering and Electronics, 2009, 20(6): 1361-1365.
[14]	Zhou Min, Deng Feiqi & Wu Sai. Coordination game model of co-opetition relationship on cluster supply chains [J]. Journal of Systems Engineering and Electronics, 2008, 19(3): 499-506.
[15]	Zhang Li, Zhang An, Zhang Yongfang & Shi Zhifu. Study on group air to ground attack-defends hierarchical dynamic decision-making [J]. Journal of Systems Engineering and Electronics, 2007, 18(3): 540-544.