MMF-SLR: a sign language recognition method based on multi-modal feature using millimeter-wave radar

doi:10.23919/JSEE.2026.000061

Journal of Systems Engineering and Electronics ›› 2026, Vol. 37 ›› Issue (2): 357-366.doi: 10.23919/JSEE.2026.000061

• ELECTRONICS TECHNOLOGY • Previous Articles

MMF-SLR: a sign language recognition method based on multi-modal feature using millimeter-wave radar

Chang CUI¹^,²^,³(), Guiyan WEI²(), Xichao DONG¹^,²^,³^,*(), Cheng HU¹^,²^,³(), Jianping WANG¹^,²^,³()

¹Radar Technology Research Institute, Beijing Institute of Technology, Beijing 100081, China
²Chongqing Innovation Center, Beijing Institute of Technology, Chongqing 401120, China
³Chongqing Key Laboratory of Novel Civilian Radar, Chongqing 401120, China

Received:2024-07-16 Accepted:2026-03-26 Online:2026-04-18 Published:2026-04-30
Contact: Xichao DONG E-mail:cuichang0329@qq.com;weigy9677@163.com;xcdong@bit.edu.cn;cchchb@163.com;jianpingwang@bit.edu.cn
About author:
CUI Chang was born in 1994. She received her B.S. and Ph.D. degrees from the School of Information and Electronics, Beijing Institute of Technology, in 2016 and 2022, respectively. She is currently a Post-Doctoral Researcher with the Beijing Institute of Technology Chongqing Innovation Center. Her research interests include geosynchronous synthetic aperture radar (SAR) signal processing and moving target indication. E-mail: cuichang0329@qq.com

WEI Guiyan was born in 1996. She is pursuing her master degree in Chongqing Three Gorges University. Her research interest is cross-modal sign language recognition technology based on millimeter wave radar. E-mail: weigy9677@163.com

DONG Xichao was born in 1986. He received his B.S. and Ph.D. degrees from the School of Information and Electronics, Beijing Institute of Technology (BIT) in 2008 and 2014, respectively. He is currently an associate professor at BIT. His research interests include radar signal processing and weather radar. E-mail: xcdong@bit.edu.cn

HU Cheng was born in 1981. He received his B.S. and Ph.D. degrees from National University of Defense Technology and Beijing Institute of Technology respectively. Since 2009, he has been with Beijing Institute of Technology (BIT)’s School of Information and Electronics, and is now a full professor, doctoral supervisor, and Vice President of the School of Information and Electronics in BIT. His research interests include synthetic aperture radar (SAR) and entomological radar signal processing. E-mail: cchchb@163.com

WANG Jianping was born in 1987. He received his Ph.D. degree in electrical engineering from Delft University of Technology in 2018. He is currently a full professor at the School of Information and Electronics, Beijing Institute of Technology, focusing on microwave imaging, signal processing, and antenna array design. E-mail: jianpingwang@bit.edu.cn
Supported by:
This work was supported by the Postdoctoral Science Foundation of Chongqing in China (CSTB2022NSCQ-BHX0699), and the Chongqing Special Program for Technological Innovation and Application Development (CSTB2023TIAD-KPX0065).

Abstract

Abstract:

Sign language recognition (SLR) can be improved by using millimeter-wave radar, which safeguards the privacy of those who are hearing-impaired. However, existing SLR methods do not fully utilize the unique features of radar echoes, resulting in limited accuracy. Sign language is composed of an individual’s poses and hand movements. To obtain these recognition features, this paper presents a multi-modal-feature-based SLR (MMF-SLR) network framework. This method first constructs a transformer-pose network to extract the human skeleton information, which represents the poses in sign language, from the radar images. Additionally, hand movement information can be represented by the range-Doppler sequence and micro-Doppler signatures. The human skeleton and hand movement information are input into a multimodal fusion network to achieve high-accuracy SLR. The experimental results demonstrate that the proposed method can enhance the recognition accuracy of the sign language with similar poses or movements compared to the traditional SLR methods.

Key words: sign language recognition, millimeter-wave radar, multimodal, pose estimation, micro-Doppler, range-Doppler

Chang CUI, Guiyan WEI, Xichao DONG, Cheng HU, Jianping WANG. MMF-SLR: a sign language recognition method based on multi-modal feature using millimeter-wave radar[J]. Journal of Systems Engineering and Electronics, 2026, 37(2): 357-366.

Figures/Tables 17

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Table 1

Fig 8

Table 2

Table 3

Table 4

Table 5

Fig 9

Table 6

Fig 10

Table 7

References 30

1	VERMIGLIO A J, FANG X M The World Health Organization (WHO) hearing impairment guidelines and a speech recognition in noise (SRN) disorder. International Journal of Audiology, 2022, 61 (10): 818- 825. doi: 10.1080/14992027.2021.1976424
2	TUBAIZ N, SHANABLEH T, ASSALEH K Glove-based continuous Arabic sign language recognition in user-dependent mode. IEEE Trans. on Human-Machine Systems, 2015, 45 (4): 526- 533. doi: 10.1109/THMS.2015.2406692
3	GUO W C, WANG M, SHENG X J, et al Towards semi-supervised myoelectric finger motion recognition based on spatial motor units activation. Science China Technological Sciences, 2022, 65 (6): 1232- 1242. doi: 10.1007/s11431-022-2035-9
4	ZOU Y X, CHENG L, HAN L J, et al Multi-modal fusion for robust hand gesture recognition based on heterogeneous networks. Science China Technological Sciences, 2023, 66 (11): 3219- 3230. doi: 10.1007/s11431-022-2345-2
5	BAI X, LI Y H, GUO S Z, et al Recognition for underground voids in C-scans based on GMM-HMM. Journal of Systems Engineering and Electronics, 2025, 36 (1): 82- 94. doi: 10.23919/JSEE.2024.000093
6	ADALOGLOU N, CHATZIS T, PAPASTRAITS I, et al A comprehensive study on deep learning-based methods for sign language recognition. IEEE Trans. on Multimedia, 2022, 24, 1750- 1762. doi: 10.1109/TMM.2021.3070438
7	CHEN J Y, WANG X H, CHEN X Track correlation algorithm based on CNN-LSTM for swarm targets. Journal of Systems Engineering and Electronics, 2024, 35 (2): 417- 429. doi: 10.23919/JSEE.2024.000033
8	XIA Z Y, LUOMEI Y X, ZHOU C L, et al Multidimensional feature representation and learning for robust hand-gesture recognition on commercial millimeter-wave radar. IEEE Trans. on Geoscience and Remote Sensing, 2021, 59 (6): 4749- 4764. doi: 10.1109/TGRS.2020.3010880
9	MA T K, CHEN Z P, WU J X, et al A 77 GHz FMCW MIMO radar system based on 65nm CMOS cascadable 2T3R transceiver. Science China Information Sciences, 2021, 64 (1): 114301. doi: 10.1007/s11432-019-1511-5
10	ZHU Z L, YANG D G, ZHANG J C, et al Dataset of human motion status using IR-UWB through-wall radar. Journal of Systems Engineering and Electronics, 2021, 32 (5): 1083- 1096.
11	DONG X C, FENG Y, CUI C, et al CEEMDAN-ICA-based radar monitoring of adjacent multi-target vital signs. Electronics, 2023, 12 (12): 2732. doi: 10.3390/electronics12122732
12	LONG T, LIANG Z N, LIU Q H Advanced technology of high-resolution radar: target detection, tracking, imaging, and recognition. Science China Information Sciences, 2019, 62, 40301. doi: 10.1007/s11432-018-9811-0
13	GURBUZ S Z, GURBUZ A C, MALAIA E A, et al American sign language recognition using RF sensing. IEEE Sensors Journal, 2021, 21 (3): 3763- 3775.
14	LI B C, YANG J Y, YANG Y, et al Sign language/gesture recognition based on cumulative distribution density features using UWB radar. IEEE Trans. on Instrumentation and Measurement, 2021, 70, 2511113. doi: 10.1109/tim.2021.3092072
15	RAHMAN M M, MDRAFI R, GURBUZ A C, et al. Word-level sign language recognition using linguistic adaptation of 77 GHz FMCW radar data. Proc. of the IEEE Radar Conference, 2021: 1−6.
16	MCCLEARY J, GARCIA L P, ILIOUDIS C, et al. Sign language recognition using micro-Doppler and explainable deep learning. Proc. of the IEEE Radar Conference (RadarConf21), 2021. DOI: 10.1109/RadarConf2147009.2021.9455257.
17	GURBUZ S Z, GURBUZ A C, MALAIA E A, et al. A linguistic perspective on radar micro-Doppler analysis of American sign language. Proc. of the IEEE International Radar Conference, 2020: 232−237.
18	LI Y D, ZHANG D H, CHEN J B, et al Towards domain-independent and real-time gesture recognition using mmWave signal. IEEE Trans. on Mobile Computing, 2023, 22 (12): 7355- 7369. doi: 10.1109/tmc.2022.3207570
19	CHEN M H, GAO H Y, DU Y N, et al Direction finding of bistatic MIMO radar in strong impulse noise. Journal of Systems Engineering and Electronics, 2024, 35 (4): 888- 898.
20	DANG X C, WEI K F, HAO Z J, et al Cross-scene sign language gesture recognition based on frequency-modulated continuous wave radar. Signals, 2022, 3 (4): 875- 894. doi: 10.3390/signals3040052
21	KURTOGLU E, GURBUZ A C, MALAIA E A, et al ASL trigger recognition in mixed activity/signing sequences for RF sensor-based user interfaces. IEEE Trans. on Human-Machine Systems, 2022, 52 (4): 699- 712. doi: 10.1109/THMS.2021.3131675
22	DONG X C, ZHAO Z W, WANG Y P, et al FMCW radar-based hand gesture recognition using spatiotemporal deformable and context-aware convolutional 5-D feature representation. IEEE Trans. on Geoscience and Remote Sensing, 2022, 60, 5107011.
23	JANKIRAMAN M. FMCW radar design. London: Artech House, 2018.
24	ZHAO M M, LI T H, ALSHEIKH M A, et al. Through-wall human pose estimation using radio signals. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 7356−7365.
25	HUANG Y, ZHANG Y J, TAO J, et al Off-grid DOA estimation via a deep learning framework. Science China Information Sciences, 2023, 66 (12): 222305. doi: 10.1007/s11432-022-3750-5
26	WANG Y Z, JIANG Z Y, LI Y D, et al RODNet: a real-time radar object detection network cross-supervised by camera-radar fused object 3D localization. IEEE Journal of Selected Topics in Signal Processing, 2021, 15 (4): 954- 967. doi: 10.1109/JSTSP.2021.3058895
27	SUN K, XIAO B, LIU D, et al. Deep high-resolution representation learning for human pose estimation. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 5693−5703.
28	LEE S P, KINI N P, PENG W H, et al. HuPR: a benchmark for human pose estimation using millimeter wave radar. Proc. of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2023: 5704−5713.
29	KO S K, SON J G, JUNG H. Sign language recognition with recurrent neural network using human keypoint detection. Proc. of the Conference on Research in Adaptive and Convergent Systems, 2018: 326−328.
30	JIANG S Y, SUN B, WANG L C, et al. Skeleton aware multi-modal sign language recognition. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 3408−3418.

Parameter name	Value	Parameter name	Value
Frequency/GHz	77	Bandwidth/GHz	4
Sweep slope/(MHz/μs)	66.578	ADC sampling rate/MHz	6.25
ADC sampling points	256	PRF/kHz	3.1
Number of frames	32	Frame period/ms	82

Key point	Head	Shoulder	Elbow	Wrist	Hip
$ {\mathrm{AP}} $	75.8	67.9	53.1	30.2	90.1

Model	mAP	AP50	AP75	Average OKS
Without Transformer module	57.1	95.5	61.5	70.8
With Transformer module	59.0	94.4	71.6	75.0

Stream	SLR accuracy/%
RA maps	51.94
Bone feature	94.95
Key points feature	95.96
Key points + Bone	96.79

SL	Without Bi-LSTM	With Bi-LSTM
A	96.6	100
B	79.6	100
C	92.7	90.9
D	96	96.4
E	80.6	76.3
F	90.6	94.1
G	62.8	70
H	94.9	100
I	89.7	100
J	88.9	93.5
Average	87.24	92.12

MMF-SLR: a sign language recognition method based on multi-modal feature using millimeter-wave radar

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 17

References 30

Related Articles 15

Recommended Articles

Metrics

Comments

Sign	A	B	C	D	E	Sign	F	G	H	I	J
Key points	100	100	100	100	100	Key points	100	90	100	70	100
Bone	80	100	100	100	100	Bone	100	80	100	100	100
Enseme	100	100	100	100	100	enseme	100	100	100	80	100

Streams	SLR accuracy/%
RA maps	51.94
RD sequence	53.05
m-D signatures	92.06
Key points + Bone	96.79
RA + RD + m-D	78.80
Key points + Bone + RD + m-D	98.99

[1]	Bingdong LIU, Ruihang YU, Zhiming XIONG, Meiping WU. A lightweight pure visual BEV perception method based on dual distillation of spatial-temporal knowledge [J]. Journal of Systems Engineering and Electronics, 2026, 37(1): 36-44.
[2]	Delong WU, Hao FANG, Yiren HAO, Aobo WANG. Outdoor navigation of millimeter-wave radar quadrotors based on optimal virtual tube [J]. Journal of Systems Engineering and Electronics, 2025, 36(4): 1057-1067.
[3]	Jun TANG, Wanting QIN, Qingtao PAN, Songyang LAO. A deep multimodal fusion and multitasking trajectory prediction model for typhoon trajectory prediction to reduce flight scheduling cancellation [J]. Journal of Systems Engineering and Electronics, 2024, 35(3): 666-678.
[4]	Weikun HE, Jingbo SUN, Xinyun ZHANG, Zhenming LIU. Micro-Doppler feature extraction of micro-rotor UAV under the background of low SNR [J]. Journal of Systems Engineering and Electronics, 2022, 33(6): 1127-1139.
[5]	Zijian ZHU, Wenhao XIANG, Ju HUO, Ming YANG, Guiyang ZHANG, Liang WEI. Non-cooperative target pose estimation based on improved iterative closest point algorithm [J]. Journal of Systems Engineering and Electronics, 2022, 33(1): 1-10.
[6]	Yong WANG, Xingyu ZHOU, Xiaofei LU, Yajun LI. An approach of motion compensation and ISAR imaging for micro-motion targets [J]. Journal of Systems Engineering and Electronics, 2021, 32(1): 68-80.
[7]	Wantian WANG, Ziyue TANG, Yichang CHEN, Yongjian SUN. Parity recognition of blade number and manoeuvre intention classification algorithm of rotor target based on micro-Doppler features using CNN [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 884-889.
[8]	Hongzhi LI, Yong WANG. Particle swarm optimization for rigid body reconstruction after micro-Doppler removal in radar analysis [J]. Journal of Systems Engineering and Electronics, 2020, 31(3): 488-499.
[9]	Jingqing Li, Sisan He, Cunqian Feng, and Yizhe Wang. Method for compensating translational motion of rotationally symmetric target based on local symmetry cancellation [J]. Systems Engineering and Electronics, 2017, 28(1): 36-.
[10]	Yuling Liu, Xizhang Wei, Bo Peng, and Yongshun Ling. Dimensions estimation for cone-cylinder target based on sliding-type scatterers analysis [J]. Systems Engineering and Electronics, 2017, 28(1): 10-.
[11]	Yong Wang and Jian Kang. Parameter estimation for rigid body after micro-Doppler removal based on L-statistics in the radar analysis [J]. Systems Engineering and Electronics, 2015, 26(3): 457-467.
[12]	Bin Deng, Yuliang Qin, Hongqiang Wang, and Yanpeng Li. Angular extent effect of micromotion target in SAR image by polar format algorithm [J]. Journal of Systems Engineering and Electronics, 2014, 25(3): 428-433.
[13]	Yuliang Qin, Bin Deng, Zonghui Huang, and Wuge Su. Hybrid micromotion-scattering center model for synthetic aperture radar micromotion target imaging [J]. Journal of Systems Engineering and Electronics, 2013, 24(6): 931-937.
[14]	Wei Wei, Yanning Zhang, and Chunna Tian. Head pose estimation method based on pose manifold and tensor decomposition [J]. Journal of Systems Engineering and Electronics, 2010, 21(5): 907-913.
[15]	Yan Wu, Yuping Wang, Xiaoxiong Liu, and Jimin Ye. Multi-population and diffusion UMDA for dynamic multimodal problems [J]. Journal of Systems Engineering and Electronics, 2010, 21(5): 777-783.