Effectiveness validation of evaluation index systems of equipment SoS operational capability: a framework and grey clustering model

doi:10.23919/JSEE.2026.000034

Abstract

Abstract:

A reliable evaluation index system (EIS) is essential for assessing the operational capabilities of equipment system-of-systems (SoS), yet systematic methods for validating its effectiveness in complex scenarios remain lacking. To address this problem, this paper proposes a validation algorithm, which includes a validation framework, a data collection format tailored to operational contexts, and a grey clustering algorithm. First, the validation framework is designed at three levels, syntactic, semantic, and pragmatic, to systematically assess the normativity, logical coherence, and practical utility of the EIS. Second, a data collection format suited to operational backgrounds is introduced, enabling multi-domain coverage across the physical, informational, cognitive, and social dimensions, and accounting for both expected and abnormal scenarios. Third, a grey clustering algorithm is developed, in which a clustering weight calculation model based on the interval fuzzy preference relation matrix ensures consistency among indicators, while rough and fine clustering analyses are combined to achieve quantitative validation of EIS effectiveness. Case studies demonstrate that the proposed method effectively validates EIS performance under diverse operational conditions. Overall, this work provides a systematic basis for EIS validation and practical guidance for enhancing SoS evaluation.

Key words: system-of-systems (SoS), equipment operational capability, effectiveness of evaluation index system (EIS), grey clustering model

Zhaojie Ke, Wenya Zhou, Shujun Tan, Yanchang Huang, Junchao Shang, Xiaoting Fang. Effectiveness validation of evaluation index systems of equipment SoS operational capability: a framework and grey clustering model[J]. Journal of Systems Engineering and Electronics, 2026, 37(5): 1587-1605.

Figures/Tables 10

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Table 1

Grey class and characteristic features of operational scenarios"

Grey class level	Name	Grey class characteristic feature
Grey class level	Name	Physical domain	Informational domain	Cognitive domain	Social domain
1	Highly suitable level	The SoS adapts to the task scenarios of any physical domains, obtaining sufficient data on operational effectiveness, task efficiency, and operational time outcomes.	No barriers to the exchange, sharing, recognition, collaboration and utilization of information.	The situation quality, the understanding of operational intent, and the synchronization of operational activities adapt to any task scenario.	No barriers to the information interaction, the formation of situational awareness, the cognitive sharing, and the collaborative decision-making.
2	Relatively suitable level	The SoS adapts to the task scenarios of multiple physical domains, obtaining big amount of data on operational effectiveness, task efficiency, and operational time outcomes.	The exchange, sharing, cognition, collaboration, and utilization of information adapt to multi-domain task scenarios.	The situation quality, the understanding of operational intent, and the synchronization of operational activities are achieved in multiple task scenarios.	The information interaction, the formation of situational awareness, the cognitive sharing, and the collaborative decision-making adapt to multiple task scenarios.
3	Generally suitable level	The SoS adapts to the task scenarios of a few physical domains, obtaining basic data on operational effectiveness, task efficiency, and operational time outcomes.	The exchange, sharing, cognition, collaboration, and utilization of information adapt to a few task scenarios.	The situation quality, the understanding of operational intent, and the synchronization of operational activities are achieved in a few task scenarios.	The information interaction, the formation of situational awareness, the cognitive sharing, and the collaborative decision-making adapt to a few task scenarios.
4	Barely suitable level	The SoS adapts to individual operational activities, obtaining minimal data on operational effectiveness, task efficiency, and operational time outcomes.	The exchange, sharing, cognition, collaboration, and utilization of information adapt to individual activity scenarios.	The situation quality, the understanding of operational intent, and synchronization of operational activities are achieved in individual operational activities.	The information interaction, the formation of situational awareness, the cognitive sharing, and the collaborative decision-making adapt to individual operational scenarios.
5	Unsuitable level	The SoS fails to adapt to any task’s scenarios or operational activities, making it challenging to obtain data on operational effectiveness, task efficiency, or operational time outcomes.	Incapable of the exchange, sharing, cognition, collaboration, or utilization of information in any scenario.	Unable to achieve the understanding of operational intent, the synchronization of operational activities, or related quality in any task or activity.	Incapable of achieving the information interaction, the formation of situational awareness, the cognitive sharing, and the collaborative decision-making in any task or activity.

Table 1

Fig 6

Fig 7

Fig 8

Fig 9

References 42

1	Wang X Y, Yang Z, Chai S Y, et al Integrated threat assessment method of beyond-visual-range air combat. Journal of Systems Engineering and Electronics, 2025, 36 (1): 176. doi: 10.23919/JSEE.2025.000011
2	Hu J W, Wang Z H, Gao Y, et al Uncertainty entropy-based exploratory evaluation method and its applications on missile effectiveness evaluation. Journal of Systems Engineering and Electronics, 2023, 34 (6): 1602. doi: 10.23919/JSEE.2023.000063
3	Yang K W, Yang Z W, Tan Y J, et al A review of equipment system evaluation methods focused on system contribution rate. Systems Engineering and Electronics, 2019, 41 (2): 311.
4	Wang X H, Zhang Y, Wang L Z, et al Robustness evaluation method for unmanned aerial vehicle swarms based on complex network theory. Chinese Journal of Aeronautics, 2020, 33 (1): 352. doi: 10.1016/j.cja.2019.04.025
5	Gao Y, Liu H, Tian Y L Inverse design of mission success space for combat aircraft contribution evaluation. Chinese Journal of Aeronautics, 2020, 33 (8): 2189. doi: 10.1016/j.cja.2020.03.039
6	Dind J F, Si G Y, Yang J Y, et al Research and analysis on the method for building the indicator system for system combat effectiveness assessment. Journal of Command and Control, 2016, 2 (3): 239.
7	Ha J X, Wang J S Method for constructing the indicator system for cyber combat command effectiveness evaluation based on social network analysis. Journal of Academy of Force Engineering, 2016, 30 (6): 1.
8	Zhao Z Q. Research on the method for constructing the indicator system for equipment combat test effectiveness evaluation[C]// 20th China Conference on System Simulation Technology and its Application, 2019: 345. (in Chinese)
9	Cao Z M, Zhou Y F Method for constructing the intelligent combat effectiveness evaluation indicator system. Command Control & Simulation, 2024, 46 (6): 137.
10	Ning X Y, Wang Y, Wang Z D, et al Link-16 anti-jamming performance evaluation based on grey relational analysis and cloud model. Journal of Systems Engineering and Electronics, 2025, 36 (1): 62. doi: 10.23919/JSEE.2023.000120
11	Han C, Xiong W, Xiong M H, et al Support vector regression-based operational effectiveness evaluation approach to reconnaissance satellite system. Journal of Systems Engineering and Electronics, 2023, 34 (6): 1626. doi: 10.23919/JSEE.2023.000020
12	Zhao D L, Tan Y J, Li J C, et al Weapon equipment system contribution evaluation based on combat circumstances. Systems Engineering and Electronics, 2017, 39 (10): 2239.
13	Ma J, Ge Y, Pu H Y A review of attribute reduction methods. Data Analysis and Knowledge Discovery, 2020, 4 (1): 40.
14	Yu L W, Guo W Y, Huang J N, et al A review of algorithms for comprehensive evaluation index reduction. Mechanical Engineer, 2023, 9, 12.
15	Fu Y, Liu Y J RST evaluation method and application of the effectiveness of indicator systems. Management Review, 2009, 21 (7): 91.
16	Li C, Zhang W D Research on the copyright risk assessment indicator system and its effectiveness: based on the perspective of network content service providers. Information Theory and Practice, 2014, 37 (7): 48.
17	Xu M M, Zhang L J, Zhang X. Good standards and measurement methods of comprehensive evaluation index system[J]. Statistics and Decision, 2015(3): 18. (in Chinese)
18	Ma Z Q, Hong T. The study on reliability and validity of index system of risk evaluation on technological [C]. 1st International Workshop on Database Technology and Applications, 2009: 662.
19	Wang F, Zhang Q H, Jin J Q Research on reliability and validity of indicator system for community health services performance[J]. Chinese General Practice, 2008, 11 (1): 77.
20	Fan W Y, Zhao X H, Wu J C, et al Study on monitoring and early warning index system for schistosomiasis in Poyang Lake eco-economic region. Chinese Journal of Disease Control & Prevention, 2019, 23 (4): 421.
21	Yu Y, Tao L J, Yang T F Empirical study on the equalization of basic public health services index system. Journal of Central South University (Medical Edition), 2015, 40 (4): 421.
22	Zhao S D, Li S H Research on system evaluation of top management-team based on the method of analytical hierarchy process and rough set. Applied Mechanics and Materials, 2011, 63164, 322.
23	Liu S F. Grey clustering evaluation models. Liu S, ed. Grey systems analysis: methods, models and applications[M]. Singapore: Springer Nature Singapore, 2025: 155.
24	Wang J H, Wang L, Zhang L, et al Supplier selection VIKOR multi-attribute decision making of grey group clustering and improved CRITIC weighting. Systems Engineering and Electronics, 2023, 45 (1): 155.
25	Xie N M, Su B T, Chen N L Construction mechanism of whitenization weight function and its application in grey clustering evaluation. Journal of Systems Engineering and Electronics, 2019, 30 (1): 121. doi: 10.21629/JSEE.2019.01.12
26	Dai J, Yan Y, He Y H Grey incidence clustering method based on multidimensional dynamic time warping distance. Journal of Systems Engineering and Electronics, 2017, 28 (5): 946. doi: 10.21629/JSEE.2017.05.12
27	Deng J L. Grey control systems[M]. Wuhan: Press of Huazhong University of Science and Technology, 1985. (in Chinese)
28	Liu S F, Zhu Y D Models based on criteria’s triangular membership functions for the evaluation of regional economics. Transactions of the Chinese Agricultural Engineering, 1993, 9 (2): 8.
29	Liu S F, Xie N M A new grey evaluation method based on reformative triangular whitenization weight function. Journal of Systems Engineering, 2011, 26 (2): 244.
30	Liu S F, Yang Y J, Fang Z G, et al Grey cluster evaluation models based on mixed triangular whitenization weight functions. Grey Systems: Theory and Application, 2015, 5 (3): 410. doi: 10.1108/GS-11-2014-0050
31	Liu S F, Zhang S G, Jian L R, et al Performance evaluation of large commercial aircraft vendors. Journal of Grey System, 2015, 27, 1.
32	Chen S J, Hwang C L. Fuzzy multiple attribute decision making: methods and applications[M]. Berlin: Springer Berlin Heidelberg, 1992.
33	Ishibuchi H, Tanaka H Multiobjective programming in optimization of the interval objective function. European Journal of Operational Research, 1990, 48 (2): 219. doi: 10.1016/0377-2217(90)90375-L
34	Li C C, Dong Y C, Xu Y J, et al An overview on managing additive consistency of reciprocal preference relations for consistency-driven decision making and fusion: taxonomy and future directions. Information Fusion, 2019, 52, 143. doi: 10.1016/j.inffus.2018.12.004
35	Tanino T Fuzzy preference orderings in group decision making. Fuzzy Sets and Systems, 1984, 12 (2): 117. doi: 10.1016/0165-0114(84)90032-0
36	Morris C H. Foundation of the theory of signs[M]. Chicago: The University of Chicago Press, 1938.
37	Zhang H J, Wei Z X, Ju H S, et al. Principles and engineering methods of weapon equipment system[M]. Beijing: Publishing House of Electronics Industry, 2019. (in Chinese)
38	Xu Z Consistency of interval fuzzy preference relations in group decision making. Applied Soft Computing, 2011, 11 (5): 3898. doi: 10.1016/j.asoc.2011.01.019
39	Wang Y M, Yang J B, Xu D L Interval weight generation approaches based on consistency test and interval comparison matrices. Applied Mathematics and Computation, 2005, 167 (1): 252. doi: 10.1016/j.amc.2004.06.080
40	Xu Y, Li K W, Wang H Consistency test and weight generation for additive interval fuzzy preference relations. Soft Computing, 2014, 18 (8): 1499. doi: 10.1007/s00500-013-1156-x
41	Lv S, Lin J, Xu Z, et al The consistency discretization adjustment method of interval fuzzy preference relationship based on individual agreement and group consensus. Computers & Industrial Engineering, 2024, 194, 110392.
42	Xu Z, Chen J Some models for deriving the priority weights from interval fuzzy preference relations. European Journal of Operational Research, 2008, 184 (1): 266. doi: 10.1016/j.ejor.2006.11.011

[1]	Xichao SU, Fang GUO, Jingyu CONG, Yang ZHANG, Zhongzheng ZHAO, Wei HAN, Xinwei WANG. An evaluation method for contribution rate of UAVs to amphibious joint landing system of systems [J]. Journal of Systems Engineering and Electronics, 2025, 36(6): 1613-1628.
[2]	Ruiwen ZHANG, Bifeng SONG, Yang PEI, Qijia YUN. Improved method for subsystems performance trade-off in system-of-systems oriented design of UAV swarms [J]. Journal of Systems Engineering and Electronics, 2019, 30(4): 720-737.