GPR-based model validation method for small samples

doi:10.23919/JSEE.2026.000113

Abstract

Abstract:

Validation for simulation models often confronts challenges with small samples due to the costs of time and money. To address this issue, this paper presents a validation method for small-sample dynamic outputs based on Gaussian process regression (GPR) models. Firstly, a validation framework based on Bayes statistics is proposed, shifting the focus from merely analyzing validation data to a more comprehensive analysis of posterior distributions. Subsequently, the posterior distributions of both the simulation outputs and the reference data are separately captured through segmented GPR. Then, the consistency of these posterior distributions is evaluated in terms of the central tendency and the distribution range. This consistency serves as a quantitative measure of the simulation model’s credibility, expressed as a value ranging from 0 to 1, where a value closer to 1 indicates higher credibility. Finally, the effectiveness of this validation method is demonstrated through a numerical example and an application example, highlighting its capability in uncertainty description and adaptability to small samples.

Key words: model validation, small samples, Gaussian process regression (GPR), Bayes statistics

Fan YANG, Ping MA, Huan ZHANG, Wei LI, Ming YANG. GPR-based model validation method for small samples[J]. Journal of Systems Engineering and Electronics, 2026, 37(3): 867-877.

Figures/Tables 11

Fig 1

Fig 2

Fig 3

Fig 4

Table 1

Fig 5

Fig 6

Table 2

Table 3

Table 4

Fig 7

References 31

1	STEVENSON D E. Verification and validation of complex systems. Proc. of the Intelligent Engineering Systems through Artificial Neural Networks, 2002: 159−164.
2	WANG Y N, LI J Q, SUN H B, et al A survey on VV&A of large-scale simulations. International Journal of Crowd Science, 2019, 3 (1): 63- 86. doi: 10.1108/IJCS-01-2019-0004
3	SARGENT R G Verification and validation of simulation models. Journal of Simulation, 2013, 7 (1): 12- 24. doi: 10.1057/jos.2012.20
4	DURST P J, ANDERSON D T, BETHEL C L A historical review of the development of verification and validation theories for simulation models. International Journal of Modeling, Simulation, and Scientific Computing, 2017, 8 (2): 1730001. doi: 10.1142/S1793962317300011
5	XI Z M, YANG R J Reliability analysis with model uncertainty coupling with parameter and experiment uncertainties: a case study of 2014 V&V challenge problem. Journal of Verification, Validation and Uncertainty Quantification, 2015, 1 (1): 011005.
6	SHANKAR S, SANKARAN M Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems. Reliability Engineering and System Safety, 2015, 138, 194- 209. doi: 10.1016/j.ress.2015.01.023
7	LIN S L, LI W, MA P, et al Structural modelling and bayesian inference for model validation and confidence extrapolation. Journal of Statistical Computation and Simulation, 2020, 90 (2): 211- 233. doi: 10.1080/00949655.2019.1675662
8	PAN Y L, HE Y Z Trustworthiness assessment method for guidance simulation system based on DS/AHP and gray cloud clustering. Electronic Measurement Technology, 2017, 40 (7): 43- 47.
9	YANG M, QIAN X C, LI W Simulation dynamic output validation method based on data feature. Systems Engineering and Electronic Technology, 2016, 38 (2): 457- 463.
10	JIA J P, HE X Q, JIN Y J. Statistics. Beijing: Renmin University Press, 2015.(in Chinese)
11	SONG T. Research on model validation method for small sample and service-oriented tools. Harbin: Harbin Institute of Technology, 2018. (in Chinese)
12	WANG J M, WU Y J Credibility assessment of small sample data based on clustered cloud model. Journal of System Simulation, 2019, 31 (7): 1263- 1271.
13	NIE K, LUAN R P A simulation model validation method based on data enhancement. Command Control and Simulation, 2019, 41 (3): 92- 96.
14	LI W, ZHOU Y C, LIN S L, et al A review of simulation model validation methods. Journal of System Simulation, 2019, 31 (7): 1249- 1256.
15	ANALLA M Model validation through the linear regression fit to actual versus predicted values. Agricultural Systems, 1998, 57 (1): 115- 119. doi: 10.1016/S0308-521X(97)00073-5
16	MAHARAJ E A Cluster of time series. Journal of Classification, 2000, 17 (2): 297- 314.
17	MAHARAJ E A Comparison and classification of stationary multivariate time series. Pattern Recognition, 1999, 32 (7): 1129- 1138.
18	RAMONI M, SEBASTIANI P, COHEN P Bayesian clustering by dynamics. Machine Learning, 2002, 47 (1): 91- 121. doi: 10.1023/A:1013635829250
19	BICEGO M, MURINO V, FIGUEIREDO M A T. Similarity-based clustering of sequences using hidden Markov models. Proc. of the International Conference on Machine Learning and Data Mining in Pattern Recognition, 2003: 86−95.
20	SEEGER M Gaussian processes for machine learning. International Journal of Neural Systems, 2004, 14 (2): 69- 106. doi: 10.1142/S0129065704001899
21	ROBERTS S, OSBORNE M, EBDEN M, et al Gaussian processes for time-series modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2013, 371 (1984): 20110550. doi: 10.1098/rsta.2011.0550
22	AIGRAIN S, FOREMAN-MACKEY D Gaussian process regression for astronomical time series. Annual Review of Astronomy and Astrophysics, 2023, 61 (1): 329- 371. doi: 10.1146/annurev-astro-052920-103508
23	PALAR P S, PARUSSINI L, BREGANT L, et al On kernel functions for bi-fidelity Gaussian process regressions. Structural and Multidisciplinary Optimization, 2023, 66 (2): 1- 22. doi: 10.1007/s00158-023-03487-y
24	LI S. Research on the validation method of missile system simulation model. Changsha: National University of Defense Technology, 2003. (in Chinese)
25	MANFREDI P Conservative Gaussian process models for uncertainty quantification and Bayesian optimization in signal integrity applications. IEEE Trans. on Components Packaging and Manufacturing Technology, 2024, 14 (7): 1261- 1272. doi: 10.1109/TCPMT.2024.3390402
26	KARVONEN T, WYNNE G, TRONARP F, et al Maximum likelihood estimation and uncertainty quantification for gaussian process approximation of deterministic functions. SIAM/ASA Journal on Uncertainty Quantification, 2020, 8 (3): 926- 958. doi: 10.1137/20M1315968
27	PARK I, AMARCHINTA H K, GRANDHI R V A Bayesian approach for quantification of model uncertainty. Reliability Engineering and System Safety, 2010, 95 (7): 777- 785. doi: 10.1016/j.ress.2010.02.015
28	REBBA R, MAHADEVAN S, HUANG S Validation and error estimation of computational models. Reliability Engineering and System Safety, 2006, 91 (10): 1390- 1397.
29	SMITH J Statistical modeling principles. Journal of Statistical Methods, 2023, 45 (2): 234- 256.
30	ANDERSON R, BROWN T Gaussian process regression for dynamic systems. Computational Bayesian Models, 2023, 29 (3): 98- 121.
31	LIU H, ONG Y S, SHEN X, et al When Gaussian process meets big data: a review of scalable GPS. IEEE Trans. on Neural Networks and Learning Systems, 2020, 31 (11): 4405- 4423. doi: 10.1109/TNNLS.2019.2957109

Strain error/%	Sample size	Data type	Data label
1	10	Reference	Data_R
1	10	Simulation	Data_S1
3	10	Simulation	Data_S2

Simulation output	Result	The 1st segment	The 2nd segment	$ C({\boldsymbol{Y}}_{S},{\boldsymbol{Y}}_{R}) $
Data_S1	$ {C}_{mx} $	0.997 9	0.998 0	0.978 2
	$ {C}_{{\mathrm{PI}}x} $	0.933 4	0.930 7
	$ C({\boldsymbol{Y}}_{Sx},{\boldsymbol{Y}}_{Rx}) $	0.978 6	0.977 8
Data_S2	$ {C}_{mx} $	0.994 1	0.994 9	0.832 7
	$ {C}_{{\mathrm{PI}}x} $	0.439 0	0.471 4
	$ C({\boldsymbol{Y}}_{Sx},{\boldsymbol{Y}}_{Rx}) $	0.827 5	0.837 9

Window size	Result	The 1st segment	The 2nd segment	The 3rd segment			The 4th segment	The 5th segment	$ C({\boldsymbol{Y}}_{S},{\boldsymbol{Y}}_{R}) $	Time/s
20	$ {C}_{mx} $	0.996 4	0.995 5	0.994 4			0.998 3	0.996 3	0.833 5	7.02
	$ {C}_{PIx} $	0.445 3	0.481 6	0.411 8			0.489 9	0.440 3
	$ C({\boldsymbol{Y}}_{Sx},{\boldsymbol{Y}}_{Rx}) $	0.831 1	0.841 3	0.819 6			0.845 8	0.829 5

Window size	Result	The 1st segment				The 2nd segment			$ C({\boldsymbol{Y}}_{S},{\boldsymbol{Y}}_{R}) $	Time/s
50	$ {C}_{mx} $	0.994 1			0.994 9				0.832 7	8.65
	$ {C}_{PIx} $	0.439 0			0.471 4
	$ C({\boldsymbol{Y}}_{Sx},{\boldsymbol{Y}}_{Rx}) $	0.827 5			0.837 9

Index	Reference data	Simulation output	Validation result
1	5	5	0.837 6
2	10	10	0.833 5
3	20	20	0.836 6
4	30	30	0.838 4
5	5	10	0.838 4
6	5	20	0.840 0
7	5	30	0.840 6

[1]	Jingru ZHANG, Zhigeng FANG, Feng YE, Ding CHEN. Key indexes identifying approach of weapon equipment system-of-systems effectiveness integrating Bayes method and dynamic grey incidence analysis model [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1482-1490.
[2]	Xu LYU, Baiqing HU, Yongbin DAI, Mingfang SUN, Yi LIU, Duanyang GAO. Gaussian process regression-based quaternion unscented Kalman robust filter for integrated SINS/GNSS [J]. Journal of Systems Engineering and Electronics, 2022, 33(5): 1079-1088.
[3]	Yunxia Chen, Zhiguo Zeng, and Rui Kang. Validation methodology for distribution-based degradation model [J]. Journal of Systems Engineering and Electronics, 2012, 23(4): 553-559.
[4]	Ke Hongfa, Chen Yongguang & Liu Yi. Data processing of small samples based on grey distance information approach [J]. Journal of Systems Engineering and Electronics, 2007, 18(2): 281-289.