Validation method for simulation models with cross iteration

doi:10.21629/JSEE.2019.03.13

Abstract

Abstract:

Cross iteration often exists in the computational process of the simulation models, especially for control models. There is a credibility defect tracing problem in the validation of models with cross iteration. In order to resolve this problem, after the problem formulation, a validation theorem on the cross iteration is proposed, and the proof of the theorem is given under the cross iteration circumstance. Meanwhile, applying the proposed theorem, the credibility calculation algorithm is provided, and the solvent of the defect tracing is explained. Further, based on the validation theorem on the cross iteration, a validation method for simulation models with the cross iteration is proposed, which is illustrated by a flowchart step by step. Finally, a validation example of a sixdegree of freedom (DOF) flight vehicle model is provided, and the validation process is performed by using the validation method. The result analysis shows that the method is effective to obtain the credibility of the model and accomplish the defect tracing of the validation.

Key words: validation method, simulation model, cross iteration, validation theorem, validation example

Ke FANG, Kaibin ZHAO, Yuchen ZHOU. Validation method for simulation models with cross iteration[J]. Journal of Systems Engineering and Electronics, 2019, 30(3): 555-563.

Figures/Tables 11

Table 1

Fig 1

Fig 2

Table 2

Fig 3

Table 3

Fig 4

Table 4

Fig 5

Table 5

Fig 6

References 32

1	SARGENT R G. Verification and validation of simulation models. Journal of Simulation, 2013, 71 (1): 12- 24.
2	SARGENT R G, GOLDSMAN D M, YAACOUB T. Use of the interval statistical procedure for simulation model validation. Proc. of the Winter Simulation Conference, 2015, 60- 72.
3	GOLDSMAN D, TOKOL G. Output analysis procedures for computer simulations. Proc. of the Winter Simulation Conference, 2000, 39- 45.
4	FANG K, ZHOU Y, ZHAO E. Discussion on the factor space of simulation model validation. Systems Engineering and Electronics, 2017, 39 (11): 2592- 2602.
5	SANKARARAMAN S, MAHADEVAN S. Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems. Reliability Engineering and System Safety, 2015, 138 (1): 194- 209.
6	SARGENT R G, GOLDSMAN D M, YAACOUB T. A tutorial on the operational validation of simulation models. Proc. of the Winter Simulation Conference, 2016, 163- 177.
7	MEHTA U, ROMERO V, EKLUND D, et al. The JANNAF simulation credibility guide on verification, uncertainty propagation and quantification, and validation. Proc. of the 53rd AIAA Aerospace Sciences Meeting, 2015, 1- 30.
8	CHANG K H, CHANG Y C, CHUNG H Y. A novel AHPbased benefit evaluation model of military simulation training systems. Mathematical Problems in Engineering, 2015, 1- 14.
9	YAGER R R. Multi-criteria decision making with interval criteria satisfactions using the golden rule representative value. IEEE Trans. on Fuzzy Systems, 2018, 26 (2): 1023- 1031. doi: 10.1109/TFUZZ.2017.2709275
10	KHELIFI L, MIGNOTTE M. A multi-objective decision making approach for solving the image segmentation fusion problem. IEEE Trans. on Image Processing, 2017, 26 (8): 3831- 3845. doi: 10.1109/TIP.2017.2699481
11	ALMULLA M, YAHYAOUI H, AL-MATORI K. A new fuzzy hybrid technique for ranking real world web services. Knowledge-Based Systems, 2015, 77, 1- 15. doi: 10.1016/j.knosys.2014.12.021
12	FANG K, YANG M, ZHOU Y. A novel method to build a factor space for model validation. Proc. of the Asian Simulation Conference, 2018, 3- 17.
13	DOROBANTU A, BALAS G J, GEORGIOU T T. Validating aircraft models in the gap metric. Journal of Aircraft, 2014, 51 (6): 1665- 1672. doi: 10.2514/1.C032580
14	NING X, WU Y, YU T, et al. Research on comprehensive validation of simulation models based on improved grey relational analysis. Acta Armamentarii, 2016, 37 (2): 338- 347.
15	LEHMANN A, WANG Z. Efficient use of V & V techniques for quality and credibility assurance of complex modeling and simulation (M & S) applications. International Journal of Industrial Engineering:Theory Applications and Practice, 2017, 24 (2): 220- 228.
16	HAKHAMANESHI M, KUTTER B L, MOORE M, et al. Validation of ASCE 41-13 modeling parameters and acceptance criteria for rocking shallow foundations. Earthquake Spectra, 2016, 32 (2): 1121- 1140. doi: 10.1193/121914EQS216M
17	ZHOU Y, FANG K, MA P, et al. Simulation credibility evaluation based on multi-source data fusion. Proc. of the Asian Simulation Conference, 2018, 18- 31. doi: 10.1007/978-981-13-2853-4_2
18	LIU Y, CHEN W, ARENDT P, et al. Toward a better understanding of model validation metrics. Journal of Mechanical Design, 2011, 133 (7): 1- 13.
19	ATKINSON A D, HILL R R, PIGNATIELLO J J, et al. Wavelet ANOVA approach to model validation. Simulation Modelling Practice and Theory, 2017, 78, 18- 27. doi: 10.1016/j.simpat.2017.08.004
20	SUN Y, JEFFREY D H, MARC G G. Nonparametric inference for periodic sequences. Techno-metrics, 2012, 54 (1): 83- 96. doi: 10.1080/00401706.2012.650499
21	AU S K. Model validity and frequency band selection in operational modal analysis. Mechanical Systems and Signal Processing, 2016, 81, 339- 359. doi: 10.1016/j.ymssp.2016.03.025
22	ZHANG Z, FANG K, WU F, et al. Detection method for credibility defect of simulation based on Sobol' method and orthogonal design. Proc. of the Asia Simulation Conference, 2013, 421- 428.
23	SARIN H, KOKKOLARAS M, HULBERT G, et al. Comparing time histories for validation of simulation models:error measures and metrics. Journal of Dynamic Systems, Measurement, and Control, 2010, 132 (6): 768- 778.
24	CROCHEMORE L, PERRIN C, ANDREASSIAN V, et al. Comparing expert judgement and numerical criteria for hydrograph evaluation. Hydrological Sciences Journal, 2015, 60 (3): 402- 423. doi: 10.1080/02626667.2014.903331
25	HAUDUC H, NEUMANN M B, MUSCHALLA D, et al. Efficiency criteria for environmental model quality assessment:a review and its application to wastewater treatment. Environmental Modeling & Software, 2015, 68, 196- 204.
26	AO D, HU Z, MAHADEVAN S. Design of validation experiments for life prediction models. Reliability Engineering and System Safety, 2017, 165, 22- 33. doi: 10.1016/j.ress.2017.03.030
27	SHIN J, YOO J, PARK P. Adaptive regularisation for normalised subband adaptive filter:mean-square performance analysis approach. IET Signal Processing, 2018, 12 (9): 1146- 1153. doi: 10.1049/iet-spr.2018.5165
28	ROY K, DAS R N, AMBURE P, et al. Be aware of error measures. Further studies on validation of predictive QSAR models. Chemo-metrics and Intelligent Laboratory Systems, 2016, 152, 18- 33. doi: 10.1016/j.chemolab.2016.01.008
29	DABBI E P, HAIGH I D, LAMBKIN D, et al. Beyond significant wave height:a new approach for validating spectral wave models. Coastal Engineering, 2015, 100, 11- 25. doi: 10.1016/j.coastaleng.2015.03.007
30	NOTTON G, PAOLI C, IVANOVA L, et al. Neural network approach to estimate 10-min solar global irradiation values on tilted planes. Renewable Energy, 2013, 50, 576- 584. doi: 10.1016/j.renene.2012.07.035
31	PARISI A V, DOWNS N, TURNER J, et al. Comparison of gome-2 UVA satellite data to ground-based spectroradiometer measurements at a subtropical site. IEEE Trans. on Geoscience and Remote Sensing, 2017, 55 (6): 3145- 3149. doi: 10.1109/TGRS.2017.2662714
32	MERLA C, PAFFI A, APOLLONIO F, et al. Portable system for practical permittivity measurements improved by homomorphic deconvolution. IEEE Trans. on Instrumentation and Measurement, 2017, 66 (3): 514- 521. doi: 10.1109/TIM.2016.2644859

Variant	Meaning
$x, y, z$	Position
$\psi _v $	Trajectory deflection angle
$q$	Dynamic pressure
$R_{0x} , R_{0y} , R_{0z} $	Radius
$fM$	Earth gravitational constant
$A_w $	Wind direction
$\delta _\varphi , \delta _\psi , \delta _\gamma $	Rudder angle
$\varphi , \psi , \gamma $	Attitude angles
$m$	Mass
$S_M $	Sectional area
$A_0 $	Launch direction
$R_a $	Earth equator radius
$\omega _{x_1 } , \omega _{y_1 } , \omega_{z_1 } $	Rotational angular velocity
${d}''{}_{3x}^\gamma, d''{}_{3z}^\gamma $	Moment
${b}''{}_{3x}^\psi, b''{}_{3z}^\psi $	coefficient
${b}''{}_{3x}^\varphi, b''{}_{3z}^\varphi $	of inertia
$V, V_x, $$V_y , V_z $	Velocity
$g_x , g_y , g_z$	Gravity acceleration
$P_x , P_y , P_z $	Thrust
$B_0 $	Launch spot latitude
$M_a $	Mach
$J_{x_1 } , J_{y_1 } , J_{z_1} $	Rotary inertia
$\theta $	Trajectory tilt angle
$C_x, C_y^\alpha , C_z^\beta $	Aerodynamic coefficient
$\alpha +\alpha _w $$\beta +\beta _w $	Velocity transformation angle
$J$	Gravitational potential coefficient
$V_w $	Wind velocity
$M_{x_1 } , M_{y_1 } , M_{z_1 } $	Rotating moment

Time/s	Sim $x$/m	Obs $x'$/m	Sim $y$/m	Obs $y'$/m	Sim $z$/m	Obs $z'$/m
0.100	150.757	150.755	20 799.960	20 799.960	0	0
0.200	294.326	294.315	20 799.850	20 799.850	0	0
$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$
25.100	34 685.460	34 323.240	16 460.620	16 592.120	118.865	114.550
25.200	34 814.140	34 448.610	16 422.750	16 555.970	119.754	115.399
$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$
49.800	62 228.430	60 814.420	1 368.693	2 639.631	403.060	392.137
49.900	62 315.850	60 897.090	1 287.806	2 567.734	403.434	392.968
50.000	62 402.990	60 979.460	1 206.855	2 495.826	403.761	393.779

Time/s	Simulation data $\varphi $/($^\circ$)	Observed data ${\varphi }'$/($^\circ$)	Simulation data $\psi $/($^\circ$)	Observed data ${\psi }'$/($^\circ$)	Simulation data $\gamma$/($^\circ$)	Observed data ${\gamma }'$/($^\circ$)
1.000	-3.125	-3.125	-0.363	-0.363	-6.201	-6.207
2.000	-5.449	-5.439	-0.567	-0.567	-5.072	-5.087
$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$
26.000	-20.101	-19.837	-0.455	-0.463	-1.568	-1.598
27.000	-20.536	-20.255	-0.46	-0.468	-1.543	-1.574
$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$
49.000	-34.601	-33.213	-0.009	-0.256	6.753	3.475
50.000	-35.459	-33.999	0.271	-0.127	10.814	5.431

Time/s	Simulation V_w/(m/s)	Observed V_w' /(m/s)
0.100	13	13
0.200	13	13
$\vdots$	$\vdots$	$\vdots$
25.100	16.37	15.564
25.200	16.601	15.785
$\vdots$	$\vdots$	$\vdots$
49.900	18.187	22.014
50.000	17.987	21.748

Time/s	Simulation data$\delta _\varphi $/($^\circ$)	Observed data${\delta }'_\varphi $/($^\circ$)	Simulation data$\delta _\psi $/($^\circ$)	Observed data${\delta }'_\psi$/($^\circ$)	Simulation data$\delta _\gamma $/($^\circ$)	Observed data${\delta }'_\gamma $/($^\circ$)
1.000	-0.22	-0.216	1.824	1.823	-0.007	0.001
2.000	0.039	0.042	5.306	5.298	-0.017	-0.018
$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$
26.000	-0.169	-0.147	8.064	8.461	-0.046	-0.042
27.000	-0.15	-0.13	7.561	7.966	-0.041	-0.038
$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$	$\vdots$
49.000	-0.011	-0.008	1.345	1.835	-0.012	-0.01
50.000	-0.014	-0.008	1.438	1.937	-0.017	-0.012