Classification of aviation incident causes using LGBM with improved cross-validation

doi:10.23919/JSEE.2024.000035

Journal of Systems Engineering and Electronics ›› 2024, Vol. 35 ›› Issue (2): 396-405.doi: 10.23919/JSEE.2024.000035

• SYSTEMS ENGINEERING • Previous Articles

Classification of aviation incident causes using LGBM with improved cross-validation

Xiaomei NI¹^,²(), Huawei WANG¹^,*(), Lingzi CHEN¹(), Ruiguan LIN¹()

¹ School of Civil Aviation, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
² School of Aeronautical Engineering, Nanjing Vocational University of Industry Technology, Nanjing 210023, China

Received:2021-04-28 Online:2024-04-18 Published:2024-04-18
Contact: Huawei WANG E-mail:905271004@qq.com;wang_hw66@163.com;13258035636@163.com;478636604@qq.com
About author:
NI Xiaomei was born in 1992. She received her Ph.D. degree in aviation engineering from Nanjing University of Aeronautics and Astronautics, Nanjing, China, in 2022. She is a lecturer in School of Aeronautical Engineering, Nanjing Vocational University of Industry Technology, Nanjing, China. Her research interests are civil aviation safety engineering and aircraft reliability. E-mail: 905271004@qq.com

WANG Huawei was born in 1974. She received her Ph.D. degree from National University of Defense Technology, Changsha, China in 2003. She is a professor of the School of Civil Aviation, Nanjing University of Aeronautics and Astronautics, Nanjing, China. Her research interests include prognostic and health management, civil aviation safety engineering, and aircraft reliability engineering. E-mail: wang_hw66@163.com

CHEN Lingzi was born in 1997. She received her M.S. degree from Nanjing University of Aeronautics and Astronautics in 2022. She is an engineer of the 41st Research Institute of China Electronics Technology Group Corporation. Her research interest is civil aviation safety engineering. E-mail: 13258035636@163.com

LIN Ruiguan was born in 1993. He received his M.S. degree in aviation engineering from Shenyang Aerospace University, Shenyang, China, in 2019. He is currently pursuing his Ph.D. degree at the School of Civil Aviation, Nanjing University of Aeronautics and Astronautics, Nanjing, China. His research interests are civil aviation safety engineering and aircraft reliability. E-mail: 478636604@qq.com
Supported by:
This work was supported by the National Natural Science Foundation of China Civil Aviation Joint Fund (U1833110), and Research on the Dual Prevention Mechanism and Intelligent Management Technology for Civil Aviation Safety Risks (YK23-03-05).

Abstract

Abstract:

Aviation accidents are currently one of the leading causes of significant injuries and deaths worldwide. This entices researchers to investigate aircraft safety using data analysis approaches based on an advanced machine learning algorithm. To assess aviation safety and identify the causes of incidents, a classification model with light gradient boosting machine (LGBM) based on the aviation safety reporting system (ASRS) has been developed. It is improved by k-fold cross-validation with hybrid sampling model (HSCV), which may boost classification performance and maintain data balance. The results show that employing the LGBM-HSCV model can significantly improve accuracy while alleviating data imbalance. Vertical comparison with other cross-validation (CV) methods and lateral comparison with different fold times comprise the comparative approach. Aside from the comparison, two further CV approaches based on the improved method in this study are discussed: one with a different sampling and folding order, and the other with more CV. According to the assessment indices with different methods, the LGBM-HSCV model proposed here is effective at detecting incident causes. The improved model for imbalanced data categorization proposed may serve as a point of reference for similar data processing, and the model’s accurate identification of civil aviation incident causes can assist to improve civil aviation safety.

Key words: aviation safety, imbalance data, light gradient boosting machine (LGBM), cross-validation (CV)

Xiaomei NI, Huawei WANG, Lingzi CHEN, Ruiguan LIN. Classification of aviation incident causes using LGBM with improved cross-validation[J]. Journal of Systems Engineering and Electronics, 2024, 35(2): 396-405.

Figures/Tables 14

Table 1

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Table 2

Fig 6

Table 3

Fig 7

Fig 8

Fig 9

Fig 10

Table 4

References 30

1	INSUA D R, ALFARO C, GOMEZ J, et al Forecasting and assessing consequences of aviation safety occurrences. Safety Science, 2019, 111, 243- 252. doi: 10.1016/j.ssci.2018.07.018
2	PRAMONO A, MIDDLETON J H, CAPONECCHIA C Civil aviation occurrences in Indonesia. Journal of Advanced Transportation, 2020, 2020 (4): 3240764.
3	TILLER L N, BLISS J P Categorization of near-collision close calls reported to the aviation safety reporting system. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2017, 61 (1): 1866- 1870. doi: 10.1177/1541931213601947
4	ZHANG X G, MAHADEVAN S Ensemble machine learning models for aviation incident risk prediction. Decision Support Systems, 2019, 116, 48- 63. doi: 10.1016/j.dss.2018.10.009
5	KUHN K D Using structural topic modeling to identify latent topics and trends in aviation incident reports. Transportation Research Part C: Emerging Technologies, 2018, 87, 105- 122. doi: 10.1016/j.trc.2017.12.018
6	REN B, YUE Z F, SI Y, et al. Novel method of aviation safety causality prediction based on forecast. Systems Engineering and Electronics, 2023, 45(3): 762−768. (in Chinese)
7	Aviation Safety Reporting System. Program briefing. https:asrs.arc.nasa.gov/docs/ASRS_ProgramBriefing.pdf.
8	CHUNG S H, MA H L, HANSEN M, et al Data science and analytics in aviation. Transportation Research Part E: Logistics and Transportation Review, 2020, 134, 101837. doi: 10.1016/j.tre.2020.101837
9	HU X, WU J, HE J R. Textual indicator extraction from aviation accident reports. Proc. of the AIAA Aviation 2019 Forum, 2019: 2939.
10	SCHLOGL M, STUTZ R, LAAHA G, et al A comparison of statistical learning methods for deriving determining factors of accident occurrence from an imbalanced high resolution dataset. Accident Analysis & Prevention, 2019, 127, 134- 149.
11	JEONG H, JANG Y, BOWMAN P J, et al Classification of motor vehicle crash injury severity: a hybrid approach for imbalanced data. Accident Analysis & Prevention, 2018, 120, 250- 261.
12	CHRISTENSEN I The elements of a commercial human spaceflight safety reporting system. Acta Astronautica, 2017, 139, 228- 232. doi: 10.1016/j.actaastro.2017.07.012
13	ZHANG C H, KONG L, XU Q, et al Fault diagnosis of key components in the rotating machinery based on Fourier transform multi-filter decomposition and optimized LGBM. Measurement Science and Technology, 2020, 32 (1): 015004.
14	PARSA A B, MOVAHEDI A, TAGHIPOUR H, et al Toward safer highways, application of XGBoost and SHAP for real-time accident detection and feature analysis. Accident Analysis & Prevention, 2020, 136, 105405.
15	SHI X, WONG Y D, LI M Z F, et al A feature learning approach based on XGBoost for driving assessment and risk prediction. Accident Analysis & Prevention, 2019, 129, 170- 179.
16	MACHADO M R, KARRY S, DE S I T. LGBM: an effective decision tree gradient boosting method to predict customer loyalty in the finance industry. Proc. of the 14th International Conference on Computer Science & Education, 2019: 1111−1116.
17	CHEN C, ZHANG Q M, MA Q, et al LGBM-PPI: predicting protein-protein interactions through LGBM with multi-information fusion. Chemometrics and Intelligent Laboratory Systems, 2019, 191, 54- 64. doi: 10.1016/j.chemolab.2019.06.003
18	TAN Y J, JIN B, CUI Q, et al. Generalizing fault detection against domain shifts using stratification-aware cross-validation. https://arxiv.org.abs/2008.08713.
19	WANG Q, DU J L, LI N Robust description and recognition of various viewpoint dynamic textures. Journal of Systems Engineering and Electronics, 2016, 27 (4): 920- 925. doi: 10.21629/JSEE.2016.04.21
20	YADAV S, SHUKLA S. Analysis of k-fold cross-validation over hold-out validation on colossal datasets for quality classification. Proc. of the 6th International Conference on Advanced Computing, 2016: 78−83.
21	XIONG M L, WANG H W, XU Y, et. al. General aviation safety research based on prediction of bird strike symptom. Systems Engineering and Electronics, 2020, 42(9): 2033−2040. (in Chinese)
22	MOT (Ministry of Transport of the People’s Republic of China). Annual implementation plan for strengthening transportation safety production. http://www.gov.cn/zhengce/zhengceku/2022-04/09/content_5684264.htm.
23	Federal Aviation Administration. Title 14 of the Code of Federal Regulations (14 CFR) part 121 air carriers. Washington D.C.: United States Department of Transport, 2020.
24	WILL K. Feature selector: simple feature selection in Python. https://github.com/WillKoehrsen/feature-selector.
25	UKKONEN P, MAKELA A Evaluation of machine learning classifiers for predicting deep convection. Journal of Advances in Modeling Earth Systems, 2019, 11 (6): 1784- 1802. doi: 10.1029/2018MS001561
26	KE G L, MENG Q, FINLEY T, et al. LGBM: a highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 2017: 3146−3154.
27	BROWNE M W Cross-validation methods. Journal of Mathematical Psychology, 2000, 44 (1): 108- 132. doi: 10.1006/jmps.1999.1279
28	PUTATUNDA S, RAMA K. A comparative analysis of hyperopt as against other approaches for hyper-parameter optimization of XGBoost. Proc. of the International Conference on Signal Processing and Machine Learning, 2018: 6−10.
29	MASSAOUDI M, REFAAT S S, CHIHI I, et al A novel stacked generalization ensemble-based hybrid LGBM-XGB-MLP model for short-term load forecasting. Energy, 2020, 214, 118874.
30	AKIBA T, SANO S, YANASE T, et al. Optuna: a next-generation hyperparameter optimization framework. Proc. of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019: 2623−2631.

Attribute	Content
Time/day	Date: 201801 Local time of day: 0601-1200
Place	Local reference.Airport: ZZZ.Airport State reference: US Altitude.Mean sea level (MSL).Single value: 1800
Environment	Flight conditions: visual flight rules Light: daylight
Aircraft	ATC/Advisory: TRACON ZZZ Aircraft operator: air carrier Make model name: B737-800 Crew size.Number of crew: 2 Operating under FAR Part: Part 121 Flight plan: IFR Flight phase: approach Airspace: class B ZZZ
Component	Aircraft component: landing gear Problem: malfunctioning
Event	Were passengers involved in even: N Detector: flight crew When detected: in-flight Result: general maintenance action
Assessment	Contributing factors/situations: aircraft Contributing factors/situations: aircraft Primary problem: human factors
Synopsis	B737 Captain reported malfunctioning landing gear

Category	C1	The other cause
C1	TP	FP
The other cause	FN	TN

Parameter	Range of grid	Final parameter	Parameter	Range of grid	Final parameter
num\_leaves	[100, 400]	143	feature\_fraction	[0.2,1]	0.63
max\_bin	[50, 300]	147	bagging\_fraction	[0.2,1]	1
max\_depth	[4,20]	6	bagging\_freq	[1,7]	7
min\_child\_weight	[1,6]	4	learning\_rate	[5, 100]	68
$ \lambda_{1} $	[0.001,1]	0	learning\_rate	[0.01,0.5]	0.04
$ \lambda_{2} $	[0.001,1]	0.75	n\_estimators	(500, 5000)	1045

Model	Accuracy	Precision	Recall	F1
LGBM-HSCV-CV	0.897	0.899	0.897	0.891
LGBM-HSCV	0.896	0.901	0.912	0.899

Classification of aviation incident causes using LGBM with improved cross-validation

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 14

References 30

Related Articles 0

Recommended Articles

Metrics

Comments