Reconstruction of time series with missing value using 2D representation-based denoising autoencoder

doi:10.23919/JSEE.2020.000081

Abstract

Abstract:

Time series analysis is a key technology for medical diagnosis, weather forecasting and financial prediction systems. However, missing data frequently occur during data recording, posing a great challenge to data mining tasks. In this study, we propose a novel time series data representation-based denoising autoencoder (DAE) for the reconstruction of missing values. Two data representation methods, namely, recurrence plot (RP) and Gramian angular field (GAF), are used to transform the raw time series to a 2D matrix for establishing the temporal correlations between different time intervals and extracting the structural patterns from the time series. Then an improved DAE is proposed to reconstruct the missing values from the 2D representation of time series. A comprehensive comparison is conducted amongst the different representations on standard datasets. Results show that the 2D representations have a lower reconstruction error than the raw time series, and the RP representation provides the best outcome. This work provides useful insights into the better reconstruction of missing values in time series analysis to considerably improve the reliability of time-varying system.

Key words: time series, missing value, 2D representation, denoising autoencoder (DAE), reconstruction

Huamin TAO, Qiuqun DENG, Shanzhu XIAO. Reconstruction of time series with missing value using 2D representation-based denoising autoencoder[J]. Journal of Systems Engineering and Electronics, 2020, 31(6): 1087-1096.

Figures/Tables 14

Fig 1

Fig 2

Fig 3

Fig 4

Table 1

Fig 5

Fig 6

Fig 7

Table 2

Table 3

Fig 8

Fig 9

Table 4

References 32

1	BOX G E P, JENKINS G M Time series analysis: forecasting and control. Journal of Time, 1977, 19 (3): 343- 344.
2	CHATFIELD C. The analysis of time series: an introduction. Boca Raton: CRC Press, 2016.
3	YOZGATLIGIL C, ASLAN S, IYIGUN C, et al Comparison of missing value imputation methods in time series: the case of Turkish meteorological data. Theoretical and Applied Climatology, 2013, 112 (1/2): 143- 167. doi: 10.1007/s00704-012-0723-x
4	PRATAMA I, PERMANASARI A E, ARDIYANTO I, et al. A review of missing values handling methods on time-series data. Proc. of the International Conference on Information Technology Systems and Innovation, 2016: 1−6.
5	LEE K J, SIMPSON J A Introduction to multiple imputation for dealing with missing data. Respirology, 2014, 19 (2): 162- 167. doi: 10.1111/resp.12226
6	PENG L A comparison study of missing value processing methods. Computer Science, 2004, 31 (10): 155- 156,174.
7	LITTLE R J A, RUBIN D B. Statistical analysis with missing data. Hoboken: Wiley, 2019.
8	LOBATO F, SALES C, ARAUJO I, et al Multi-objective genetic algorithm for missing data imputation. Pattern Recognition Letters, 2015, 68 (15): 126- 131.
9	BANBURA M, MODUGNO M Maximum likelihood estimation of factor models on data sets with arbitrary pattern of missing data. Journal of Applied Econometrics, 2014, 29 (1): 133- 160. doi: 10.1002/jae.2306
10	HAYKIN S. Neural networks: a comprehensive foundation. Upper Saddle River: Prentice-Hall, 1999.
11	LECUN Y, BENGIO Y, HINTON G Deep learning. Nature, 2015, 521 (7553): 436- 444. doi: 10.1038/nature14539
12	BENGIO Y, COURVILLE A, VINCENT P Representation learning: a review and new perspectives. IEEE Trans. on Pattern Analysis & Machine Intelligence, 2013, 35 (8): 1798- 1828.
13	FAWAZ H I, FORESTIER G, WEBER J, et al Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 2019, 33, 917- 963. doi: 10.1007/s10618-019-00619-1
14	DENG Q W, LU H Z, HU M F, et al Exo-atmospheric infrared objects classification using recurrence-plots-based convolutional neural networks. Applied Optics, 2019, 58 (1): 164- 171. doi: 10.1364/AO.58.000164
15	YOON J. GAIN: missing data imputation using generative adversarial nets. Proc. of the 35th International Conference on Machine Learning, 2018: 1−8.
16	GUO Z J, WAN Y M, YE H A data imputation method for multivariate time series based on generative adversarial network. Neurocomputing, 2019, 360, 185- 197. doi: 10.1016/j.neucom.2019.06.007
17	ASSENDORP J P. Deep learning for anomaly detection in multivariate time series data. Hamburg: Hochschule für Angewandte Wissenschaften, 2017.
18	ESTEBAN C, HYLAND S L, RTSCH G. Real-valued (medical) time series generation with recurrent conditional gans. arXiv, 2017: arXiv170602633E.
19	VINCENT P, LAROCHELLE H, BENGIO Y, et al. Extracting and composing robust features with denoising autoencoders. Proc. of the 25th International Conference on Machine learning, 2008: 1096–1103.
20	HINTON G E, SALAKHUTDINOV R R Reducing the dimensionality of data with neural networks. Science, 2006, 313 (5786): 504- 507. doi: 10.1126/science.1127647
21	KAMPFFMEYER M, LOKSE S, BIANCHI F M, et al. Deep kernelized autoencoders. SHARMA P, BIANCHI F, ed. Image Analysis. Cham: Springer, 2017: 419–430.
22	BIANCHI F M, LIVI L, Mikalsen K, et al. Learning representations for multivariate time series with missing data using temporal kernelized autoencoders. arXiv, 2018: arXiv1805.03473.
23	SILVA D F, SOUZA V M A D, BATISTA G E A P A. Time series classification using compression distance of recurrence plots. Proc. of the IEEE International Conference on Data Mining, 2013: 687−696.
24	SOUZA V M A D, SILVA D F, BATISTA G E A P A. Extracting texture features for time series classification. Proc. of the 22nd International Conference on Pattern Recognition, 2014: 1425−1430.
25	WANG Z, OATES T. Imaging time-series to improve classification and imputation. Proc. of the 24th International Joint Conference on Artificial Intelligence, 2015: 3939−3945.
26	ECKMANN J P, KAMPHORST S O, RUELLE D Recurrence plots of dynamical systems. Europhysics Letters, 1987, 4 (9): 973- 977. doi: 10.1209/0295-5075/4/9/004
27	MARWAN N, DONGES J F, ZOU Y, et al Complex network approach for recurrence analysis of time series. Physics Letters A, 2009, 373 (46): 4246- 4254. doi: 10.1016/j.physleta.2009.09.042
28	MARWAN N, ROMANO M C, THIEL M, et al Recurrence plots for the analysis of complex systems. Physics Reports, 2007, 438 (5/6): 237- 329. doi: 10.1016/j.physrep.2006.11.001
29	HATAMI N, GAVET Y, DEBAYLE J Bag of recurrence patterns representation for time-series classification. Pattern Analysis & Applications, 2019, 22, 877- 887.
30	DAU H A, BAGNALL A, KAMGAR K, et al The UCR time series classification archive. IEEE/CAA Journal of Automatica Sinica, 2019, 6 (6): 1293- 1305. doi: 10.1109/JAS.2019.1911747
31	LI J H, STRUZIK Z, ZHANG L Q, et al Feature learning from incomplete EEG with denoising autoencoder. Neurocomputing, 2015, 165, 23- 31. doi: 10.1016/j.neucom.2014.08.092
32	KINGMA D P, BA J. Adam: a method for stochastic optimization. arXiv, 2014: arXiv1412.6980K.

Dataset	Class	Length	Training/Validation/Test
ECG200	2	96	100/50/50
Face all	14	131	560/845/845
Swedish leaf	15	128	500/313/312
OSU leaf	6	427	200/121/121
Wafer	2	152	1 000/3 082/3 082
50 words	50	270	450/272/273
Coffee	2	287	28/14/14

Dataset	MSE
Dataset	Raw time series	GASF time series	Mean	AR
ECG200	0.000 5	0.000 3	0.018 6	0.031 4
Face all	0.000 7	0.001 7	0.029 9	0.059 2
Swedish leaf	0.001 8	0.000 9	0.014 3	0.040 3
OSU leaf	0.001 0	0.000 6	0.018 8	0.026 9
Wafer	0.000 2	0.002 4	0.031 3	0.031 4
50 words	0.001 9	0.001 1	0.018 5	0.052 3
Coffee	0.057 9	0.004 0	0.013 1	0.006 1

Dataset	MSE
Dataset	GASF	RP
ECG200	0.004 8	0.003 7
Face all	0.013 4	0.022 1
Swedish leaf	0.009 8	0.009 2
OSU leaf	0.007 7	0.012 1
Wafer	0.024 0	0.006 9
50 words	0.010 1	0.010 8
Coffee	0.033 6	0.023 4

Dataset	1D-CNN	1D-CNN-0.2	GASF-CNN	GASF-CNN-0.2	RP-CNN	RP-CNN-0.2
Swedish leaf	0.913 4	0.330 1	0.923 0(64)	0.900 6(64)	0.942 3(32)	0.916 6(32)
Face all	0.830 7	0.801 1	0.757 3(64)	0.738 4(64)	0.757 0(64)	0.738 4(64)
ECG200	0.859 9	0.840 0	0.920 0(20)	0.910 0(20)	0.930 0(20)	0.890 0(20)
Wafer	0.994 4	0.993 8	0.994 4(64)	0.990 9(64)	0.997 4(20)	0.995 0(20)
OSU leaf	0.454 5	0.446 2	0.590 0(64)	0.545 4(64)	0.595 0(64)	0.573 0(64)
Coffee	1.000	1.000	1.000(64)	1.000(64)	1.000(64)	1.000(64)

[1]	Xinjian MA, Shiqian LIU, Huihui CHENG. Civil aircraft fault tolerant attitude tracking based on extended state observers and nonlinear dynamic inversion [J]. Journal of Systems Engineering and Electronics, 2022, 33(1): 180-187.
[2]	Zhiyuan SHEN, Qianqian WANG, Xinmiao CHENG. A sparsity adaptive compressed signal reconstruction based on sensing dictionary [J]. Journal of Systems Engineering and Electronics, 2021, 32(6): 1345-1353.
[3]	Shuai LIU, Xue ZHANG, Fenggang YAN, Jun WANG, Ming JIN. Fast and accurate covariance matrix reconstruction for adaptive beamforming using Gauss-Legendre quadrature [J]. Journal of Systems Engineering and Electronics, 2021, 32(1): 38-43.
[4]	Ying LI, Guanghong GONG, Lin SUN. A fast, accurate and dense feature matching algorithm for aerial images [J]. Journal of Systems Engineering and Electronics, 2020, 31(6): 1128-1139.
[5]	Ke ZHANG, Le CUI, Yao YIN. A multivariate grey incidence model for different scale data based on spatial pyramid pooling [J]. Journal of Systems Engineering and Electronics, 2020, 31(4): 770-779.
[6]	Haiwen SUN, Xiaofang XIE. Threat evaluation method of warships formation air defense based on AR(p)-DITOPSIS [J]. Journal of Systems Engineering and Electronics, 2019, 30(2): 297-307.
[7]	Kun ZHANG, Weiren KONG, Peipei LIU, Jiao SHI, Yu LEI, Jie ZOU. Assessment and sequencing of air target threat based on intuitionistic fuzzy entropy and dynamic VIKOR [J]. Journal of Systems Engineering and Electronics, 2018, 29(2): 305-310.
[8]	Yong Shuai, Tailiang Song, and Jianping Wang. Integrated parallel forecasting model based on modified fuzzy time series and SVM [J]. Systems Engineering and Electronics, 2017, 28(4): 766-.
[9]	Xiaoshi Fan, Yingjie Lei, and Yanan Wang. Adaptive partition intuitionistic fuzzy time series forecasting model [J]. Systems Engineering and Electronics, 2017, 28(3): 585-596.
[10]	Chengguang Wu, Hongqiang Wang, Bin Deng, Yuliang Qin, and Wuge Su. Autofocus technique for ISAR imaging of uniformly rotating targets based on the ExCoV method [J]. Systems Engineering and Electronics, 2017, 28(2): 267-275.
[11]	Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural networks for time series classification [J]. Systems Engineering and Electronics, 2017, 28(1): 162-.
[12]	Yan Zhang, Jichang Guo, and Xianguo Li. Efficient recovery of group-sparse signals with truncated and reweighted l_2,1-regularization [J]. Systems Engineering and Electronics, 2017, 28(1): 19-.
[13]	Ya’nan Wang, Yingjie Lei, Yang Lei, Xiaoshi Fan. Multi-factor high-order intuitionistic fuzzy time series forecasting model [J]. Journal of Systems Engineering and Electronics, 2016, 27(5): 1054-1062.
[14]	Lin Zhang and Yicheng Jiang. Imaging algorithm of multi-ship motion target based on compressed sensing [J]. Systems Engineering and Electronics, 2016, 27(4): 790-.
[15]	Yong Han, Qingyuan Fang,Fenggang Yan,Ming Jin, and Xiaolin Qiao. Joint DOA and polarization estimation for unequal power sources based on reconstructed noise subspace [J]. Systems Engineering and Electronics, 2016, 27(3): 501-513.