Distributed spatio-temporal generative adversarial networks

doi:10.23919/JSEE.2020.000026

Abstract

Abstract:

Owing to the wide range of applications in various fields, generative models have become increasingly popular. However, they do not handle spatio-temporal features well. Inspired by the recent advances in these models, this paper designs a distributed spatio-temporal generative adversarial network (STGAN-D) that, given some initial data and random noise, gene-rates a consecutive sequence of spatio-temporal samples which have a logical relationship. This paper builds a spatio-temporal discriminator to distinguish whether the samples generated by the generator meet the requirements for time and space coherence, and builds a controller for distributed training of the network gradient updated to separate the model training and parameter updating, to improve the network training rate. The model is trained on the skeletal dataset and the traffic dataset. In contrast to traditional generative adversarial networks (GANs), the proposed STGAN-D can generate logically coherent samples with the corresponding spatial and temporal features while avoiding mode collapse. In addition, this paper shows that the proposed model can generate different styles of spatio-temporal samples given different random noise inputs, and the controller can improve the network training rate. This model will extend the potential range of applications of GANs to areas such as traffic information simulation and multi-agent adversarial simulation.

Key words: distributed spatio-temporal generative adversarial network (STGAN-D), spatial discriminator, temporal discriminator, speed controller

Chao QIN, Xiaoguang GAO. Distributed spatio-temporal generative adversarial networks[J]. Journal of Systems Engineering and Electronics, 2020, 31(3): 578-592.

Figures/Tables 20

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Fig 8

Table 1

Table 2

Fig 9

Table 3

Fig 10

Fig 11

Fig 12

Fig 13

Fig 14

Fig 15

Fig 16

Fig 17

References 31

1	HINTON G E, OSINDERO S, TEH Y W. A fast learning algorithm for deep belief nets. Neural Computation, 2006, 18 (7): 1527- 1554. doi: 10.1162/neco.2006.18.7.1527
2	BOUSMALIS K, SILBERMAN N, DOHAN D, et al. Unsupervised pixel-level domain adaptation with generative adversarial networks. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 3722- 3731.
3	KIM T, CHA M, KIM H, et al. Learning to discover cross-domain relations with generative adversarial networks. Proc. of the 34th International Conference on Machine Learning, 2017, 1857- 1865.
4	LIN K, LI D, HE X, et al. Adversarial ranking for language generation. Proc. of Advances in Neural Information Processing Systems, 2017, 3155- 3165.
5	HSU C C, HWANG H T, WU Y C, et al. Voice conversion from unaligned corpora using variational autoencoding Wasserstein generative adversarial networks. https://arxiv.org/abs/1704.00849.
6	DAI W, LIANG X, ZHANG H, et al. Structure correcting adversarial network for chest X-rays organ segmentation. US Patent Application: 15/925, 998, 2018-9-27.
7	MARDANI M, GONG E, CHENG J Y, et al. Deep generative adversarial networks for compressed sensing automates MRI. https://arxiv.org/abs/1706.00051.
8	SHI H, DONG J, WANG W, et al. SSGAN: secure steganography based on generative adversarial networks. Proc. of the Pacific Rim Conference on Multimedia, 2017, 534- 544.
9	VOLKHONSKIY D, NAZAROV I, BORISENKO B, et al. Steganographic generative adversarial networks. Proc. of the 12th International Conference on Machine Vision, 2020, 11433.
10	SALIMANS T, KARPATHT A, CHEN X, et al. PixelCNN++: improving the PixelCNN with discretized logistic mixture likelihood and other modifications. https://arxiv.org/abs/1701.05517.
11	OORD A, KALCHBRENNER N, KAVUKCUOGLU K. Pixel recurrent neural networks. https: //arxiv.org/ abs/1601.06759.
12	NOWOZIN S, CSEKE B, TOMIOKA R. F-GAN: training generative neural samplers using variational divergence minimization. Proc. of Advances in Neural Information Processing Systems, 2016, 271- 279.
13	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets. Proc. of Advances in Neural Information Processing Systems, 2014, 2672- 2680.
14	SMOLENSKY P. Information processing in dynamical systems: foundations of harmony theory. RUMELHART D E, MCCLELLAND J L. Ed. Parallel distributed processing. Cambridge: MIT Press, 1986: 194-281.
15	ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN. https://arxiv.org/abs/1701.07875.
16	LIU M Y, BREUEL T, KAUTZ J. Unsupervised image-to-image translation networks. Proc. of Advances in Neural Information Processing Systems, 2017, 700- 708.
17	GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of Wasserstein GANs. Proc. of Advances in Neural Information Processing Systems, 2017, 5767- 5777.
18	RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks. https://arxiv.org/abs/1511.06434.
19	ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 1125- 1134.
20	YU L, ZHANG W, WANG J, et al. SeqGAN: sequence genera-tive adversarial nets with policy gradient. Proc. of the AAAI Conference on Artificial Intelligence, 2017, 2852- 2858.
21	MAO X, LI Q, XIE H, et al. Least squares generative adversarial networks. Proc. of the IEEE International Conference on Computer Vision, 2017, 2813- 2821.
22	SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 1- 9.
23	MA X, TAO Z, WANG Y, et al. Long short-term memory neural network for traffic speed prediction using remote microwave sensor data. Transportation Research Part C: Emerging Technologies, 2015, 54, 187- 197. doi: 10.1016/j.trc.2015.03.014
24	SHAHROUDY A, LIU J, NG T T, et al. NTU RGB+ D: a large scale dataset for 3D human activity analysis. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 1010- 1019.
25	CHEN C, PETTY K, SKABARDONIS A, et al. Freeway performance measurement system: mining loop detector data. Transportation Research Record: Journal of the Transportation Research Board, 2001, 1748, 96- 102. doi: 10.3141/1748-12
26	KINGMA D P, BA J. Adam: a method for stochastic optimization. https://arxiv.org/abs/1412.6980.
27	SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GANs. https://arxiv.org/abs/1606.03498.
28	XIAO T, LI H, OUYANG W L, et al. Learning deep feature representations with domain guided dropout for person re-identification. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 1249- 1258.
29	IRSOY O, ALPAYDIN E. Dropout regularization in hierarchical mixture of experts. https://arxiv.org/abs/1812.10158.
30	CAI Y Q, LI Q X, SHEN Z W. On the convergence and robustness of batch normalization. https://arxiv.org/abs/1810.00122.
31	JUNG W K, JUNG D J, KIM B H, et al. Restructuring batch normalization to accelerate CNN training. https://arxiv.org/abs/1807.01702.

$T$	0	1	2	4	6	8
Iteration	2 400	2 300	2 100	1 800	1 500	1 100

Method	Score
GAN	3.2 $\pm $ 0.84
WGAN	4.4 $\pm $ 0.82
SeqGAN	4.2 $\pm $ 0.47
STGAN	5.3 $\pm $ 0.46
STGAN-D	5.3 $\pm $ 0.49