Journal of Systems Engineering and Electronics ›› 2023, Vol. 34 ›› Issue (3): 723-743.doi: 10.23919/JSEE.2023.000073

• CONTROL THEORY AND APPLICATION • Previous Articles    

Adaptive resource allocation for workflow containerization on Kubernetes

Chenggang SHAN1,2(), Chuge WU1(), Yuanqing XIA1(), Zehua GUO1(), Danyang LIU1(), Jinhui ZHANG1,*()   

  1. 1 School of Automation, Beijing Institute of Technology, Beijing 100081, China
    2 School of Artificial Intelligence, Zaozhuang University, Zaozhuang 277100, China
  • Received:2022-09-23 Online:2023-06-15 Published:2023-06-30
  • Contact: Jinhui ZHANG E-mail:uzz_scg@163.com;wucg@bit.edu.cn;xia_yuanqing@bit.edu.cn;guo@bit.edu.cn;liudanyang093@163.com;zhangjinh@bit.edu.cn
  • About author:
    SHAN Chenggang was born 1982. He received his M.S. degree in computer applied technology from Qiqihr University, China, in 2007. He is working toward his Ph.D. degree with the School of Automation, Beijing Institute of Technology, Beijing, China. He was an associate professor with the School of Artificial Intelligence, Zaozhuang University, China, in 2017. His research interests include networked control systems, cloud computing, cloud-edge collaboration, wireless networks. E-mail: uzz_scg@163.com

    WU Chuge was born in 1993. She received her B.E. degree in automatic control from Tsinghua University, Beijing, China, in 2015, and her M.S. and Ph.D. degrees in control theory and its applications from Tsinghua University, Beijing, China, in 2021. She was a visiting scholar with the University of Sydney, NSW, Australia, in 2018. She is currently a assistant processor for the School of Automation, Beijing Institute of Technology. Her current research interests include the scheduling and optimization theory and algorithms for cloud computing, fog computing systems, real-time scheduling, and evolutionary algorithms. E-mail: wucg@bit.edu.cn

    XIA Yuanqing was born in 1971. He received his Ph.D. degree in control theory and control engineering from Beijing University of Aeronautics and Astronautics, Beijing, China, in 2001. From January 2002 to November 2003, he was a postdoctoral research associate with the Institute of Systems Science, Academy of Mathematics and System Sciences, Chinese Academy of Sciences, Beijing, China. From November 2003 to February 2004, he was with National University of Singapore as a research fellow, where he worked on variable structure control. From February 2004 to February 2006, he was with University of Glamorgan, Pontypridd, U.K., as a research fellow. From February 2007 to June 2008, he was a guest professor with Innsbruck Medical University, Innsbruck, Austria. Since 2004, he has been with the School of Automation, Beijing Institute of Technology, Beijing, first as an associate professor, then, since 2008, as a professor. His research interests include networked control systems, robust control and signal processing, and active disturbance rejection control. E-mail: xia_yuanqing@bit.edu.cn

    GUO Zehua was born in 1985. He received his B.S. degree from Northwestern Polytechnical University, Xi’an, China, M.S. degree from Xidian University, Xi’an, China, and Ph.D. degree from Northwestern Polytechnical University. He was a research fellow at the Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University, New York, NY, USA, and a research associate at the Department of Computer Science and Engineering, University of Minnesota Twin Cities, Minneapolis, MN, USA. His research interests include programmable networks (e.g., software-defined networking, network function virtualization), machine learning, and network security. E-mail: guo@bit.edu.cn

    LIU Danyang was born in 1993. He received his B.S. degree in mathematics and information science from Shijiazhuang University, Shijiazhuang, China, in 2016, and M.S. degree from Hebei University of Science and Technology University, Shijiazhuang, China, in 2020. He is currently pursuing his Ph.D. degree in control science and engineering from the School of Automation, Beijing Institute of Technology, Beijing, China. His research interests include cloud computing and data center networks. E-mail: liudanyang093@163.com

    ZHANG Jinhui was born in 1982. He received his Ph.D. degree in control science and engineering from Beijing Institute of Technology, Beijing, China, in 2011. He was a research associate in the Department of Mechanical Engineering, University of Hong Kong, from February 2010 to May 2010, a senior research associate in the Department of Manufacturing Engineering and Engineering Management, City University of Hong Kong, from December 2010 to March 2011, and a visiting fellow with the School of Computing, Engineering & Mathematics, University of Western Sydney, Sydney, Australia, from February 2013 to May 2013. He was an associate professor in the Beijing University of Chemical Technology, Beijing, from March 2011 to March 2016, a professor in the School of Electrical and Automation Engineering, Tianjin University, Tianjin, from April 2016 to September 2016. He joined Beijing Institute of Technology in October 2016, where he is currently an tenured professor. His research interests include networked control systems and composite disturbance rejection control. E-mail: zhangjinh@bit.edu.cn
  • Supported by:
    This work was supported by the National Natural Science Foundation of China (61873030, 62002019)

Abstract:

In a cloud-native era, the Kubernetes-based workflow engine enables workflow containerized execution through the inherent abilities of Kubernetes. However, when encountering continuous workflow requests and unexpected resource request spikes, the engine is limited to the current workflow load information for resource allocation, which lacks the agility and predictability of resource allocation, resulting in over and under-provisioning resources. This mechanism seriously hinders workflow execution efficiency and leads to high resource waste. To overcome these drawbacks, we propose an adaptive resource allocation scheme named adaptive resource allocation scheme (ARAS) for the Kubernetes-based workflow engines. Considering potential future workflow task requests within the current task pod’s lifecycle, the ARAS uses a resource scaling strategy to allocate resources in response to high-concurrency workflow scenarios. The ARAS offers resource discovery, resource evaluation, and allocation functionalities and serves as a key component for our tailored workflow engine (KubeAdaptor). By integrating the ARAS into KubeAdaptor for workflow containerized execution, we demonstrate the practical abilities of KubeAdaptor and the advantages of our ARAS. Compared with the baseline algorithm, experimental evaluation under three distinct workflow arrival patterns shows that ARAS gains time-saving of 9.8% to 40.92% in the average total duration of all workflows, time-saving of 26.4% to 79.86% in the average duration of individual workflow, and an increase of 1% to 16% in centrol processing unit (CPU) and memory resource usage rate.

Key words: resource allocation, workflow containerization, Kubernetes, workflow management engine