Adaptive resource allocation for workflow containerization on Kubernetes

doi:10.23919/JSEE.2023.000073

Journal of Systems Engineering and Electronics ›› 2023, Vol. 34 ›› Issue (3): 723-743.doi: 10.23919/JSEE.2023.000073

• • 上一篇

收稿日期:2022-09-23 出版日期:2023-06-15 发布日期:2023-06-30

Adaptive resource allocation for workflow containerization on Kubernetes

Chenggang SHAN¹^,²(), Chuge WU¹(), Yuanqing XIA¹(), Zehua GUO¹(), Danyang LIU¹(), Jinhui ZHANG¹^,*()

¹ School of Automation, Beijing Institute of Technology, Beijing 100081, China
² School of Artificial Intelligence, Zaozhuang University, Zaozhuang 277100, China

Received:2022-09-23 Online:2023-06-15 Published:2023-06-30
Contact: Jinhui ZHANG E-mail:uzz_scg@163.com;wucg@bit.edu.cn;xia_yuanqing@bit.edu.cn;guo@bit.edu.cn;liudanyang093@163.com;zhangjinh@bit.edu.cn
About author:
SHAN Chenggang was born 1982. He received his M.S. degree in computer applied technology from Qiqihr University, China, in 2007. He is working toward his Ph.D. degree with the School of Automation, Beijing Institute of Technology, Beijing, China. He was an associate professor with the School of Artificial Intelligence, Zaozhuang University, China, in 2017. His research interests include networked control systems, cloud computing, cloud-edge collaboration, wireless networks. E-mail: uzz_scg@163.com

WU Chuge was born in 1993. She received her B.E. degree in automatic control from Tsinghua University, Beijing, China, in 2015, and her M.S. and Ph.D. degrees in control theory and its applications from Tsinghua University, Beijing, China, in 2021. She was a visiting scholar with the University of Sydney, NSW, Australia, in 2018. She is currently a assistant processor for the School of Automation, Beijing Institute of Technology. Her current research interests include the scheduling and optimization theory and algorithms for cloud computing, fog computing systems, real-time scheduling, and evolutionary algorithms. E-mail: wucg@bit.edu.cn

XIA Yuanqing was born in 1971. He received his Ph.D. degree in control theory and control engineering from Beijing University of Aeronautics and Astronautics, Beijing, China, in 2001. From January 2002 to November 2003, he was a postdoctoral research associate with the Institute of Systems Science, Academy of Mathematics and System Sciences, Chinese Academy of Sciences, Beijing, China. From November 2003 to February 2004, he was with National University of Singapore as a research fellow, where he worked on variable structure control. From February 2004 to February 2006, he was with University of Glamorgan, Pontypridd, U.K., as a research fellow. From February 2007 to June 2008, he was a guest professor with Innsbruck Medical University, Innsbruck, Austria. Since 2004, he has been with the School of Automation, Beijing Institute of Technology, Beijing, first as an associate professor, then, since 2008, as a professor. His research interests include networked control systems, robust control and signal processing, and active disturbance rejection control. E-mail: xia_yuanqing@bit.edu.cn

GUO Zehua was born in 1985. He received his B.S. degree from Northwestern Polytechnical University, Xi’an, China, M.S. degree from Xidian University, Xi’an, China, and Ph.D. degree from Northwestern Polytechnical University. He was a research fellow at the Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University, New York, NY, USA, and a research associate at the Department of Computer Science and Engineering, University of Minnesota Twin Cities, Minneapolis, MN, USA. His research interests include programmable networks (e.g., software-defined networking, network function virtualization), machine learning, and network security. E-mail: guo@bit.edu.cn

LIU Danyang was born in 1993. He received his B.S. degree in mathematics and information science from Shijiazhuang University, Shijiazhuang, China, in 2016, and M.S. degree from Hebei University of Science and Technology University, Shijiazhuang, China, in 2020. He is currently pursuing his Ph.D. degree in control science and engineering from the School of Automation, Beijing Institute of Technology, Beijing, China. His research interests include cloud computing and data center networks. E-mail: liudanyang093@163.com

ZHANG Jinhui was born in 1982. He received his Ph.D. degree in control science and engineering from Beijing Institute of Technology, Beijing, China, in 2011. He was a research associate in the Department of Mechanical Engineering, University of Hong Kong, from February 2010 to May 2010, a senior research associate in the Department of Manufacturing Engineering and Engineering Management, City University of Hong Kong, from December 2010 to March 2011, and a visiting fellow with the School of Computing, Engineering & Mathematics, University of Western Sydney, Sydney, Australia, from February 2013 to May 2013. He was an associate professor in the Beijing University of Chemical Technology, Beijing, from March 2011 to March 2016, a professor in the School of Electrical and Automation Engineering, Tianjin University, Tianjin, from April 2016 to September 2016. He joined Beijing Institute of Technology in October 2016, where he is currently an tenured professor. His research interests include networked control systems and composite disturbance rejection control. E-mail: zhangjinh@bit.edu.cn
Supported by:
This work was supported by the National Natural Science Foundation of China (61873030, 62002019)

摘要/Abstract

Abstract:

In a cloud-native era, the Kubernetes-based workflow engine enables workflow containerized execution through the inherent abilities of Kubernetes. However, when encountering continuous workflow requests and unexpected resource request spikes, the engine is limited to the current workflow load information for resource allocation, which lacks the agility and predictability of resource allocation, resulting in over and under-provisioning resources. This mechanism seriously hinders workflow execution efficiency and leads to high resource waste. To overcome these drawbacks, we propose an adaptive resource allocation scheme named adaptive resource allocation scheme (ARAS) for the Kubernetes-based workflow engines. Considering potential future workflow task requests within the current task pod’s lifecycle, the ARAS uses a resource scaling strategy to allocate resources in response to high-concurrency workflow scenarios. The ARAS offers resource discovery, resource evaluation, and allocation functionalities and serves as a key component for our tailored workflow engine (KubeAdaptor). By integrating the ARAS into KubeAdaptor for workflow containerized execution, we demonstrate the practical abilities of KubeAdaptor and the advantages of our ARAS. Compared with the baseline algorithm, experimental evaluation under three distinct workflow arrival patterns shows that ARAS gains time-saving of 9.8% to 40.92% in the average total duration of all workflows, time-saving of 26.4% to 79.86% in the average duration of individual workflow, and an increase of 1% to 16% in centrol processing unit (CPU) and memory resource usage rate.

Key words: resource allocation, workflow containerization, Kubernetes, workflow management engine

. [J]. Journal of Systems Engineering and Electronics, 2023, 34(3): 723-743.

Chenggang SHAN, Chuge WU, Yuanqing XIA, Zehua GUO, Danyang LIU, Jinhui ZHANG. Adaptive resource allocation for workflow containerization on Kubernetes[J]. Journal of Systems Engineering and Electronics, 2023, 34(3): 723-743.

图/表 11

Notation	Meaning
$ v_i $	A K8s node (VM), $ v_{i} \in \mathcal{V} $
$ s_{i,j} $	The $ j $ th task in workflow $ w_{i} $ ( $ 1 \leq i \leq k $ , $ 1 \leq j \leq n $ )
${\rm{allocated}}_{{\rm{cpu}}}$	The allocated CPU resource number through the adaptive resource allocation algorithm
${\rm{allocated}}_{{\rm{mem}}}$	The allocated memory resource number through the adaptive resource allocation algorithm
${\rm{request}}.{\rm{cpu}}$	The accumulated CPU resource number for many task requests
${\rm{request}}.{\rm{mem}}$	The accumulated memory resource number for many task requests
${\rm{Re}}_{{\rm{max}}}^{{\rm{cpu}}}$	The maximum remaining resource amount of CPU among K8s cluster nodes
${\rm{Re}}_{{\rm{max}}}^{{\rm{mem}}}$	The maximum remaining resource amount of memory among K8s cluster nodes
${\rm{totalResidual}}.{\rm{cpu}}$	The total of residual CPU resource across K8s cluster nodes
${\rm{totalResidual}}.{\rm{mem}}$	The total of residual memory resource across K8s cluster nodes
${\rm{task}}^{{\rm{req}}}$	The current task request with respects to $ s_{i,j} $
${\rm{task}}_{i,j}^{{\rm{redis}}}$	A record of task-state data from Redis database
${\rm{PodLister}}$	An interface of acquiring pod list in Informer component
${\rm{NodeLister}}$	An interface of acquiring node list in Informer component
${\rm{ResidualMap}}$	A data dictionary for storing remaining resources (CPU and memory) of each node
${\rm{nodeReq}}.{\rm{cpu}}$	The accumulated CPU resource requirements for all pods on a node
${\rm{nodeReq}}.{\rm{mem}}$	The accumulated memory resource requirements for all pods on a node
${\rm{allocatable}}.{\rm{cpu}}$	The accumulated allocatable CPU resource across K8s cluster nodes
${\rm{allocatable}}.{\rm{mem}}$	The accumulated allocatable memory resource across K8s cluster nodes
${\rm{residual}}.{\rm{cpu}}$	The residual CPU resource in K8s cluster
${\rm{residual}}.{\rm{mem}}$	The residual memory resource in K8s cluster
${\rm{pod}}_i$	A data struct from Podlist, which contains many key fields about container’s features
${\rm{cpu}}_{{\rm{cut}}}$	The allocated CPU resource amount for task request based on (9)
${\rm{mem}}_{{\rm{cut}}}$	The allocated memory resource amount for task request based on (9)
$ \alpha $	A proportional value derived from experience, $ \alpha \in (0,1) $
$ \beta $	A constant value derived from experience, $ \beta \geq 20 $

Workflow type	Metrics	Constant arrival		Linear arrival		Pyramid arrival
	Metrics	Adaptive	Baseline	Adaptive	Baseline	Adaptive	Baseline
	Number of workflow rquests	30		30		34
	Interval between two requests bursts/s	300		300		300
Montage	Total duration of all workflows/min (standard deviation)	33.18 ( $ \delta = 0.21 $ )	36.79 ( $ \delta = 2.26 $ )	26.95 ( $ \delta = 0.38 $ )	36.45 ( $ \delta = 6.31 $ )	49.31 ( $ \delta = 2.46 $ )	54.69 ( $ \delta = 1.74 $ )
	Average workflow duration/min (standard deviation)	5.74 ( $ \delta = 0.49 $ )	7.80 ( $ \delta = 0.36 $ )	5.41 ( $ \delta = 0.26 $ )	11.33 ( $ \delta = 4.28 $ )	7.22 ( $ \delta = 1.36 $ )	11.73 ( $ \delta = 0.88 $ )
	CPU resource usage (standard deviation)	0.28 ( $ \delta = 0.00 $ )	0.27 ( $ \delta = 0.02 $ )	0.35 ( $ \delta = 0.01 $ )	0.31 ( $ \delta = 0.07 $ )	0.26 ( $ \delta = 0.03 $ )	0.20 ( $ \delta = 0.01 $ )
	Memory resource usage (standard deviation)	0.28 ( $ \delta = 0.00 $ )	0.27 ( $ \delta = 0.13 $ )	0.35 ( $ \delta = 0.01 $ )	0.31 ( $ \delta = 0.07 $ )	0.26 ( $ \delta = 0.03 $ )	0.20 ( $ \delta = 0.01 $ )
Epigenomics	Total duration of all workflows/min (standard deviation)	30.55 ( $ \delta = 0.19 $ )	39.06 ( $ \delta = 1.84 $ )	34.3 ( $ \delta = 7.29 $ )	43.66 ( $ \delta = 4.37 $ )	51.42 ( $ \delta = 4.28 $ )	62.12 ( $ \delta = 4.32 $ )
	Average workflow duration/min (standard deviation)	4.24 ( $ \delta = 0.05 $ )	9.35 ( $ \delta = 1.56 $ )	9.81 ( $ \delta = 5.11 $ )	16.53 ( $ \delta = 4.41 $ )	9.65 ( $ \delta = 3.33 $ )	19.41 ( $ \delta = 6.04 $ )
	CPU resource usage (standard deviation)	0.34 ( $ \delta = 0.02 $ )	0.27 ( $ \delta = 0.01 $ )	0.32 ( $ \delta = 0.06 $ )	0.25 ( $ \delta = 0.00 $ )	0.21 ( $ \delta = 0.00 $ )	0.20 ( $ \delta = 0.00 $ )
	Memory resource usage (standard deviation)	0.34 ( $ \delta = 0.02 $ )	0.27 ( $ \delta = 0.01 $ )	0.32 ( $ \delta = 0.06 $ )	0.25 ( $ \delta = 0.00 $ )	0.21 ( $ \delta = 0.01 $ )	0.20 ( $ \delta = 0.01 $ )
CyberShake	Total duration of all workflows/min (standard deviation)	38.30 ( $ \delta = 5.77 $ )	50.29 ( $ \delta = 5.29 $ )	34.06 ( $ \delta = 6.16 $ )	49.46 ( $ \delta = 1.18 $ )	46.76 ( $ \delta = 4.02 $ )	66.41 ( $ \delta = 6.56 $ )
	Average workflow duration/min (standard deviation)	9.19 ( $ \delta = 3.72 $ )	17.29 ( $ \delta = 2.89 $ )	9.41 ( $ \delta = 4.27 $ )	20.61 ( $ \delta = 0.86 $ )	4.94 ( $ \delta = 2.07 $ )	19.47 ( $ \delta = 6.50 $ )
	CPU resource usage (standard deviation)	0.26 ( $ \delta = 0.03 $ )	0.24 ( $ \delta = 0.02 $ )	0.27 ( $ \delta = 0.04 $ )	0.24 ( $ \delta = 0.01 $ )	0.22 ( $ \delta = 0.03 $ )	0.19 ( $ \delta = 0.01 $ )
	Memory resource usage (standard deviation)	0.26 ( $ \delta = 0.03 $ )	0.24 ( $ \delta = 0.02 $ )	0.27 ( $ \delta = 0.04 $ )	0.23 ( $ \delta = 0.01 $ )	0.22 ( $ \delta = 0.03 $ )	0.19 ( $ \delta = 0.01 $ )
LIGO	Total duration of all workflows/min (standard deviation)	30.82 ( $ \delta = 0.38 $ )	52.17 ( $ \delta = 3.99 $ )	44.02 ( $ \delta = 10.9 $ )	53.87 ( $ \delta = 4.20 $ )	45.26 ( $ \delta = 0.52 $ )	63.56 ( $ \delta = 1.60 $ )
	Average workflow duration/min (standard deviation)	4.26 ( $ \delta = 0.09 $ )	21.15 ( $ \delta = 0.44 $ )	16.21 ( $ \delta = 7.68 $ )	28.05 ( $ \delta = 7.88 $ )	4.20 ( $ \delta = 0.15 $ )	14.07 ( $ \delta = 1.33 $ )
	CPU resource usage (standard deviation)	0.40 ( $ \delta = 0.00 $ )	0.24 ( $ \delta = 0.02 $ )	0.28 ( $ \delta = 0.08 $ )	0.23 ( $ \delta = 0.02 $ )	0.31 ( $ \delta = 0.01 $ )	0.23 ( $ \delta = 0.00 $ )
	Memory resource usage (standard deviation)	0.40 ( $ \delta = 0.00 $ )	0.24 ( $ \delta = 0.02 $ )	0.28 ( $ \delta = 0.08 $ )	0.23 ( $ \delta = 0.02 $ )	0.31 ( $ \delta = 0.01 $ )	0.23 ( $ \delta = 0.00 $ )

参考文献 40

1	BURNS B, GRANT B, OPPENHEIMER D, et al Borg, omega, and kubernetes. Communications of the ACM, 2016, 59 (5): 50- 57. doi: 10.1145/2890784
2	BERNSTEIN D Containers and cloud: from LXC to Docker to Kubernetes. IEEE Cloud Computing, 2014, 1 (3): 81- 84. doi: 10.1109/MCC.2014.51
3	JUVE G, CHERVENAK A, DEELMAN E, et al Characterizing and profiling scientific workflows. Future Generation Computer Systems, 2013, 29 (3): 682- 692. doi: 10.1016/j.future.2012.08.015
4	LEE Y C, ZOMAYA A Y Stretch out and compact: workflow scheduling with resource abundance. Proc. of the IEEE/ACM 13th International Symposium on Cluster, Cloud, and Grid Computing, 2013, 219- 226.
5	ZHENG C, TOVAR B, THAIN D Deploying high throughput scientific workflows on container schedulers with makeflow and mesos. Proc. of the IEEE/ACM 17th International Symposium on Cluster, Cloud and Grid Computing, 2017, 130- 139.
6	SILVER A Software simplified. Nature, 2017, 546 (7656): 173- 174. doi: 10.1038/546173a
7	DI T P, CHATZOU M, FLODEN E W, et al Nextflow enables reproducible computational workflows. Nature Biotechnology, 2017, 35 (4): 316- 319. doi: 10.1038/nbt.3820
8	DEELMAN E, VAHI K, JUVE G, et al Pegasus, a workflow management system for science automation. Future Generation Computer Systems, 2015, 46, 17- 35. doi: 10.1016/j.future.2014.10.008
9	DEELMAN E, VAHI K, RYNGE M, et al The evolution of the pegasus workflow management software. Computing in Science & Engineering, 2019, 21 (4): 22- 36.
10	JALILI V, AFGAN E, GU Q, et al The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2020 update. Nucleic Acids Research, 2020, 48 (W1): W395- W402. doi: 10.1093/nar/gkaa434
11	BADER J, THAMSEN L, KULAGINA S, et al Tarema: adaptive resource allocation for scalable scientific workflows in heterogeneous clusters. Proc. of the IEEE International Conference on Big Data, 2021, 65- 75.
12	HOENISCH P, SCHULTE S, DUSTDAR S, et al Self-adaptive resource allocation for elastic process execution. Proc. of the IEEE 6th International Conference on Cloud Computing, 2013, 220- 227.
13	HOENISCH P, SCHULTE S, DUSTDAR S. Workflow scheduling and resource allocation for cloud-based execution of elastic processes. Proc. of the IEEE 6th International Conference on Service-oriented Computing and Applications, 2013. DOI: 10.1109/SOCA.2013.44.
14	WITT C, WAGNER D, LESER U Feedback-based resource allocation for batch scheduling of scientific workflows. Proc. of the International Conference on High Performance Computing & Simulation, 2019, 761- 768.
15	KHATUA S, SUR P K, DAS R K, et al Heuristic-based resource reservation strategies for public cloud. IEEE Trans. on Cloud Computing, 2014, 4 (4): 392- 401.
16	ABDULLAH M, IQBAL W, BUKHARI F, et al Diminishing returns and deep learning for adaptive CPU resource allocation of containers. IEEE Trans. on Network and Service Management, 2020, 17 (4): 2052- 2063. doi: 10.1109/TNSM.2020.3033025
17	CHEN Z Y, HU J, MIN G, et al Adaptive and efficient resource allocation in cloud datacenters using actor-critic deep reinforcement learning. IEEE Trans. on Parallel and Distributed Systems, 2021, 33 (8): 1911- 1923.
18	CHEN M S, HUANG S, FU X, et al Statistical model checking-based evaluation and optimization for cloud workflow resource allocation. IEEE Trans. on Cloud Computing, 2016, 8 (2): 443- 458.
19	SCHULER L, JAMIL S, KUHL N AI-based resource allocation: reinforcement learning for adaptive auto-scaling in serverless environments. Proc. of the IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing, 2021, 804- 811.
20	SHAN C G, WANG G, XIA Y Q, et al Containerized workflow builder for Kubernetes. Proc. of the IEEE 23rd International Conference on High Performance Computing and Communications, 2021, 685- 692.
21	SHAN C G, WANG G, XIA Y Q, et al. KubeAdaptor: a docking framework for workflow containerization on Kubernetes. https://arxiv.53yu.com/abs/2207.01222.
22	IGLESIA D G D L, WEYNS D MAPE-K formal templates to rigorously design behaviors for self-adaptive systems. ACM Trans. on Autonomous and Adaptive Systems, 2015, 10 (3): 15.
23	ARCAINIP, RICCOBENE E, SCANDURRA P Modeling and analyzing MAPE-K feedback loops for self-adaptation. Proc. of the IEEE/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, 2015, 13- 23.
24	RZADCA K, FINDEISEN P, SWIDERSKI J, et al Autopilot: workload autoscaling at google. Proc. of the 15th European Conference on Computer Systems, 2020, 16.
25	GitHub-source code. https://github.com/CloudControlSystems/ResourceAllocation.
26	LEE K, PATON N W, SAKELLARIOU R, et al Adaptive workflow processing and execution in Pegasus. Concurrency and Computation: Practice and Experience, 2009, 21 (16): 1965- 1981. doi: 10.1002/cpe.1446
27	ISLAM S, KEUNG J, LEE K, et al Empirical prediction models for adaptive resource provisioning in the cloud. Future Generation Computer Systems, 2012, 28 (1): 155- 162. doi: 10.1016/j.future.2011.05.027
28	MAO Y, YAN W F, SONG Y, et al. Differentiate quality of experience scheduling for deep learning inferences with docker containers in the cloud. IEEE Trans. on Cloud Computing, 2022. DOI: 10.1109/TCC.2022.3154117.
29	YIN L, LUO J, LUO H B Tasks scheduling and resource allocation in fog computing based on containers for smart manufacturing. IEEE Trans. on Industrial Informatics, 2018, 14 (10): 4712- 4721. doi: 10.1109/TII.2018.2851241
30	HU S H, SHI W S, LI G H. CEC: a containerized edge computing framework for dynamic resource provisioning. IEEE Trans. on Mobile Computing, 2022. DOI: 10.1109/TMC.2022.3147800.
31	CHANG C C, YANG S R, YEH E H, et al. A Kubernetes-based monitoring platform for dynamic cloud resource provisioning. Proc. of the IEEE Global Communications Conference, 2017. DOI: 10.1109/GLOCOM.2017.8254046.
32	MAO Y, FU Y Q, GU S W, et al. Resource management schemes for cloud-native platforms with computing containers of docker and Kubernetes, 2020. https://arxiv.53yu.com/abs/2010.10350.
33	CHAKRABORTY J, MALTZAHN C, JIMENEZ L. Enabling seamless execution of computational and data science workflows on hpc and cloud with the popper container-native automation engine. Proc. of the 2nd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC, 2020: 8−18.
34	WAIBEL P, HOCHREINER C, SCHULTE S, et al ViePEP-C: a container-based elastic process platform. IEEE Trans. on Cloud Computing, 2019, 9 (4): 1657- 1674.
35	SCHULTE S, HOENISCH P, VENUGOPAL S, et al Introducing the vienna platform for elastic processes. Proc. of the International Conference on Service-Oriented Computing, 2013, 179- 190.
36	HOENISCH P, SCHULLER D, SCHULTE S, et al Optimization of complex elastic processes. IEEE Trans. on Services Computing, 2015, 9 (5): 700- 713.
37	KEPHART J O, CHESS D M The vision of autonomic computing. Computer, 2003, 36 (1): 41- 50. doi: 10.1109/MC.2003.1160055
38	Pegasus. Workflow gallery. https://pegasus.isi.edu/workflow_gallery.
39	Kubernetes. Configure quality of service for pods. https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod.
40	GitHub-source code. https://github.com/CloudControlSystems/OOM-Test.

Adaptive resource allocation for workflow containerization on Kubernetes

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 40

相关文章 0

编辑推荐

Metrics

本文评价