Executing cyclic scientific workflows in the cloud

Our pa­per with the title “Ex­ecut­ing cyc­lic sci­entific work­flows in the cloud” has just been re­leased in Spring­er’s Journal of Cloud Com­put­ing. In this pa­per, we present an al­gorithm and a soft­ware ar­chi­tec­ture for a cloud-based sys­tem that ex­ecutes cyc­lic sci­entific work­flows whose struc­ture may change dur­ing run time.

Ex­ist­ing ap­proaches either rely on work­flow defin­i­tions based on dir­ec­ted acyc­lic graphs (DAGs) or re­quire work­arounds to im­ple­ment cyc­lic struc­tures. In con­trast, our sys­tem sup­ports cycles nat­ively, avoids work­arounds, and as such re­duces the com­plex­ity of work­flow mod­el­ling and main­ten­ance. Our al­gorithm tra­verses work­flow graphs and trans­forms them it­er­at­ively into lin­ear se­quences of ex­ecut­able ac­tions. We call these se­quences pro­cess chains. Our soft­ware ar­chi­tec­ture dis­trib­utes the pro­cess chains to mul­tiple com­pute nodes in the cloud and over­sees their ex­e­cu­tion.

We eval­u­ate our ap­proach by ap­ply­ing it to two prac­tical use cases from the do­mains of as­tro­nomy and en­gin­eer­ing. We also com­pare it with two ex­ist­ing work­flow man­age­ment sys­tems. The eval­u­ation demon­strates that our al­gorithm is able to ex­ecute dy­nam­ic­ally chan­ging work­flows with cycles and that design and main­ten­ance of com­plex work­flows is easier than with ex­ist­ing solu­tions. It also shows that our soft­ware ar­chi­tec­ture can run pro­cess chains on mul­tiple com­pute nodes in par­al­lel to sig­ni­fic­antly speed up the work­flow ex­e­cu­tion.

An im­ple­ment­a­tion of our al­gorithm and the soft­ware ar­chi­tec­ture is avail­able with the Steep Work­flow Man­age­ment Sys­tem that we re­leased un­der an open-source li­cense. The re­sources for the first prac­tical use case are also avail­able as open source for re­pro­duc­tion.

Reference

Krämer, M., Würz, H. M., & Al­ten­hofen, C. (2021). Ex­ecut­ing cyc­lic sci­entific work­flows in the cloud. Journal of Cloud Com­put­ing, 10(25), 1–26. ht­tps://​doi.org/​10.1186/​s13677-021-00229-7

Download

The pa­per has been pub­lished un­der the CC-BY 4.0 li­cense. You may down­load the fi­nal manuscript here.


Posted by Michel Krämer
on April, 7th 2021.