Efficient scheduling of workflow actions in the cloud

My paper “Efficient Scheduling of Scientific Workflow Actions in the Cloud Based on Required Capabilities” has just been published in Springer’s Communications in Computer and Information Science book series (CCIS).

Distributed scientific workflow management systems processing large data sets in the Cloud face the following challenges: (a) workflow tasks require different capabilities from the machines on which they run, but at the same time, the infrastructure is highly heterogeneous, (b) the environment is dynamic and new resources can be added and removed at any time, (c) scientific workflows can become very large with hundreds of thousands of tasks, (d) faults can happen at any time in a distributed system.

In this paper, I present a software architecture and a capability-based scheduling algorithm that cover all these challenges in one design. My architecture consists of loosely coupled components that can run on separate virtual machines and communicate with each other over an event bus and through a database. The scheduling algorithm matches capabilities required by the tasks (e.g. software, CPU power, main memory, graphics processing unit) with those offered by the available virtual machines and assigns them accordingly for processing. My approach utilises heuristics to distribute the tasks evenly in the Cloud. This reduces the overall run time of workflows and makes efficient use of available resources. My scheduling algorithm also implements optimisations to achieve a high scalability. I perform a thorough evaluation based on four experiments and test if my approach meets the challenges mentioned above.

The paper finishes with a discussion, conclusions, and future research opportunities. An implementation of my algorithm and software architecture is publicly available with the open-source workflow management system Steep.

Reference

Krämer, M. (2021). Efficient Scheduling of Scientific Workflow Actions in the Cloud Based on Required Capabilities. In S. Hammoudi, C. Quix, & J. Bernardino (Eds.), Data Management Technologies and Applications. Communications in Computer and Information Science (Vol. 1446, pp. 32–55). Springer. https://doi.org/10.1007/978-3-030-83014-4_2

[ BibTeX | PDF ]

Download

According to Springer’s self-archiving policy, you may download the manuscript pre-print here. The final authenticated version is available on the publisher’s website.

Posted by Michel Krämer
on 23 July 2021