Steep - Run Scientific Workflows in the Cloud

For the last couple of years, I’ve been working on a scientific workflow management system called Steep. I’m more than happy to announce that, here at Fraunhofer IGD, we decided to make Steep open-source! You can download the binaries and the source code of Steep from its GitHub repository.

A scientific workflow management system is an application that can execute data-driven workflows, which apply a series of processing services (often called actions, tasks, or microservices) to input files to produce a certain output. Steep is designed to be scalable. It can run on your laptop but also in the Cloud, in a Grid, or a Cluster and distribute the individual processing services to multiple compute nodes. As such, it is very well suited to harness the possibilities of distributed computing in order to parallelise work and to speed up your data processing workflows, no matter how complex they are and regardless of how much data you need to process.

The following is a list of Steep’s key features:

Cyclic workflow graphs

In contrast to other scientific workflow management systems, Steep supports cyclic workflow graphs without a priori runtime knowledge. Workflows are converted incrementally and on-demand to so-called process chains. This allows Steep to execute workflows that dynamically change its structure during runtime.

Capability-based scheduling

Steep has an optimized, capability-based scheduler that executes process chains in parallel by distributing them to multiple agents (i.e. Steep instances running in the Cloud or in a Cluster).

Automatic failover

Crashed workflows can be resumed without loss of information—even if no database is configured.

Microservice integration

In order to be able to integrate and execute processing services or microservices with almost arbitrary interfaces, Steep makes use of so-called service metadata. You can use this metadata to describe the interface of your existing binaries without the need to modify them and to match them to a certain processing framework.

Runtime environments

Steep has built-in runtime environments for executable microservices that are provided as binaries or Docker images.

Plugins

Plugins let you modify generated process chains, the way agents collect results, or add custom runtime environments (e.g. Python, AWS Lambda, Web Processing Services).

Optional databases

If you want your workflows to be persisted in a database over a long period of time, Steep offers support for MongoDB and PostgreSQL. In the default configuration, Steep requires no database and keeps everything in memory. Note that, even in this setup, individual Steep instances in your cluster share the same memory, which already provides a good fault tolerance.

HTTP and web interfaces

Steep has a REST-like HTTP interface, a web-based user interface for monitoring, and can provide metrics to Prometheus.

Scalability

Its asynchronous event-driven architecture allows Steep to scale horizontally across multiple machines in your cluster and to support complex dynamic workflows with thousands of tasks.

Production-ready

Steep is very reliable and has been used in production for many years to execute workflows from various domains. The source code has a very high test coverage.

Open-Source

Did I mention Steep is free and Open-Source? It is released under the Apache License, Version 2.0. The code can be found in the GitHub repository.

Version 5.0.0

Today, we also released the new version 5.0.0 of Steep with the following new features:

• Process chains are now executed in the order they have been added to the registry.
• New capability-based scheduling algorithm (see below)
• Do not keep temporary process chain results in memory if not needed
• Allow users to disable scheduler
• Do not log errors while trying to connect to new VM via SSH
• Add possibility to specify a minimum number of VMs per setup
• Allow alternative setups with similar provided capabilities to be specified
• New process chain adapter plugins

One of the highlights, is the new capability-based scheduling algorithm. With the old algorithm, workflow execution could stall if there was a process chain that could not be executed because of a missing agent, even if there were other agents that would have been able to execute the remaining process chains. The new algorithm executes process chains in the order they were added to the registry and always fetches as many of them as possible from the registry if there are enough agents available that can execute them. Process chains that cannot be executed because they require capabilities none of the agents can provide are skipped but resumed immediately as soon as an agent with matching capabilities joins the cluster.

Publications

Steep was initially developed within the research project “IQmulus” (A High-volume Fusion and Analysis Platform for Geospatial Point Clouds, Coverages and Volumetric Data Sets) funded from the 7^th Framework Programme of the European Commission, call identifier FP7-ICT-2011-8, under the Grant agreement no. 318787 from 2012 to 2016. It was previously called the ‘IQmulus JobManager’ or just the ‘JobManager’. Under this name, it has appeared in at least the following publications:

Krämer, M. (2018). A Microservice Architecture for the Processing of Large Geospatial Data in the Cloud (Doctoral dissertation). Technische Universität Darmstadt. https://doi.org/10.13140/RG.2.2.30034.66248

[ BibTeX | Abstract | Source code | PDF ]

Böhm, J., Bredif, M., Gierlinger, T., Krämer, M., Lindenbergh, R., Liu, K., … Sirmacek, B. (2016). The IQmulus Urban Showcase: Automatic Tree Classification and Identification in Huge Mobile Mapping Point Clouds. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLI-B3, 301–307. https://doi.org/10.5194/isprs-archives-XLI-B3-301-2016

[ BibTeX | Abstract | PDF ]

Krämer, M., & Senner, J. (2015). A modular software architecture for processing of big geospatial data in the cloud. Computers & Graphics, 49, 69–81. https://doi.org/10.1016/j.cag.2015.02.005

[ BibTeX | Abstract ]

Documentation and getting started

I’m currently working on a comprehensive web page describing the features of Steep and how you can execute workflows with it. I will keep you updated here on this site and let you know when the web page is ready.

Posted by Michel Krämer
on 6 February 2020