GinFlow is a decentralised adaptive workflow engine.

Decentralisation. GinFlow decentralises the coordination of the execution of workflow-based applications, using an architecture of multiple workflow engines, or service agents (SA), coordinating themselves in a P2P fashion. Initially in a shared space, the workflow description is sent to the SAs that then communicate directly during the execution. More information about the architecture here.

Adaptivity. Designing a workflow is not a one-step process. The need to go back and forth between design and testing is present in many science domains where the uncertainty is part of the workflow design process. This requires to develop ways for the workflow designer to define variants for a workflow, a variant being triggered if the initial workflow does not provide satisfying results, typically intermediate output data showing the workflow does not reach the goal of the targeted application. Then, the engine needs to be able to switch between variants dynamically. GinFlow allows this.

API. GinFlow exposes a high-level API to let the workflow designer specify a workflow and lets you run it over various distributed environments. The workflow designer is provided three ways to specify a workflow (and its variants): either through the programming Java API, a JSON description or by using the GinFlow GUI.

Deployment. GinFlow supports different workflow executors (component responsible to deploy the SAs):

•  The centralized executor runs your workflow using a centralized engine
•  The SSH-based executor runs your workflow in a distributed fashion using a set of distributed workflow co-engines. It simply relies on a SSH  to connect and launch agents on a set of worker nodes which will execute the different tasks of the workflow.
•  The Mesos-based executor submits your workflow to a dedictated Mesos Framework and uses Mesos primitives to control the agents.

Commication. The communication between SAs relies on a persistent communication middleware. Currently, GinFlow supports two  message brokers (to be chosen based on the user’s preferences):

• ActiveMQ (the default broker)
• Kafka + Zookeeper (which provides state recovery of an agent in case of a failure.)

Test it

• online using allgo
• download the latest version from the download page

More information

• go to the publications page