Makeflow

Makeflow is an open source workflow engine that can be uses to execute advanced workflows on clouds, grids and clusters.

Makeflow includes a Work Queue system and has support for the Condor and SGE distributed computing systems. It doesn't require a distributed file system.

What is new in this release:

• Support for workers hierarchies, with a master-foremen-workers paradigm.
• Additionally, a worker now is able to handle more than one task at a time.
• Support has been added for task categories.
• A rule can be labeled with a category, and required computational resources (disk, memory, and CPU) can be specified per category.
• Makeflow then automatically communicates these requirements to the work queue or condor.

What is new in version 3.7.0:

• Work Queue now allows you to specify chunks (pieces) of an input file to be used as input for worker tasks.
• Makeflow now supports a -i switch which outputs pre-execution analysis of the Makeflow DAG.
• Work Queue and Makeflow support submitting tasks to the PBS batch submission platform.
• The Makeflow makeflow_log_parser now ignores comments in Makeflow logs.
• Work Queue supports querying workers and tasks at run-time.
• Makeflow Input file (dependency) renaming is supported with a new "->" operator.
• Work Queue Tasks are now killed using SIGKILL.
• Work Queue Protocol based keep-alives were added to workers