doit comes from the idea of bringing the power of build-tools to execute any kind of task


python logo 
pip install doit

People often compare doit to tools like make, grunt, rake, scons, snakemake.

They appreciate doit strong features, flexibility, simplicity of authoring and ease of use.

A game developer uses doit to

  • Automate all project related tasks (code generation, cross-compilation, resource generation)
  • Simplify cumbersome command line calls
  • Optimize processing time by skipping tasks already done

A bioinformatics developer uses doit to

  • Create a reproduceable computational pipeline
  • Manage a complex workflow (set of depending tasks)
  • Optimize processing time by skipping tasks already done

Nikola, a static site generator, uses doit to

  • Provide the command line interface
  • Speed up by parallel task execution
  • Optimize processing time by skipping tasks already done
View all use-cases

Sample Code

Define functions returning python dict with task's meta-data.

Snippet from tutorial.

def task_imports():
    """find imports from a python module"""
    for name, module in PKG_MODULES.by_name.items():
        yield {
            'name': name,
            'file_dep': [module.path],
            'actions': [(get_imports, (PKG_MODULES, module.path))],
        }

def task_dot():
    """generate a graphviz's dot graph from module imports"""
    return {
        'targets': ['requests.dot'],
        'actions': [module_to_dot],
        'getargs': {'imports': ('imports', 'modules')},
        'clean': True,
    }

def task_draw():
    """generate image from a dot file"""
    return {
        'file_dep': ['requests.dot'],
        'targets': ['requests.png'],
        'actions': ['dot -Tpng %(dependencies)s -o %(targets)s'],
        'clean': True,
    }

$ doit list
dot       generate a graphviz's dot graph from module imports
draw      generate image from a dot file
imports   find imports from a python module
$ doit
.  imports:requests.models
.  imports:requests.__init__
.  imports:requests.help
(...)
.  dot
.  draw


Task Runner

doit allows you to easily define ad-hoc tasks, helping you to organize all your project related tasks in an unified easy-to-use & discoverable way.

python powered

doit uses plain python to define tasks.

Task's meta-data are better described in a declarative way, but often you want to create this meta-data programmatically.

easy authoring

NO API: Tasks are described by a python dict (can also be easily customized)

Tasks can execute external process (shell commands) or python code

debugger & self documented

Since plain python is used to define your tasks the python debugger pdb is available

doit command allows you to list and obtain help/documentation for tasks


Build tool & Pipelines

Simple task runners simply do not scale-up. doit as other build-tools can be much more efficient at repeateadly running tasks.

cache task results
aka incremental-builds

doit creates a DAG and ensures that only required tasks will be executed and in the correct order.

doit checks if the task is up-to-date and skips its execution if the task would produce the same result of a previous execution.

up-to-date check

dependencies can be dynamically calculated by other tasks

the up-to-date check to cache task results is not restricted to looking for file modification on dependencies. Nor requires target files.

Pipelines

Traditional build-tools were created mainly to deal with compile/link process of source code. doit was designed to solve a broader range of workflows.

results from a task can be used by another task without resorting to the creation of intermediate files


Advanced Features

Extensible

Custom output
Command line output can be completely customized through reporters

Plugins
allow you to create/modify sub-commands, storage backend, task loader, and output reporter

Framework
API is exposed so you can create new applications/tools leveraging doit functionality

Batteries included

parallel execution
built-in support for parallel (threaded or multi-process) task execution

watch/auto execution
built-in support watching for file changes and automatically re-execute tasks based on file changes by external process [linux/mac only]

tab-completion
built-in support tab-completion for commands/task (supports bash and zsh)

DAG Visualisation
create task's dependency-graph image using graphviz

IPython
IPython integration provide %doit magic function that loads tasks defined directly in IPython's global namespace

strace
Integration with strace helps you understand effects of third-part commands

Testimonials

  • Congratulations! Your tool follows the KISS principle very closely. I always wondered why build tools had to be that complicated.

    Elena

    source

  • Let me start by saying I'm really lovin doit, at first the interface seemed verbose but quickly changed my mind when I started using it and realized the flexibility. Many thanks for the great software!

    Michael Gliwinski

    source

  • I love all the traditional unix power tools, like cron, make, perl, ..., I also like new comprehensive configuration management tools like CFEngine and Puppet. But I find doit to be so versatile and so productive.

    Charlie Guo

    source

  • I needed a sort of make tool to glue things together and after trying out all kinds, doit ... has actually turned out to be beautiful. Its easy to add and manage tasks, even complex ones-- gluing things together with decorators and 'library' functions I've written to do certain similar things.

    Matthew

    source

  • I went back and forth on different Pythonic build tools for awhile. Scons is pretty great if you're doing 'standard' sorts of builds, but I found it a little heavy for my tastes and really hard to customize to my tool flow (in FPGA land, there are all kinds of nonstandard vendor tools that all need to play together). I've been using doit more and more over the past few months, and I'm continually impressed by the tool (aside from the goofy name). It works amazingly well for automating tricky/exotic build processes. Check it out!

    SkOink

    source

  • Some time ago, I grew frustrated with Make and Ant and started porting my build files to every build tool I found (SCons, Waf, etc.). Each time, as soon as I stepped out of already available rules, I ran into some difficult to overcome stumbling blocks. Then I discovered this little gem of simplicity: doit. It's Python-based. It doesn't try to be smart, it does not try to be cool, it just works. If you are looking for a flexible little build tool for different languages and tasks, give it a chance. (...)

    lelele

    source

Status

doit is under active development. Version 0.32 released on 2019-12.

doit runs on Python 3.4 through 3.6 (including PyPy). For python 2 support please use doit version 0.29.

This blog post explains how everything started in 2008.

doit core features are quite stable. If there is no recent development, it does NOT mean the project is not being maintained... The project has 100% unit-test code coverage.

Development is done based on real world use cases. It is well designed and has a small code base, so adding new features is not hard. Contributions are welcome.

Project Details

LICENSE
This is an open-source project (MIT license) written in python.
DOWNLOAD
from PyPi
SUPPORT
See support page
DEVELOP
Project management (bug tracker, feature requests and source code ) on github
WEBSITE
This web site is hosted on http://pages.github.com
Powered by Universal template and Sphinx.