Course Catalog
Notebook Driven Development
Workflow Automation and Multi-Kernel Support in Jupyter
Dependency Management with PyDoit

Dependency Management with PyDoit

Authors
Dr. Sangeetha Nandakumar | Dr. Nicholas Del Grosso
Download Materials

Setup

Import Libraries 

from doit import load_ipython_extension
load_ipython_extension()

Download Data 

import os
import owncloud

# Ensure the 'data' directory exists
if not os.path.exists('data'):
    print('Creating directory for data')
    os.mkdir('data')

# Ensure the 'notebooks' directory exists
if not os.path.exists('nb_active_trials.ipynb'):
    print('Creating directory for notebooks')
    os.mkdir('notebooks')

# Download steinmetz_all.csv
if not os.path.exists('data/steinmetz_all.csv'):
    oc = owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/CE4fcAAr3HE4naE', folder_password="ibots"
)
    oc.get_file('/', 'data/steinmetz_all.csv')

# Download nb_active_trials.ipynb
if not os.path.exists('nb_active_trials.ipynb'):
    oc = owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/pJY34rKDq29E6EC', folder_password="ibots"
)
    oc.get_file('/', 'nb_active_trials.ipynb')

# Download nb_stats.ipynb
if not os.path.exists('nb_stats.ipynb'):
    oc = owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/X8FjRFFmTRNa7zj', folder_password="ibots"
)
    oc.get_file('/', 'nb_stats.ipynb')

# Download nb_plots.ipynb
if not os.path.exists('nb_plots.ipynb'):
    oc = owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/gJ8rPRtKqMQXfZg', folder_password="ibots"
)
    oc.get_file('/', 'nb_plots.ipynb')

In Pydoit, dependency management is a core feature that ensures tasks are executed only when necessary. Dependencies can be files, task results, or parameters, and Pydoit will automatically skip tasks if none of their dependencies have changed.

Section 1: File Dependency

In pydoit, file dependencies (file_dep) specify which files a task depends on to complete its action. A task will be executed only if one or more of its dependencies have changed since the last time the task ran. If none of the files have been modified, pydoit will skip the task.

def task_name():
    return {
        'actions': ['command to execute'],
        'file_dep': ['file_name']
    }

Exercises

Example: Make process depend on file data/steinmetz_all.csv and run %doit.

def task_process():
    return {
        'actions': ['papermill nb_active_trials.ipynb process.ipynb']
    }
def task_process():
    return {
        'actions': ['papermill nb_active_trials.ipynb process.ipynb'],
        'file_dep': ['data/steinmetz_all.csv']
    }
%doit process

Exercise: Run process again. What difference do you notice?

Solution 
%doit process

Example: Make stats depend on file data/active_trials.csv and run %doit.

def task_stats():
    return {
        'actions': ['papermill nb_stats.ipynb stats.ipynb']
    }
Solution 
def task_stats():
    return {
        'actions': ['papermill nb_stats.ipynb stats.ipynb'],
        'file_dep': ['data/active_trials.csv']
    }
%doit stats

Exercise: Run stats again. What do you see?

Solution 
%doit stats

Exercise: Make plot depend on file data/active_trials.csv and run %doit.

def task_plot():  
    return {
        'actions': ['papermill nb_plots.ipynb plots.ipynb']
    }
Solution 
def task_plot():  
    return {
        'actions': ['papermill nb_plots.ipynb plots.ipynb'],
        'file_dep': ['data/active_trials.csv']
    }
%doit plot

Exercise: Run %doit again. What do you see?

Solution 
%doit plot

Section 2: Targets

In pydoit, targets (targets) specify the output files that a task is expected to create. The task will only run if the target files are missing or outdated based on file dependencies. Here’s how you can define the targets for each of your tasks

def task_name():
    return {
        'actions': ['command to execute'],
        'file_dep': ['file_name'],
        'targets': ['target_file_name']
    }

Exercises

Example: Make process task depend on its output data/active_trials.csv

def task_process():
    return {
        'actions': ['papermill nb_active_trials.ipynb process.ipynb'],
        'file_dep': ['data/steinmetz_all.csv'],  
        'targets': ['data/active_trials.csv']    
    }
%doit process

Exercise: Run process task again. Does it run?

Solution 
%doit process

Delete active_trials.csv file only which is the target of process task and run it again. Does it run now?

%doit process

Exercise: Make stats task depend on its output data/stats.csv. What do you see is happening here?

Solution 
def task_stats():
    return {
        'actions': ['papermill nb_stats.ipynb stats.ipynb'],
        'file_dep': ['data/active_trials.csv'],
        'targets': ['data/stats.csv']
    }
%doit stats

Exercise: Run stats task again. What do you see?

Solution 
%doit stats

Exercise: Delete stats.csv and run stats task again.

Solution 
%doit stats

Exercise: Delete active_trials.csv and run stats process. What happened now?

Solution 
%doit stats

Exercise: Make plot task depend on its output response_time_histogram.png.

Solution 
def task_plot():  
    return {
        'actions': ['papermill nb_plots.ipynb plots.ipynb'],
        'file_dep': ['data/active_trials.csv'],
        'targets': ['response_time_histogram.png']
    }
%doit plot

Exercise: Delete response_time_histogram.png and run plot task again.

Solution 
%doit plot

Section 3: Adding papermill output notebooks as targets

We can also add the output notebooks from papermill executions as targets since they are generated by the tasks.

Exercises

Example: Make process depend on its output process.ipynb and data/active_trials.csv

def task_process():
    return {
        'actions': ['papermill nb_active_trials.ipynb process.ipynb'],
        'file_dep': ['data/steinmetz_all.csv'],  
        'targets': ['data/active_trials.csv', 'process.ipynb']    
    }

Exercise: Make stats depend on its output stats.ipynb and data/stats.csv

Solution 
def task_stats():
    return {
        'actions': ['papermill nb_stats.ipynb stats.ipynb'],
        'file_dep': ['data/active_trials.csv'],
        'targets': ['data/stats.csv', 'stats.ipynb']
    }

Exercise: Make stats depend on its output plots.ipynb and response_time_histogram.png

Solution 
def task_plot():  
    return {
        'actions': ['papermill nb_plots.ipynb plots.ipynb'],
        'file_dep': ['data/active_trials.csv'],
        'targets': ['response_time_histogram.png', 'plots.ipynb']
    }