Course Catalog
Notebook Driven Development
Workflow Automation and Multi-Kernel Support in Jupyter
Building Workflows with Notebooks

Building Workflows with Notebooks

Authors
Dr. Sangeetha Nandakumar | Dr. Nicholas Del Grosso
Download Materials

Setup

Import Libraries 

from doit import load_ipython_extension
load_ipython_extension()
import pandas as pd
import hvplot.pandas

Download Data 

import os
import owncloud

# Ensure the 'data' directory exists
if not os.path.exists('data'):
    print('Creating directory for data')
    os.mkdir('data')

# Download steinmetz_all.csv
if not os.path.exists('data/steinmetz_all.csv'):
    oc = owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/CE4fcAAr3HE4naE', folder_password="ibots"
)
    oc.get_file('/', 'data/steinmetz_all.csv')

# Ensure the directory exists for the notebooks
# Download nb_active_trials.ipynb
if not os.path.exists('nb_active_trials.ipynb'):
    oc = owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/pJY34rKDq29E6EC', folder_password="ibots"
)
    oc.get_file('/', 'nb_active_trials.ipynb')

# Download nb_stats.ipynb
if not os.path.exists('nb_stats.ipynb'):
    oc = owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/X8FjRFFmTRNa7zj', folder_password="ibots"
)
    oc.get_file('/', 'nb_stats.ipynb')

# Download nb_plots.ipynb
if not os.path.exists('nb_plots.ipynb'):
    oc = owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/gJ8rPRtKqMQXfZg', folder_password="ibots"
)
    oc.get_file('/', 'nb_plots.ipynb')

Pydoit is a task management and automation tool designed to execute commands and scripts in a structured, reproducible way. In this notebook, we will understand the structure of doit workflows by writing doit tasks and execute them.

Section 1: Functions

Functions are reusable blocks of code that perform a specific task. They help organize code, make it more readable, and allow you to avoid repetition by encapsulating logic that can be called multiple times throughout a program.

A function is defined using the def keyword followed by the function name and parentheses (). The code that performs the task is placed inside the function body, indented under the function definition

def function_name(param1, param2, ...):  
    # use param1, param2, ...
    return value  # return (optional)

Exercises

Example: Make a function called active_trials the replaces the below code

input_csv = 'data/steinmetz_all.csv'
output_csv = 'data/active_trials.csv'
df = pd.read_csv(input_csv)
df_active = df[df['active_trials'] == True]
df_active.to_csv(output_csv, index=False)
Solution 
def active_trials():
    input_csv = 'data/steinmetz_all.csv'
    output_csv = 'data/active_trials.csv'
    df = pd.read_csv(input_csv)
    df_active = df[df['active_trials'] == True]
    df_active.to_csv(output_csv, index=False)    

active_trials()

Exercise: Make a function called descriptive_stats that replaces the below code

active_trials_csv = 'data/active_trials.csv'
stats_csv = 'data/stats.csv'
df = pd.read_csv(active_trials_csv)
df_stats = df.describe().reset_index()
df_stats.to_csv(stats_csv, index=False)
Solution 
def descriptive_stats():
    active_trials_csv = 'data/active_trials.csv'
    stats_csv = 'data/stats.csv'
    df = pd.read_csv(active_trials_csv)
    df_stats = df.describe().reset_index()
    df_stats.to_csv(stats_csv, index=False)

descriptive_stats()

Exercise: Make a function called histogram_plot that replaces the code below

import matplotlib.pyplot as plt
active_trials_csv = 'data/active_trials.csv'
hist_col_name = 'response_time'
df = pd.read_csv(active_trials_csv)
df[hist_col_name].plot.hist()
plt.savefig(f'{hist_col_name}_histogram.png')
Solution 
def histogram_plot():
    import matplotlib.pyplot as plt
    active_trials_csv = 'data/active_trials.csv'
    hist_col_name = 'response_time'
    df = pd.read_csv(active_trials_csv)
    df[hist_col_name].plot.hist()
    plt.savefig(f'{hist_col_name}_histogram.png')

histogram_plot()

Exercise: Make input_csv as a parameter for active_trials

Solution 
def active_trials(input_csv):
    output_csv = 'data/active_trials.csv'
    df = pd.read_csv(input_csv)
    df_active = df[df['active_trials'] == True]
    df_active.to_csv(output_csv, index=False)    

active_trials('data/steinmetz_all.csv')

Exercise: Make active_trials_csv as a parameter of descriptive_stats

Solution 
def descriptive_stats(active_trials_csv):
    stats_csv = 'data/stats.csv'
    df = pd.read_csv(active_trials_csv)
    df_stats = df.describe().reset_index()
    df_stats.to_csv(stats_csv, index=False)

descriptive_stats('data/active_trials.csv')

Exercise: Make active_trials_csv as a parameter of histogram_plot

Solution 
def histogram_plot(active_trials_csv):
    import matplotlib.pyplot as plt
    hist_col_name = 'response_time'
    df = pd.read_csv(active_trials_csv)
    df[hist_col_name].plot.hist()
    plt.savefig(f'{hist_col_name}_histogram.png')

histogram_plot('data/active_trials.csv')

Example: Make input_csv and output_csv as parameters for active_trials

def active_trials(input_csv, output_csv):
    df = pd.read_csv(input_csv)
    df_active = df[df['active_trials'] == True]
    df_active.to_csv(output_csv, index=False)    

active_trials('data/steinmetz_all.csv', 'data/active_trials.csv')

Exercise: Make active_trials_csv and stats_csv as parameters of descriptive_stats

Solution 
def descriptive_stats(active_trials_csv, stats_csv):
    df = pd.read_csv(active_trials_csv)
    df_stats = df.describe().reset_index()
    df_stats.to_csv(stats_csv, index=False)

descriptive_stats('data/active_trials.csv', 'data/stats.csv')

Exercise: Make active_trials_csv and stats_csv as parameters of histogram_plot

Solution 
def histogram_plot(active_trials_csv, hist_col_name):
    import matplotlib.pyplot as plt
    df = pd.read_csv(active_trials_csv)
    df[hist_col_name].plot.hist()
    plt.savefig(f'{hist_col_name}_histogram.png')

histogram_plot('data/active_trials.csv', 'response_time')

Section 2: Building doit Workflows

doit is a task automation tool that helps manage dependencies and execute tasks. doit workflows are made up of units called tasks which are python functions.

Basic syntax of doit task:

def task_name():
    return {
        'actions': ['command to execute'],
    }

task_name is name of the task (must start with task_)

actions is a list of commands, Python functions, etc.

Let’s get some practice building doit tasks.

Exercises

Example: Add a doit task called process that implements the below code:

def active_trials():
    input_csv = 'data/steinmetz_all.csv'
    output_csv = 'data/active_trials.csv'
    df = pd.read_csv(input_csv)
    df_active = df[df['active_trials'] == True]
    df_active.to_csv(output_csv, index=False)   
def task_process():
    def active_trials():
        input_csv = 'data/steinmetz_all.csv'
        output_csv = 'data/active_trials.csv'
        df = pd.read_csv(input_csv)
        df_active = df[df['active_trials'] == True]
        df_active.to_csv(output_csv, index=False)       

    return {
        'actions': [active_trials]
    }
%doit list
process

Exercise: Add a doit task called stats for the below code

def task_stats():
    def descriptive_stats():
        active_trials_csv = 'data/active_trials.csv'
        stats_csv = 'data/stats.csv'
        df = pd.read_csv(active_trials_csv)
        df_stats = df.describe().reset_index()
        df_stats.to_csv(stats_csv, index=False)
    
    return {
        'actions': [descriptive_stats]
    }
Solution 
%doit list
process   
stats

Exercise: Add a doit task called plot for the below code

def histogram_plot():
    import matplotlib.pyplot as plt
    active_trials_csv = 'data/active_trials.csv'
    hist_col_name = 'response_time'
    df = pd.read_csv(active_trials_csv)
    df[hist_col_name].plot.hist()
    plt.savefig(f'{hist_col_name}_histogram.png')
Solution 
def task_plot():
    def histogram_plot():
        import matplotlib.pyplot as plt
        active_trials_csv = 'data/active_trials.csv'
        hist_col_name = 'response_time'
        df = pd.read_csv(active_trials_csv)
        df[hist_col_name].plot.hist()
        plt.savefig(f'{hist_col_name}_histogram.png')    
    
    return {
        'actions': [histogram_plot]
    }
%doit list
plot      
process   
stats

We can also run notebooks as a doit task. Run the below cell to download notebooks.

nb_active_trials: Same as active_trials function
nb_stats: Same as descriptive_stats function
nb_plots: Same as histogram_plot function

Example: Run nb_active_trials notebook inside process task.

def task_process():
    return {
        'actions': ['papermill nb_active_trials.ipynb process.ipynb']
    }

Exercise: Run nb_stats notebook inside stats task.

Solution 
def task_stats():
    return {
        'actions': ['papermill nb_stats.ipynb stats.ipynb']
    }

Exercise: Run nb_plots notebook inside plot task

Solution 
def task_plot():  
    return {
        'actions': ['papermill nb_plots.ipynb plots.ipynb']
    }

Section 3: Running doit tasks

Now that we know how to make doit tasks, let see how to run them.

Delete data directory and run the below cell to get only steinmetz_all.csv

Exercises

Example: Run process task

%doit process

Exercise: Run stats task

Solution 
%doit stats

Exercise: Run plot task

Solution 
%doit plot
.  plot

doit does not enforce a specific order of task execution unless task dependencies are explicitly defined. By default, tasks are executed in parallel or in whatever order doit chooses, which might not match the logical order you expect.

Delete data directory and run the below cell to get only steinmetz_all.csv

Example: Run everything (Can you?)

%doit

Example: Specify that plot task depends on process task

def task_plot():  
    return {
        'actions': ['papermill nb_plots.ipynb plots.ipynb'],
        'task_dep': ['process']

    }
%doit

Exercise: Specify that stats task depends on process task

Solution 
def task_stats():
    return {
        'actions': ['papermill nb_stats.ipynb stats.ipynb'],
        'task_dep': ['process']
    }
%doit