Task Performance Analysis with Pandas and Seaborn

In the experiment reported in Steinmetz et al., 2019 in Nature, mice were tasked with turning a wheel to the left or to the right based on the relative contrast levels of two simultaneously-presented gradient stimuli:

In this notebook, we’ll examine the response_time and response_type of each trial across all sessions, to determine whether the mice successfully performed the task, and whether the difference in contrast levels between the two stimuli affected their performance.

We’ll use this as an opportunity to get to know the Seaborn Python package’s core syntax. Seaborn is a statistical plotting library that takes Pandas Dataframes and turns them into plots (including making errorbars)!

Setup

Import Libraries

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from pathlib import Path

This data has been pre-processed from Steinmetz and al 2019, and is hosted on Sciebo here: https://uni-bonn.sciebo.de/s/wjsBtZzUVjKaB3J. The code below should download it to the folder data/steinmetz_all.csv

Download Data

import owncloud

Path('data').mkdir(exist_ok=True, parents=True)

owncloud.Client.from_public_link('https://uni-bonn.sciebo.de/s/wjsBtZzUVjKaB3J').get_file('/', f'data/steinmetz_all.csv')

True

Section 1: Analyzing Behavioral Task Performance with Pandas and Seaborn: Psychometric Analysis on Ordered Categorical Data

Load Data: Run the cell below where the pd.read_csv() function is used to load the steinmetz_all.csv file in the data folder into a Pandas DataFrame:

Exercises

df = pd.read_csv('data/steinmetz_all.csv')
df.head()

Note that our CSV file has this column called "Unnamed: 0", which is not really wanted. We can drop it using the .drop method of the Pandas DataFrame:

df = df.drop(columns="Unnamed: 0")
df.head()

Performance Analysis: Average Response Type for Each Stimulus Contrast Level

How did the mice perform in the task, overall? Let’s use seaborn to make a basic statistical analysis comparing different variables against each other.

What makes Seaborn particularly nice is that most of its functions have the same syntax:

sns.typeofplot(       #  Function name (what type of plot do you want to make?
    data=df,          #  Dataframe variable (what data will this plot be made from?)
    x="column1",      #  Column to use for the x axis of the plot.
    y="column2",      #  Column to use for the y axis of the plot.
    hue="column3",    #  Column to use for splitting the data into different colors.
    ...               # ...more columns can be added, to make a richer and more complex plot!
)

In this notebook, we’ll look at the plots that help compare a continuous variable across different levels of a categorical variable:

On average, how did the subjects respond when presented with various contrast levels on the right and left stimuli? For each exercise, we’ll use seaborn to make the requested plot.

Bar Plots

Example: Make a bar plot with contrast_right levels on the x-axis and response_type on the y-axis. What do you observe?

sns.barplot(data=df, x='contrast_right', y='response_type');

Exercise: Make a bar plot with contrast_left levels on the x-axis and response_type on the y-axis. What do you observe?

Solution

sns.barplot(data=df, x="contrast_left", y="response_type")

Exercise: Let’s create the same plot but “transposed”: such that we have the bars horizontally instead of vertically.

Make a bar plot with response_type as the x-axis and contrast_left levels on the y-axis. What does seaborn do here? Is this what we want? (note: to fix this, you can add the argument orient='h'. By default, seaborn has orient='v')

Solution

sns.barplot(data=df, x="response_type", y="contrast_left", orient="h")

Exercise: Let’s combine the x-axis and the hue together! Make a bar plot with contrast_left levels on both the x-axis and hue and response_type on the y-axis. (Note: If you don’t like having the legend anymore, you can additionally set legend=False.)

Solution

sns.barplot(data=df, x="contrast_left", y="response_type", hue="contrast_left", legend=False)

Exercise: Make a bar plot with contrast_left levels on the x-axis and response_type as the hue (i.e. the color of the bars). No need to specify anything for the y-axis. What do you observe?

Solution

sns.barplot(data=df, x="contrast_left", hue="response_type")

Exercise: How does the response_time look like for different combinations of left_contrast and right_contrast? Make a bar plot with contrast_left levels on the x-axis, contrast_right on the hue, and response_type on the y-axis.

Solution

sns.barplot(data=df, x="contrast_left", y="response_type", hue="contrast_right")

Point Plots

Example: Make a point plot with contrast_right on the x-axis and response_type on the y-axis. What do you observe?

sns.pointplot(data=df, x='contrast_right', y='response_type');

Exercise: Make a point plot with contrast_left on the x-axis and response_type on the y-axis. What do you observe?

Solution

sns.pointplot(data=df, x='contrast_left', y='response_type');

Exercise: Call the sns.pointplot() function twice (on two seperate lines), first with contrast_left on the x-axis and then with contrast_right on the x-axis. What do you observe? What’s good and bad about this plot? Hint: Look at the axis labels. And which line corresponds to which contrast data?

Solution

sns.pointplot(data=df, x='contrast_right', y='response_type');
sns.pointplot(data=df, x='contrast_left', y='response_type');

Exercise: How does the left_contrast values compare with the right_contrast values? Make a point plot with contrast_left levels on the x-axis, contrast_right on the hue, and response_type on the y-axis. What do you observe? (Note: to shift values a bit to keep errorbars from obscuring each other, try setting dodge=True)

Solution

sns.pointplot(data=df, x='contrast_left', y='response_type', hue="contrast_right", dodge=True);

Section 2: Interpreting and Setting Error Bar Types: Confidence Interval vs Standard Deviation vs Standard Error

Bar plots and point plots show “point estimates” of a variable; also known as “aggregations” or “descriptive statistics”. For example, the mean and median are point estimates; they take a range of data and estimate its center at a single point.

Error bars are essential because they tell us either the variation in the data underlying that estimate, or alternatively the quality of the estimate (by showing the range of uncertainty underlying it). Here are the four most common types of errorbars and their purposes:

Seaborn by default calculates the mean of the data, and shows its uncertainty using a 95% Confidence Interval. This is a good default, but it can be changed to other settings. Let’s get a feel for these error bar types!

In each of the following exercises, explore the errorbars by re-making the last plot in the previous section, setting the errorbars to different types and levels:

Exercises

Example: Make 95% confidence intervals.

fig, axes = plt.subplots(ncols = 2, figsize = (15,6))

sns.pointplot(data=df, ax = axes[0], x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('ci', 95));

axes[0].set_title('Confidence interval')

sns.pointplot(data=df, ax = axes[1], x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('se', 1.96));

axes[1].set_title('Std. error')

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('ci', 95));

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('se', 1.96));

Exercise: Make 0% confidence intervals. What do you expect to see?

Solution

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('ci', 0));

Exercise: Make 25% confidence intervals. What do you expect to see?

Solution

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('ci', 25));

Exercise: Make 99% confidence intervals. What do you expect to see?

Solution

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('ci', 99));

Exercise: Set the error bars to show one standard deviation (sd) of the data. What do you notice about these errorbars, compared to the ones made previously?

Solution

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('sd', 1));

Exercise: Set the error bars to show three standard deviations of the data (roughly the full range of data’s variation in the dataset, assuming normally-distributed data).

Solution

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('sd', 3));

Exercise: Account for the number of observations by setting the error bars to show one “Standard Error of the Mean” (se).

Solution

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('se', 1));

Exercise: Set the error bars to show two Standard Errors of the Mean, and compare this to the plot showing a 95% Confidence Interval. What do you notice?

Solution

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('se', 2));

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('ci', 95));

Exercise: Set the error bars to show 50% Intervals (pi). What is strange about this plot? Try setting it to different levels, and see what you get. What does this indicate about the data?

Solution

sns.pointplot(data=df, x='contrast_left', hue='contrast_right', y='response_type', dodge=True, errorbar=('pi', 50));

Section 3: Analyzing Response Time and Response Type: Transforming and Filtering Data, and Comparing Estimates

Making New Columns in Pandas DataFrames

In the exercises below, let’s practice making new columns in our Pandas DataFrame. We will then use the newly created column in the next set of analyses.

Exercise: How salient, overall, were the stimuli? Make a new column called "contrast_total" that is the sum of the two stimuli’s contrast levels.

df['contrast_total'] = df['contrast_right'] + df['contrast_left']
df.sample(3)

Exercise: The mice had to make a decision of whether to move the wheel left or right based on the difference in contrast between the left and right stimulus, not on the actual levels of the data. Make a new column called contrast_diff that contains the contrast difference between the left and right stimuli.

df["contrast_diff"] = df["contrast_left"] - df["contrast_right"]

Exercise: The mice were not allowed to respond immediately after the stimuli appeared; instead, they had to wait for the “go cue” to appear, which was at a randomized time point after the stimuli. Let’s calculate the a new response_time_corrected column, which subtracts the gocue_time from the response_time.

df['response_time_corrected'] = df['response_time'] - df['gocue_time']

Exercise: Instead of calculating the contrast_diff, let’s calculate the contrast_diff_absolute, where the absolute value of the contrast difference is used, so it is always positive. This more closely shows the decision that the mice had to make, and allows us to use twice as many values for each point estimation.

df['contrast_diff_absolute'] = df['contrast_diff'].abs()

Exercise: It’s only useful to calculate the response time when the mice actually made a response! Make a column called did_respond that is True when the response_type column is not equal to 0.

df['did_respond'] = df['response_type'] != 0

Section 4: Plotting Response Times and Response Types: Comparing Different Point Estimate Functions

Estimators

Seaborn is a statistical plotting tool, so it gives you control over which statistics you want to plot! To change which statistic (called an “estimate” by Seaborn) is used, set the estimator parameter to what you need. Some common estimators like mean, median, and std are already built-in, so you can just write them as a string, but Seaborn will accept any statistics function you’d like it to use:

This has a big effect on the plots. Let’s try it out!

Multilevel Bootstrapping Error Bars Units

If you have a nested structure to your study where observations are grouped (for example, in this study it is made up of sessions), then you should set units= to the grouping column; this gets you more accurate estimates and errorbars that account for between-session variance.

Let’s see how the estimator used changes the information conveyed in a plot. For each of the two measurements (response type and response time), which estimators do you think are the most valuable for this experiment?

Exercise: For only the trials where the mouse responded, plot the mean response type for each stimulus contrast difference. Account for the differences between sessions by setting units to session_id.

Solution

sns.pointplot(data=df[df['did_respond']], x='contrast_diff', y='response_type', estimator='mean', units='session_id');

Exercise: Where did the response type vary the most? For only the trials where the mouse responded, plot the standard deviation (std) of the response types for each stimulus contrast difference. Account for the differences between sessions by setting units to session_id.

Solution

sns.pointplot(data=df[df['did_respond']], x='contrast_diff', y='response_type', estimator='std', units='session_id');

Exercise: Is there any relationship between how fast the mice responded and the difference in the contrast?

For only the trials where the mouse responded, plot the mean corrected response time for each absolute stimulus contrast difference, using color to show the results for each response type. Account for the differences between sessions by setting units to session_id.

Solution

sns.pointplot(data=df[df['did_respond']], x='contrast_diff_absolute', hue='response_type', y='response_time_corrected', dodge=True, estimator="mean", units='session_id');

Exercise: For only the trials where the mouse responded, plot the median corrected response time for each absolute stimulus contrast difference, using color to show the results for each response type. Account for the differences between sessions by setting units to session_id.

Solution

sns.pointplot(data=df[df['did_respond']], x='contrast_diff_absolute', hue='response_type', y='response_time_corrected', dodge=True, estimator="median", units='session_id');

Exercise: For only the trials where the mouse responded, see where there was the most variance in response time by plotting the standard deviation (std) of the corrected response times for each absolute stimulus contrast difference, using color to show the results for each response type. Account for the differences between sessions by setting units to session_id.

Solution

sns.pointplot(data=df[df['did_respond']], x='contrast_diff_absolute', hue='response_type', y='response_time_corrected', dodge=True, estimator="std", units='session_id');

Section 5: (Demo) Wrapping it Up: Exporting a Figure

Below is an example of how to export our two main results into a single figure with two subplots, then save it as a file (say, for a poster presentation).

Exercises

plt.style.available 

['Solarize_Light2',
 '_classic_test_patch',
 '_mpl-gallery',
 '_mpl-gallery-nogrid',
 'bmh',
 'classic',
 'dark_background',
 'fast',
 'fivethirtyeight',
 'ggplot',
 'grayscale',
 'petroff10',
 'seaborn-v0_8',
 'seaborn-v0_8-bright',
 'seaborn-v0_8-colorblind',
 'seaborn-v0_8-dark',
 'seaborn-v0_8-dark-palette',
 'seaborn-v0_8-darkgrid',
 'seaborn-v0_8-deep',
 'seaborn-v0_8-muted',
 'seaborn-v0_8-notebook',
 'seaborn-v0_8-paper',
 'seaborn-v0_8-pastel',
 'seaborn-v0_8-poster',
 'seaborn-v0_8-talk',
 'seaborn-v0_8-ticks',
 'seaborn-v0_8-white',
 'seaborn-v0_8-whitegrid',
 'tableau-colorblind10']

Example: Run the plot below to see it all put together, along with subplots and explicit titles and labels; feel free to modify the code (especially the style='ggplot' part!).

# Figure 1b
import matplotlib.pyplot as plt

with plt.style.context(style='ggplot', after_reset=True):

    plt.figure(figsize=(16, 7))  # changing the figure size is an quick-and-dirty way to change the font size

    # Subplot 1
    plt.subplot(1, 2, 1)

    df['contrast_diff'] = df['contrast_right'] - df['contrast_left']
    mask = df['response_type'] != 0
    sns.pointplot(data=df[mask], x='contrast_diff', y='response_type', color='dimgrey', units='session_id', n_boot=300);
    plt.title('Performance Reaches >80%\nAbove a 50% Difference in Contrast')
    plt.xlabel('Contrast Difference (%)')
    plt.ylabel("Response Type")

    # Subplot 2
    plt.subplot(1, 2, 2)

    mask = df['response_type'] != 0
    df['contrast_diff_absolute'] = df['contrast_diff'].abs()
    sns.pointplot(
        data=df[mask].astype({'response_type': 'category'}), 
        x='contrast_diff_absolute', y='response_time_corrected',
        hue='response_type', 
        units='session_id', 
        dodge=True,
        n_boot=300,
    );
    plt.title('Response Time Decreases as \n Stimulus Contrast Difference Increases', )
    plt.xlabel('Absolute Contrast Difference (%)')
    plt.ylabel("Response Time after Go Cue")
    plt.legend(title="Response Type", frameon=False);

    # Correct the spacing between subplots, to fix some kinds of accidental overlap.
    plt.tight_layout()

    # Save the figure 
    # plt.savefig('performance.svg', dpi=200)  # for editing later in a vector graphics editor (i.e. Inkscape, Adobe Illustrator)
    plt.savefig('performance.png', dpi=200)