Courses
Building Robust Neuroscience Experiments with Python and PsychoPy
Modularization and Testing of Experimental Code
Automated Testing with Pytest

Automated Testing with Pytest

Author
Dr. Ole Bialas
Open in JupyterLite Download Exercises

Manual testing is an integral part of software development. However, it does not scale well with the size of a project. The more your codebase grows the more time you have to spend testing. What’s more, as things become more complex it becomes difficult to understanded all the interactions between different part of your code and it is easy to break one part of your program by modifying another one. This is where automated testing are invaluable. By building a suite of tests you can constantly verify that your code works as expected and that new features are being integrated properly!

import random
import json
from psychopy.sound import Sound
from psychopy.visual import Window
from psychopy.event import waitKeys
import pytest
import ipytest

ipytest.autoconfig()
Sound(stereo=False);

Section 1: Writing Assert Statements

A key component of every test is the assert statement. It tests that result of some part of the code is what you would expect. assert is really simple - it raises an AssertionError if whateber you are asserting evaluates to False and does nothing otherwise. Generally, more asserts are better but often it is a good idea to focus your test on interesting edge cases, like: what happens when you pass in an empty list to a function that accepts lists? What if a time parameter is zero or negative? And so on …

Code Duration
assert a == b Raise an AssertionError if a is NOT equal to b, otherwise do nothing
assert isinstance(a, int) Raise an AssertionError if a is not an integer, otherwise do nothing
assert a > b Raise an AssertionError if a is NOT greater than b, otherwise do nothing
tone = Sound(value=350, secs=0.5) Create a tone at 350Hz with a duration of 0.5 seconds
key = waitKeys(keyList="space", timeStamped=True) Wait until the "space" key was pressed and return a list of lists with [[name, time]].

Exercises

Example: Write an assert statement to check that

  • the value x below is positive

and print "Success"!.

x = random.random();
assert x > 0
"Success!"
'Success!'

Exercise: Write an assert statment to check that

  • the absolute difference between x and y is smaller than 1

and and print "Success"!.

x = random.random()
y = random.random();
Solution 
assert abs(x - y) < 1
"Success!"
'Success!'

Exercise: Write assert statements to check that

  • the length of x is 3
  • the first element x[0] is a string

and print "Success"!

x = ["1", "2", "3"]
Solution 
assert len(x) == 3
assert isinstance(x[0], str)
"Success!"
'Success!'

Exercise: Write two assert statements that check that:

  • the value returned by subtract(3, 5) is -2
  • the value returned by subtract(10, 7) is 3

and print "Success"!.

def subtract(a,b):
    return a-b
Solution 
assert subtract(3, 5) == -2
assert subtract(10, 7) == 3
"Success!"

Exercise: Write an assert statement to check that

  • the the length of the list returned by concatenate_lists(a,b) is equal to len(a) + len(b)

and print "Success"!.

def concatenate_lists(a,b):
    return a+b

a = [1, 2, 3]
b = [1, 2]
Solution 
assert len(concatenate_lists(a, b)) == len(a) + len(b)
"Success!"
'Success!'

Exercise: Write an assert statement that check that:

  • the tone returned by make_tone(freq=800, dur=0.5) has the attributes tone.sound=800 and tone.secs=0.5

and print "Success"!

def make_tone(freq, dur):
    return Sound(value=freq, secs=dur)
Solution 
tone = make_tone(freq=800, dur=0.5)
assert tone.sound == 800
assert tone.secs == 0.5
"Success!"
'Success!'

Section 2: Writing Test Functions

Usually assert statments are not used on their own but wrapped inside test functions. Organizing your tests into functions provides the same advantages as organizing your code in functions: it makes them more readable, maintainable and robust. What’s more, it allow tools such as Pytest to do the testing automatically and report to you any failed tests. In this section we are going to use Ipytest which is a convenient tool to use Pytest inside a Jupyter notebook: simply include an %%ipytest tag at the top of a cell and, when you execute it, Pytest will run every function in the cell that starts with test_. The rules for writing good test functions are the same as the rules for writing good function: each test should have a clear purpose an a name that reflects this purpose like test_file_was_written().

Code Description
%%ipytest Use Pytest to execute every function in a cell that starts with test_
with pytest.raises(ValueError):
     divide(1, "hi)		 Assert that divide(1,"hi") raises a ValueError

First, we are going to test the trial_sequence() function defined below

Exercises 

def trial_sequence(conditions: list, n_reps: int, shuffle:bool = True):
    trials = conditions * n_reps
    if shuffle:
        random.shuffle(trials)
    return trials

Example: Write a test function test_trial_sequence_is_shuffled() to tests that

  • the returned list is shuffled when shuffle=True

Add the %%ipytest tag to the top of the cell and run it to execute the test.

%%ipytest
def test_trial_sequence_is_shuffled():
    trials1 = trial_sequence(conditions=[1,2,3], n_reps=1000, shuffle=True)
    trials2 = trial_sequence(conditions=[1,2,3], n_reps=1000, shuffle=True)
    assert trials1 != trials2
.                                                                                            [100%]
1 passed in 0.00s

Exercise: Write a function test_trial_sequence_is_ordered() to tests that

  • the returned list is ordered when shuffle=False

Add the %%ipytest tag to the top of the cell and run it to execute the test.

Solution 
%%ipytest
def test_trial_sequence_is_ordered():
    trials1 = trial_sequence(conditions=[1,2,3], n_reps=1000, shuffle=False)
    trials2 = trial_sequence(conditions=[1,2,3], n_reps=1000, shuffle=False)
    assert trials1 == trials2
.                                                                                            [100%]
1 passed in 0.00s

Exercise: Write a function test_trial_list_length() to tests that

  • the len() of the returned list is 3000 when conditions=[1,2,3] and n_reps=1000
  • the len() of the returned list is 0 when conditions=[] and n_reps=1000

Add the %%ipytest tag to the top of the cell and run it to execute the test.

Solution 
%%ipytest
def test_trial_list_length():
    trials1 = trial_sequence(conditions=[1,2,3], n_reps=1000, shuffle=True)
    assert len(trials1) == 3000
    trials2 = trial_sequence(conditions=[], n_reps=1000, shuffle=True)
    assert len(trials2) == 0
.                                                                                            [100%]
1 passed in 0.00s

Next, we are going to test the load_config function defined in the cell below.

def load_config(fpath="config.json"):
    with open(fpath) as f:
        config = json.load(f)
    return config

Exercise: Write a function test_config_has_keys() to tests that

  • the returned dictionary contains the keys "conditions" and "n_trials"

Add the %%ipytest tag to the top of the cell and run it to execute the test.

Solution 
%%ipytest
def test_config_has_keys():
    config = load_config()
    assert "conditions" in config.keys()
    assert "n_trials" in config.keys()
.                                                                                            [100%]
1 passed in 0.00s

Exercise: Write a test_config_dtypes() function to test that

  • the value stored under the key "conditions" returned dictionary is a list
  • the value stored under the key "n_trials" returned dictionary is an int

Add the %%ipytest tag to the top of the cell and run it to execute the test.

Solution 
%%ipytes
def test_config_dtypes():
    config = load_config()
    assert isinstance(config["conditions"], list)
    assert isinstance(config["n_trials"], int)
.                                                                                            [100%]
1 passed in 0.00s

Exercise: Write a test_load_config_raises_error() function that uses pytest.raises to test that

  • load_config raises a FileNotFoundError when trying to load a file that does not exist

Add the %%ipytest tag to the top of the cell and run it to execute the test.

Solution 
%%ipytest
def test_load_config_raises_error():
    with pytest.raises(FileNotFoundError) as err:
        load_config("Beep")
.                                                                                            [100%]
1 passed in 0.00s

Section 3: Runnig Pytest from the Terminal

Now that we understand how Pytest works we can start to develop our test module. The tests work exactly the same as in a notebook but you put them in a separate file. Usually this file is named like the file it tests with the “test_” prefix (e.g. test_my_module.py would contain the tests for my_module.py). In the test module, you can import the functions you want to test and then test them. Organizing your test like this has the advantage that you can run all of them by simply calling pytest a single time. Per default, Pytest will open every Python file that starts with test_ and, inside that file, run every function that starts with test_.

In the same folder as this notebook, create a new file called sequencegen.py, then copy the code below and save it to that file. In the following exercises, we will test the functions from the sequencegen module!

Exercises 

import random

def make_sequence(conditions, n_trials, min_dist=0, max_iter=1000):
    n_reps = int(n_trials / len(conditions))
    trials = conditions * n_reps
    random.shuffle(trials)
    count = 0
    while has_repetitions(trials, min_dist):
        random.shuffle(trials)
        count += 1
        if count >= max_iter:
            raise StopIteration(f"Could not find a sequence after {count} iterations!")
    return trials


def has_repetitions(trials, min_dist=1):
    trial_is_repeat = []
    trial_is_repeat.append(
        len(set(trials[:min_dist])) < len(trials[:min_dist])
    )  # check beginning
    for i in range(min_dist, len(trials)):  # check rest of list
        this_trial_is_repeat = []
        for j in range(1, min_dist + 1):
            this_trial_is_repeat.append(trials[i] == trials[i - j])
        trial_is_repeat.append(any(this_trial_is_repeat))
    return any(trial_is_repeat)


def save_sequence(trials, fname):
    with open(fname, 'w') as f:
        for i, t in enumerate(trials):
            if i < len(trials) - 1:
                f.write(str(t) + '\n')
            else:
                f.write(str(t))  # no newline for last element

Example: In the same folder as this noebook, create a new file called test_sequencegen.py and import the has_repetitions function from squencegen. Write a test_has_repetitions() function to test that:

  • has_repetitions() returns True when passed a list with repetitions
  • has_repetitions() returns False when passed a list without repetitions

Then, run !pytest to execute the tests.

Solution 
# The content of test_sequencegen.py may look like this:
from sequencegen import has_repetitions

def test_has_repetitions():
    assert has_repetitions([1,1,2,3]) == True
    assert has_repetitions([1,2,3,1]) == False
!pytest test_sequencegen.py::test_has_repetitions
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/olebi/projects/new-learning-platform/notebooks/psychopy/03_modularization_and_testing/02_automated_testing
plugins: anyio-4.11.0
collected 1 item                                                               

test_sequencegen.py .                                                    [100%]

============================== 1 passed in 0.00s ===============================

Exercise: add another function called test_has_repetitions_respects_min_dist to test that:

  • has_repetitions(x) with x=[1,2,3,1,2,3] is False when min_dist=1 and min_dist=2
  • has_repetitions(x) with x=[1,2,3,1,2,3] is True when min_dist=3 and min_dist=4

Then, run !pytest to execute the tests.

Solution 
# the function in test_sequencegen may look like this:
def test_has_repetitions_respects_min_dist():
    x = [1,2,3,1,2,3]
    assert has_repetitions(x, min_dist=1) == False
    assert has_repetitions(x, min_dist=2) == False
    assert has_repetitions(x, min_dist=3) == True 
    assert has_repetitions(x, min_dist=4) == True 
!pytest test_sequencegen.py::test_has_repetitions_respects_min_dist
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/olebi/projects/new-learning-platform/notebooks/psychopy/03_modularization_and_testing/02_automated_testing
plugins: anyio-4.11.0
collected 1 item                                                               

test_sequencegen.py .                                                    [100%]

============================== 1 passed in 0.01s ===============================

Exercises: Import make_sequence() from sequencegen in test_sequencegen.py and add a test_sequence_has_correct_len() function to test that

  • make_sequence() returns a list with 50 elements when conditions=[1,2] and n_trials=50
  • make_sequence() returns a list with 3 elements when conditions=[1,2,3] and n_trials=3
  • make_sequence() returns a list with 0 elements when conditions=[] and n_trials=100

Then, run !pytest to execute the tests.

BONUS: What is the length of the returned list when conditions=[1,2] and n_trials=9?

Solution 
# The function in test_sequence.py may look like this:
from sequencegen import has_repetitions, make_sequence

def test_sequence_has_correct_len():
    assert len(make_sequence([1,2], 50)) == 50
    assert len(make_sequence([1,2,3], 3)) == 3
    # BONUS
    assert len(make_sequence([1, 2], 9)) == 8
!pytest test_sequencegen.py::test_sequence_has_correct_len
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/olebi/projects/new-learning-platform/notebooks/psychopy/03_modularization_and_testing/02_automated_testing
plugins: anyio-4.11.0
collected 1 item                                                               

test_sequencegen.py .                                                    [100%]

============================== 1 passed in 0.00s ===============================

Exercises: Add a test_max_iterations() function to test_sequencegen.py that uses pytest.raises to test that

  • make_sequence() raises a StopIteration when conditions=[1,2,3], n_trials=50 and min_dist=5

Then, run !pytest to execute the tests.

Solution 
# The function in test_sequencegen.py may look like this:
def test_max_iterations():
    with pytest.raises(StopIteration) as stop:
        make_sequence(conditions=[1,2,3], n_trials=50, min_dist=5)
!pytest test_sequencegen.py::test_max_iterations
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/olebi/projects/new-learning-platform/notebooks/psychopy/03_modularization_and_testing/02_automated_testing
plugins: anyio-4.11.0
collected 1 item                                                               

test_sequencegen.py .                                                    [100%]

============================== 1 passed in 0.02s ===============================

You can also run Pytest on all of your tests at once, simply run !pytest without any arguments and it will run every module and function that starts with test_

!pytest
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/olebi/projects/new-learning-platform/notebooks/psychopy/03_modularization_and_testing/02_automated_testing
plugins: anyio-4.11.0
collected 4 items                                                              

test_sequencegen.py ....                                                 [100%]

============================== 4 passed in 0.02s ===============================

Another nice feature of pytest is that it can determine our test coverage which tells us the percentage of code that is covered by our tests. A test coverage below 100% means that there are certain parts of your code that are never executed. This can help us to identify any blind spots in our code. Just run the cell below to let Pytest measure the test coverage for sequencegen.py

!pytest --cov
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/olebi/projects/new-learning-platform/notebooks/psychopy/03_modularization_and_testing/02_automated_testing
plugins: cov-7.0.0, anyio-4.11.0
collected 4 items                                                              

test_sequencegen.py ....                                                 [100%]

================================ tests coverage ================================
_______________ coverage: platform linux, python 3.12.12-final-0 _______________

Name                  Stmts   Miss  Cover
-----------------------------------------
sequencegen.py           33     10    70%
test_sequencegen.py      18      0   100%
-----------------------------------------
TOTAL                    51     10    80%
============================== 4 passed in 0.06s ===============================

Section 4: Parameterizing Tests

Writing assert statements to tests all the different combinations of parameters that may be important for you program can be laborious. This is where parameterization is extremely useful. It provides us with an esy interface to run a large number of test. For example, we could test a function like add with a large number of values by parameterizing it with the range() function. Pytets will take all those values and run our test function with every single one. For this to work, the name in parameterize must have the same name as the parameter of the test function. The example below parameterizes the test_sum function so pytest will run it 1000 times, passing the values from range(1000) to the variable a.

Exercises 

%%ipytest
@pytest.mark.parametrize("a", range(1000))
def test_sum(a):
    assert sum([a,3]) == a+3 
............................................................................................ [  9%]
............................................................................................ [ 18%]
............................................................................................ [ 27%]
............................................................................................ [ 36%]
............................................................................................ [ 46%]
............................................................................................ [ 55%]
............................................................................................ [ 64%]
............................................................................................ [ 73%]
............................................................................................ [ 82%]
............................................................................................ [ 92%]
................................................................................             [100%]
1000 passed in 1.53s

We can also parameterize multiple variables. In this case, we pass in both variable names followed by a list of tuples, each of which contains the parameters for one run of the test. In the example below, the parameterized test will run three times, where a will take on the values 1, 3, and 4 and b will take on the values 2, 4 and 6.

%%ipytest
@pytest.mark.parametrize("a, b", [(1,2), (3,4), (5,6)])
def test_sum(a, b):
    assert sum([a,b]) == a+b 
...                                                                                          [100%]
3 passed in 0.01s

Exercise: Parameterize the function test_make_sequence_runs_all_iterations() to test that make_sequence raises a StopIteration

  • after 10 iterations if max_iter=10
  • after 100 iterations if max_iter=100
  • after 1000 iterations if max_iter=1000

And run the cell to execute the tests

%%ipytest
from sequencegen import make_sequence

def test_make_sequence_runs_all_iterations(max_iter):
    with pytest.raises(StopIteration, match=f"Could not find a sequence after {max_iter} iterations!"):\
        make_sequence(conditions=[1,2,3], n_trials=100, min_dist=5)
Solution 
%%ipytest
from sequencegen import make_sequence

@pytest.mark.parametrize("max_iter", [10, 100, 1000])
def test_make_sequence_runs_all_iterations(max_iter):
    with pytest.raises(StopIteration, match=f"Could not find a sequence after {max_iter} iterations!"):
        make_sequence(conditions=[1,2,3], n_trials=100, min_dist=5, max_iter=max_iter)
...                                                                                          [100%]
3 passed in 0.04s

Exercise: Parameterize the test_condition_counts_are_balanced() function below to test that all conditions appear in euqal numbers when:

  • conditions = [1,2]
  • conditions = [1,2,3]
  • conditions = ["A", "B"]

Then run the cell to execute the tests.

%%ipytest

def test_condition_counts_are_balanced(conditions):
    seq = make_sequence(conditions, n_trials=100)
    counts = [seq.count(c) for c in conditions]
    assert len(set(counts)) == 1
Solution 
%%ipytest

@pytest.mark.parametrize("conditions", [[1, 2], [1, 2, 3], ['A', 'B']])
def test_condition_counts_are_balanced(conditions):
    seq = make_sequence(conditions, n_trials=100)
    counts = [seq.count(c) for c in conditions]
    assert len(set(counts)) == 1
...                                                                                          [100%]
3 passed in 0.01s
%%ipytest

@pytest.mark.parametrize("conditions, min_dist", [
    ([1,2], 2),
    ([1,2,3], 3),
    ([1,2,3,4], 4)
    ])
def test_impossible_sequence_is_not_made(conditions, min_dist):
    with pytest.raises(StopIteration):
        make_sequence(conditions, n_trials=100, min_dist=min_dist)
...                                                                                          [100%]
3 passed in 0.09s