Testing guidelines

  • Do repeat yourself: This is expected when you write tests

Goals

  • Tests should be simple.
  • Tests should be fast.
  • Tests should be independent.

Rules to follow

Those rules are based around the unit testing and doesn't cover unnitest library or advanced API testing.

Reminder: Python PEP8 standard must be followed.

Test functions only test one thing

A test function only has one purpose: to test one, and only one behavior. If it tests two behaviors, it should be split into two tests.

def function():
    if x:
        a()
    else:
        b()

Tests should be:

def test_function_a_was_called()
    # assert a.call
    # assert b.not_call

Test function name must indicate what it test

A test function name is self explanatory and must follow those rules:

  • test_ : It must start by test_ to be detected by pytest.
  • The rest of the test function name must be the function name it is testing follow by what behavior is tested.

Passing parameters to your tests

@pytest.fixture is the best way to parametrize test values in order to test different result based on various inputs.

Before(bad)

def test_format_data_for_display():
    people = [
        {
            "given_name": "Foo",
            "family_name": "Bar",
            "title": "Senior Software Engineer",
        },
        {
            "given_name": "Fii",
            "family_name": "Baz",
            "title": "Project Manager",
        },
    ]

    assert format_data_for_display(people) == [
        "Foo Bar: Senior Software Engineer",
        "Fii Baz: Project Manager",
    ]

After(good)

@pytest.fixture
def example_people_data():  # Create a function that return all the test data
    return [
        {
            "given_name": "Foo",
            "family_name": "Bar",
            "title": "Senior Software Engineer",
        },
        {
            "given_name": "Fii",
            "family_name": "Baz",
            "title": "Project Manager",
        },
    ]

def test_format_data_for_display(example_people_data): # Pass the function as a param
    assert format_data_for_display(example_people_data) == [
        "Foo Bar: Senior Software Engineer",
        "Fii Baz: Project Manager",
    ]

OR, @pytest.mark.parametrize can be use when the input data has a direct impact on the test output (exceptions, errors...):

@pytest.mark.parametrize(
    ("give_name", "family_name", "title"), # Data scheme of the next nested list
    [
        [   # First data input
            "Foo",
            "Bar",
            "Senior Software Engineer",
        ],
        [   # Second data input
            "Fii",
            "Baz",
            "Project Manager",
        ],
    ]
)
def test_format_data_for_display(give_name, family_name, title): # Pass the data scheme as params
    assert format_data_for_display([give_name, family_name, title]) == [
        "Foo Bar: Senior Software Engineer",
        "Fii Baz: Project Manager",
    ]

All public functions must be tested independently

Public function are tested one at a time. If the tested function calls other public functions, those other public functions must be mocked.

# Src module: path = src.math
def addition(x, y) -> int:
    odd = is_odd(x)
    if odd:
        #...
    else:
        #...
    return x + y

def is_odd(value) -> bool:
    # ...

# Test module
@patch("src.math.is_odd") # This patch replaces all is_odd() return values by a MagicMock()
def test_addition(mocked_is_odd): # The mocked function with patch must be placed as a param
    result = addition(1, 2)

    assert result == 3
    mocked_is_x_odd.assert_called()

Never mock a private function

Private functions are not tested on their own. Those functions are tested when a public function that calls them is tested.

Tests are divided in 3 sections

  1. Test setup: Lines before the function name up to a new empty line.
  2. Tested function call: Lines with the tested function call.
  3. Asserts: Lines with all the asserts needed.

Every section must be separated by one and only one empty line.

def test_addition():
    # Setup section
    x = 1
    y = 2

    # Function call section
    result = addition(x, y)

    # Assert section
    assert result == x + y

Tests functions don't need a return type

Every test function has the same -> None return type, there is no need to specify it.

Test a raised exception

Raised exception can be test with: with pytest.raises(ERROR):

def test_addition():
    assert format_data_for_display(people) == [
        "Foo Bar: Senior Software Engineer",
        "Fii Baz: Project Manager",
    ]
    with pytest.raises(ValueError): # Test will fail if no ValueError was raised
        result = addition(1, 2)
        # ...

Test logging in a function

If a function is logging anything, a test function must assert that the log was processed. Caplog is level-based and doesn't process any log record below the selected log level.

caplog example:

import logging

def test_addition_log(caplog): # Add caplog as a param
    result = addition(1, 2)

    assert result == 3
    assert caplog.records      # Ensure that logs were created
    for record in caplog.records:
        assert logging.WARNING == record.levelno # Verify the log level

How to mock

pytest is use for assertions and unittest for mocks.

from unittest.mock import MagicMock, Mock, patch

import pytest

@patch VS with patch

@patch is a cleaner way to use the patch:

  • All patches are in the setup section.
  • Patches are not creating useless indentation.

with patch(...) can only be use with fixture setup when a programmer wants to pass a setup function to all subsequent test functions, but this is a rare scenario.

Mocked function names must be explicit

Any mocked function is named with the following regex:

  • *mock*
def addition():
    # ...

@patch("src.math.addition")
def test_addition(mock_addition): # mock_addition or mocked_addition or addition_mock, for example

Patched paths must use a path constant

In a test module, instead of typing the tested module over and over in every @patch annotation, a path constant string can be added at the beginning of the module.

MODULE_PATH = "src.math"

@patch(f"{MODULE_PATH}.addition")
def test_addition(mocked_addition):

This makes refactoring easier when the module is moved or renamed.

Mock environment variables

Environment variables can be mock with @patch.dict(os.environ, ...):

@patch.dict(os.environ, {"toto": "test"})
def test_addition():
    # ...

@patch uses a MagicMock by default

A @patch annotation will replace a function call by a MagicMock().

MagicMocks can be modified to:

  • Return a specific value:
@patch(f"{MODULE_PATH}.addition")
def test_addition(mock_addition):
    mock_addition.return_value = 3
  • Raise an exception:
@patch(f"{MODULE_PATH}.addition")
def test_addition(mock_addition):
    mock_addition.side_effect = ValueError
Git flow