Running the Test Suite

Learn how to run and write tests for CPython

Testing is a critical part of CPython development. This guide covers running the test suite, interpreting results, and writing new tests.

Running Tests

Basic Test Run

Run the entire test suite:

make test

The test suite produces output showing which tests pass, fail, or are skipped. You can generally ignore messages about skipped tests due to optional features.

By default, tests are prevented from overusing resources like disk space and memory.

Resource-Intensive Tests

To run tests that use more resources (like the buildbots do):

make buildbottest

Running Specific Tests

Single Test Module

Run a specific test file:

./python -m test test_os

Multiple Test Modules

Run several specific tests:

./python -m test test_os test_pathlib test_shutil

With Make

You can also use make with the TESTOPTS variable:

make test TESTOPTS="-v test_os test_gdb"

Test Options

Verbose Output

Get detailed output for debugging:

./python -m test -v test_os

Running Failed Tests

If tests fail, re-run them in verbose mode:

# After initial run shows failures
make test TESTOPTS="-v test_os test_gdb"

Common Test Options

# Verbose output
./python -m test -v test_module

# Run tests matching a pattern
./python -m test -m test_pattern test_module

# Run with timeout
./python -m test --timeout=300 test_module

# Run in random order
./python -m test -r test_module

# Fail fast (stop on first failure)
./python -m test -x test_module

# Use multiple processes
./python -m test -j4

# List tests without running
./python -m test --list-tests test_module

# Get help on all options
./python -m test --help

Understanding Test Output

Successful Test

test_os passed

Skipped Test

test_ssl skipped -- No module named '_ssl'

This is normal if you haven’t installed the required dependencies.

Failed Test

test_os failed

A failure indicates a problem. Check the traceback for details.

Test with Errors

test_os crashed -- Traceback (most recent call last):
  ...

Crashes or core dumps indicate serious problems.

Debug Builds and Testing

When testing with a debug build (--with-pydebug):

# Show reference counts
./python -X showrefcount
>>> 23
23
[8288 refs, 14332 blocks]
>>>

This helps detect memory leaks - if the count increases without storing new objects, there’s likely a leak.

Special Build Testing

Reference Counting Tests

With Py_REF_DEBUG enabled (included in --with-pydebug):

import sys
print(sys.gettotalrefcount())  # Available in debug builds

Trace References

For deep reference debugging:

./configure --with-trace-refs
make

# Set environment variable before running
export PYTHONDUMPREFS=1
./python script.py

Running Test Subsets

By Category

# Core tests only
./python -m test --core

# Standard library tests
./python -m test --stdlib

# Tests requiring network
./python -m test -u network

# Tests requiring CPU resources
./python -m test -u cpu

By Pattern

# All tests matching pattern
./python -m test -m test_dict*

# All tests in a directory
./python -m test test.test_asyncio

Writing Tests

Test File Structure

Create test files in Lib/test/:

import unittest
from test import support

class MyTestCase(unittest.TestCase):
    def test_something(self):
        self.assertEqual(1 + 1, 2)

    def test_something_else(self):
        with self.assertRaises(ValueError):
            int('invalid')

if __name__ == '__main__':
    unittest.main()

Using Test Support

The test.support module provides utilities:

from test import support

# Skip tests if feature unavailable
@support.requires_subprocess()
def test_subprocess_feature(self):
    pass

# Clean up resources
with support.temp_dir() as tmpdir:
    # Use temporary directory
    pass

C Extension Tests

For testing C API:

import _testcapi

class CAPITest(unittest.TestCase):
    def test_c_function(self):
        result = _testcapi.test_function()
        self.assertEqual(result, expected)

Continuous Integration

CPython uses multiple CI systems:

GitHub Actions

View build status:

• GitHub Actions
• Runs on Linux, macOS, and Windows
• Tests multiple configurations

Azure Pipelines

View build status:

• Azure DevOps
• Additional platform coverage

Reporting Test Failures

If tests fail and it appears to be a CPython problem:

1. Re-run in verbose mode

   make test TESTOPTS="-v test_failing_module"
   
2. Verify it's not your environment

   • Try a clean build
   • Check system dependencies
   • Test on a different platform if possible
3. File a bug report

   Visit github.com/python/cpython/issues and include:

   • CPython version/commit
   • Platform and OS version
   • Verbose test output
   • Steps to reproduce

Troubleshooting

Tests are too slow

Use parallel testing:

./python -m test -j4

Or run fewer tests:

./python -m test test_specific_module

Random test failures

Some tests can be flaky. Try:

# Run test multiple times
./python -m test -x -r test_module

If it’s consistently flaky, report it as a bug.

Out of memory errors

Some tests use significant memory. Either:

• Run tests individually
• Use make test instead of make buildbottest
• Skip memory-intensive tests

Permission errors in tests

Some tests require specific permissions. Run with appropriate access or skip those tests:

./python -m test -x test_module

Test Coverage

Generate coverage reports:

# Configure with coverage support
./configure --with-pydebug
make

# Run with coverage
make coverage

# View coverage report
make coverage-report

Next Steps

• Learn the development workflow
• Read the code style guide
• Running & Writing Tests (detailed guide)

Additional Resources

• Python Test Documentation
• unittest Module
• CPython Developer Guide - Testing
• GitHub Issue Tracker