# 10x faster python test iteration via fork(2)

DevFeed: [10x faster python test iteration via fork(2)](<https://devfeed.tech/articles/10x-faster-python-test-iteration-via-fork-2-20122.md>)

Original publisher: [Read original article](<https://benchling.engineering/10x-faster-python-test-iteration-via-fork-2-3aae52d2f6?source=rss----3d4aa8fb07ea---4>)

Author: raylu

Published: 2023-07-20T16:01:45Z

Content type: tutorial

Language: en

Sources: [Benchling](<https://devfeed.tech/sources/benchling.md>)

Topics: [Python](<https://devfeed.tech/topics/python.md>), [Testing](<https://devfeed.tech/topics/testing.md>), [SQLAlchemy](<https://devfeed.tech/topics/sqlalchemy.md>), [modules](<https://devfeed.tech/topics/modules.md>), [import](<https://devfeed.tech/topics/import.md>), [Code](<https://devfeed.tech/topics/code.md>)

Tags: [benchling](<https://devfeed.tech/tags/benchling.md>), [code](<https://devfeed.tech/tags/code.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [developer-productivity](<https://devfeed.tech/tags/developer-productivity.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [fork](<https://devfeed.tech/tags/fork.md>), [import](<https://devfeed.tech/tags/import.md>), [modules](<https://devfeed.tech/tags/modules.md>), [python](<https://devfeed.tech/tags/python.md>), [sqlalchemy](<https://devfeed.tech/tags/sqlalchemy.md>), [testing](<https://devfeed.tech/tags/testing.md>)

## AI overview

This Benchling Engineering article explains how the Build team reduced Python test iteration time in a dependency-heavy codebase. It discusses the limitations of importlib.reload() and describes a fork-based approach that made the second test run start 10 times faster, reducing waiting by 90%.

## Source excerpt

It's ideal to get feedback on your code faster -- to make a code change and see the result instantly. But, as projects get larger, reload times get longer. Each incremental dependency or bootstrap code block that adds 200ms feels worth it, but 50 of them later and it takes 10 seconds to see the result of a code change. On the Build team at Benchling, that's where we found ourselves one day. We used 146 packages which pull in 128 transitive dependencies for a total of 274 packages. We also spent a lot of time waiting for SQLAlchemy models to initialize. The result is our test harness took 10 seconds to set up. After making a code change, you'd start the test runner, wait a few seconds, alt+tab to your browser, get distracted for a few minutes, and then find out you had a typo in your code. This is a common challenge for a growing codebase, but it's something we knew we needed to fix. Here's the process we arrived at which allowed the second run of tests to start 10x faster -- 90% less waiting. While it'll work a little differently for your codebase depending on the language, dependencies, etc. you're using, hopefully this can inspire you on your journey to faster feedback and testing. importlib.reload() Since the problem is that we spend so long setting up a bunch of modules just right and then want to see the change in a single file we're editing, the most obvious solution is to use importlib.reload from the standard library. import importlib import sys import test_harness_stuff # takes 10 seconds import tests def rerun_tests(changed_path): for mod in sys.modules.values(): if mod.__file__ == changed_path: importlib.reload(mod) tests.run_tests() break if __name__ == '__main__': setup_file_watcher(rerun_tests) tests.run_tests() This (with some special handling for built-in modules, relative path resolution, and batching to handle editors that perform multiple filesystem operations per save) works alright when the file being changed is a test file (or any other leaf node