# Refactoring Python with Multiprocessing Pools

DevFeed: [Refactoring Python with Multiprocessing Pools](<https://devfeed.tech/articles/the-complete-idiot-s-guide-to-refactoring-python-using-multiprocessing-pools-28239.md>)

Original publisher: [Read original article](<http://fuzzyblog.io/blog/python/2020/07/31/the-complete-idiot-s-guide-to-refactoring-python-using-multiprocessing-pools.html>)

Author: Fuzzygroup

Published: 2020-07-31T05:48:00Z

Content type: tutorial

Language: en

Sources: [Scott Johnson](<https://devfeed.tech/sources/scott-johnson.md>)

Topics: [Python](<https://devfeed.tech/topics/python.md>), [Concurrent Programming](<https://devfeed.tech/topics/concurrent-programming.md>), [Refactoring](<https://devfeed.tech/topics/refactoring.md>)

Tags: [cpu](<https://devfeed.tech/tags/cpu.md>), [processes](<https://devfeed.tech/tags/processes.md>), [python](<https://devfeed.tech/tags/python.md>), [refactoring](<https://devfeed.tech/tags/refactoring.md>), [scalability](<https://devfeed.tech/tags/scalability.md>)

## AI overview

A practical Python tutorial on restructuring a data-pipeline entry point so it can run through multiprocessing pools. It discusses choosing processes rather than threads, sizing pools around CPU cores for non-I/O-bound work, and benchmarking deployment configurations.

## Source excerpt

While I would happily proclaim that my ur language is Ruby, I spend increasingly large amounts of time these days using Python. And while there are many things that I don't like about Python (the syntax makes my eyes want to weep and then die; thanks Tim Curry / Psych; around 20 seconds in), the strength of the Python ecosystem is outstanding. Today I'm going to talk about the Python Multiprocessing library which is a standard part of Python and can be used without even needing to install anything. And this isn't going to be a theoretical explanation of processes / threads / parallelism. Instead it is going to be a simple explanation about how my favorite Python guru taught me to love the zen of multiprocessing with a very specific example. But we do need a few basics: In Python you want to use processes not threads. The reason for this is the infamous GIL issue which Real Python does a great job discussing so I'm just not going to get into it. Unless your python processes are heavily IO bound (example - calling networked APIs), you generally want to use a pool of processes tied to your CPU / Core count. Happily this is astonishingly trivial as the multiprocessing library gives you multiprocessing.cpu_count() as a core primitive. Please note that I recognize that I have vastly oversimplified this issue and that many people argue for number of cores - 1. As with all complex computing issues, well, ymmv. Debugging parallel software is always harder than you think it is so I only, ever, do this at the end of project when I know that my code works and where the bottlenecks are (i.e. is it IO bound for example). Consistency of coding practices makes a huge difference. In the code base I just left, I was able to transition all of it to a multiprocessing architecture trivially because I had invested heavily in consistency. Your deployment tooling makes a huge difference. If you want to experiment with multiprocessing then you need the ability to change your instance type /