“What wall might you hit in Python programs?”
“At some point in writing Python programs
“What might happen even after optimizing your code?”
“Even after optimizing your code
“What is a reasonable assumption for modern computers?”
“On modern computers that have an increasing number of CPU cores
“Does the GIL allow true parallelism in threads?”
“Unfortunately
“What is another common suggestion for performance?”
“Another common suggestion is to re-write your performance-critical code as an extension module
“What can C do for Python programs?”
“C gets you closer to the bare metal and can run faster than Python
“Can C extensions start native threads?”
“C extensions can also start native threads independent of the Python interpreter than run in parallel and utilize multiple CPU cores with no concern for the GIL.”
“Is Python’s C extension API well documented?”
“Python’s API for C extensions is well documented and a good choice for an escape hatch.”
“What tools can help with C extensions?”
“It’s also worth checking out tools like SWIG and CLIF.”
“What is the cost of rewriting in C?”
“However
“How does Python code compare to C code in terms of complexity?”
“Code that is short and understandable in Python can become verbose and complicated in C.”
“What does porting to C require?”
“Such a port requires extensive testing to ensure that the functionality is equivalent to the original Python code and that no bugs have been introduced.”
“Is it sometimes worth porting to C?”
“Sometimes it’s worth it
“What tools ease the transition to C?”
“There are even open-source tools such as Cython and Numba that ease the transition to C.”
“Is moving one piece to C sufficient?”
“The problem is that moving one piece of your program to C isn’t sufficient most of the time.”
“What do optimized Python programs usually have as sources of slowness?”
“Optimized Python programs usually don’t have one major source of slowness; rather
“What would you need to port to get C’s benefits?”
“To get the benefits of C’s bare metals and threads
“Is there a better way than porting to C?”
“There has to be a better way to preserve your investment in Python to solve difficult computation problems.”
“What module may be what you need for parallelism?”
“The multiprocessing module
“What does multiprocessing enable?”
“It enables Python to use multiple CPU cores in parallel by running additional interpreters as child processes.”
“Are child processes separate from the main interpreter?”
“These child processes are separate from the main interpreter
“What can each child fully utilize?”
“Each child can fully utilize one CPU core.”
“What does each child have?”
“Each child has a link to the main process where it receives instructions to do computation and return results.”
“Can you change ThreadPoolExecutor to ProcessPoolExecutor?”
“In some cases