Python - Concurrent

Python has 2 builtin concurrent utilities which is widely utilized, i.e., process pool and thread pool, dedicated to CPU intensity and I/O intensity respectively.

• Due to global interpreter lock (GIL), a process pool should be used for a CPU intensive task. In this case, each CPU involved (e.g., all the CPU by default) can be fully utilized.
• While for a I/O intensive task, a thread pool should be used instead.

A simple example can be found below.

  from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed

  # Create a process pool with 4 workers.
  pool = ProcessPoolExecutor(4)
  # Create a thread pool with 4 workers.
  pool = ThreadPoolExecutor(4)

  # Submit tasks to the pool.
  futures = [pool.submit(FUNCTION, ARG) for ARG in ARGS]

  # Get the results.
  for f in as_completed(futures):
      f.result()