OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

How does connection pooling work in python multiprocessing?

  • Thread starter Thread starter Sanjay S
  • Start date Start date
S

Sanjay S

Guest
With python multiprocessing, Let's assume, I'm creating 10 processes by forking the main process. We know that child process inherit the state of the parent process in its own memory space.

I have a SQLAlchemy engine created in the parent process with pool size of 5. During the time of forking there aren't any connections in the pool.

Since the child processes have their own memory space that means the engine inherited should be only accessible to them, right?

  1. If that's the case, Does it mean the total number of active connections possible would be 2 [ number of processes ] * 5 [ pool_size ] = 10?
  2. If not, How does the connection pool work here?

Code:
    import multiprocessing
    import os
    
    # Create an engine with a connection pool in the parent process
    engine = create_engine('sqlite:///example.db', pool_size=5)
    
    def initializer():
        # Connections in the pool are gracefully removed!
        engine.dispose(close=False)
    
    def get_stuff():
      with engine.connect() as conn:
            conn.execute(text("..."))
    
    
    with multiprocessing.Pool(10,initializer=initializer) as pool:
      pool.apply(get_stuff)
<p>With python multiprocessing, Let's assume, I'm creating 10 processes by forking the main process. We know that child process inherit the state of the parent process in its own memory space.</p>
<p>I have a SQLAlchemy engine created in the parent process with pool size of 5. During the time of forking there aren't any connections in the pool.</p>
<p>Since the child processes have their own memory space that means the engine inherited should be only accessible to them, right?</p>
<ol>
<li>If that's the case, Does it mean the total number of active
connections possible would be 2 [ number of processes ] * 5 [ pool_size ] = 10?</li>
<li>If not, How does the connection pool work here?</li>
</ol>
<pre><code> import multiprocessing
import os

# Create an engine with a connection pool in the parent process
engine = create_engine('sqlite:///example.db', pool_size=5)

def initializer():
# Connections in the pool are gracefully removed!
engine.dispose(close=False)

def get_stuff():
with engine.connect() as conn:
conn.execute(text("..."))


with multiprocessing.Pool(10,initializer=initializer) as pool:
pool.apply(get_stuff)
</code></pre>
 

Latest posts

Top