OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

bottleneck in multiprocessing in python

  • Thread starter Thread starter Souroy
  • Start date Start date
S

Souroy

Guest
I have a code that uses two parameters (r,t) to create a matrix H. I then find the eigenvalues of this matrix and do some computations using the eigenvalues. Now, I need to do this for many different values of (r,t), (around 200). Since each operation is independent I tried to use multiprocessing in python to have all these operations done simultaneously.

When I run the code for a single (r,t), it takes around 1.5 hours. I was under the impression that running it for 200 (r,t)s should take around the same time using multiprocessing. However, it seems t be taking significantly longer. I am well into the 2nd day of the code running. I am using my university's high performance unit so I believe there should be enough cores to assign. My code is something like below-

Code:
import numpy as np
import multiprocessing

rs = np.linspace(0.5,1.5,20)
thetas = np.linspace(0,np.pi/2,10)

def matrix_stuff(r,t):
#constructs a matrix H(r,t). Diagonalize and do stuff.


processes = []
rets = []
q = multiprocessing.Queue()

for i in range(len(thetas)):
    for j in range(len(rs)):
        theta = thetas[i]
        r = rs[I]
        p = multiprocessing.Process(target=matrix_stuff, args = (r,theta,i,j))
        processes.append(p)
        p.start()

for p in processes:
    ret = q.get()
    rets.append(ret)

for p in processes:
    p.join()


#store output in deltas

deltas = np.zeros((len(rs),len(thetas)))


for ret in rets:
    # ret has format (value, i,j)
    deltas[rets[1],rets[2]] = rets[0]

np.savetxt('new_deltas.txt',deltas)

My question is then are there some other areas where my program could be getting bottlenecked that I am not considering? This my is first time using multiprocessing so I am still not very clear on the details.
<p>I have a code that uses two parameters (r,t) to create a matrix H. I then find the eigenvalues of this matrix and do some computations using the eigenvalues. Now, I need to do this for many different values of (r,t), (around 200). Since each operation is independent I tried to use multiprocessing in python to have all these operations done simultaneously.</p>
<p>When I run the code for a single (r,t), it takes around 1.5 hours. I was under the impression that running it for 200 (r,t)s should take around the same time using multiprocessing. However, it seems t be taking significantly longer. I am well into the 2nd day of the code running. I am using my university's high performance unit so I believe there should be enough cores to assign. My code is something like below-</p>
<pre><code>import numpy as np
import multiprocessing

rs = np.linspace(0.5,1.5,20)
thetas = np.linspace(0,np.pi/2,10)

def matrix_stuff(r,t):
#constructs a matrix H(r,t). Diagonalize and do stuff.


processes = []
rets = []
q = multiprocessing.Queue()

for i in range(len(thetas)):
for j in range(len(rs)):
theta = thetas
r = rs
p = multiprocessing.Process(target=matrix_stuff, args = (r,theta,i,j))
processes.append(p)
p.start()

for p in processes:
ret = q.get()
rets.append(ret)

for p in processes:
p.join()


#store output in deltas

deltas = np.zeros((len(rs),len(thetas)))


for ret in rets:
# ret has format (value, i,j)
deltas[rets[1],rets[2]] = rets[0]

np.savetxt('new_deltas.txt',deltas)

</code></pre>
<p>My question is then are there some other areas where my program could be getting bottlenecked that I am not considering? This my is first time using multiprocessing so I am still not very clear on the details.</p>
 

Latest posts

Online statistics

Members online
0
Guests online
5
Total visitors
5
Top