OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Broken Process Pool: Processpool executor inside django app hosted on apache

  • Thread starter Thread starter jack wilson
  • Start date Start date
J

jack wilson

Guest
I have a django application that I would like to host on my apache server. Use case of the application is to receive requests and performs compute intensive operations over a set of given inputs. To parallelise my computations I used future.ProcessPool executor in the core of the django app. Parallelisation in indispensable for my application.

While the above setup works locally, I face BrokenProcessPool error especially when the app is hosted on the apache server. Tried few tricks in some related questions (here) but I have no clear idea what is happening or how to resolve it. Could someone please help me understand:

  1. Technically, what is the underlying issue in my setup?
  2. How do I resolve it without massively changing my existing framework(Django, Apache)?
  3. Can you provide me references for an ideal workflow used in some well known data-science or graphics web applications using Django and parallel computations. This could help me revise my current framework.

Below is some relevant information


  • views.py looks like

    Code:
     import core.ray_trace
    
      def get_heatmap(request):
         list_of_inputs = request.POST.get('inputs')
          # parallel computation of list happens inside raytrace
          response_data = ray_trace(list_of_inputs)
          return JsonResponse(response_data, status=200)

  • Apache config for my project

    Code:
       WSGIDaemonProcess sp python-path=/home/ubuntu/_/spaceplanning python-home=/home/ubuntu/_/venv processes=3 threads=10 request-timeout=600
       WSGIProcessGroup sp
       WSGIScriptAlias / /home/ubuntu/_/sp/spAPI/wsgi.py

  • ps aux | grep apache enter image description here


  • Error message and Traceback: concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. enter image description here

I read switching my frame work to Celery or gunicorn server could help but honestly I could not catch on the intricates of the details provided.

Finally, are there any good web hosting services catering for data science and graphics applications that I could just use to unburden myself from hosting?
<p>I have a django application that I would like to host on my apache server. Use case of the application is to receive requests and performs compute intensive operations over a set of given inputs. To parallelise my computations I used <code>future.ProcessPool</code> executor in the core of the django app. Parallelisation in indispensable for my application.</p>
<p>While the above setup works locally, I face <code>BrokenProcessPool</code> error especially when the app is hosted on the apache server. Tried few tricks in some related questions (<a href="https://stackoverflow.com/questions/63669056/how-to-use-multiprocess-in-django-command">here</a>) but I have no clear idea what is happening or how to resolve it. Could someone please help me understand:</p>
<ol>
<li>Technically, what is the underlying issue in my setup?</li>
<li>How do I resolve it without massively changing my existing framework(Django, Apache)?</li>
<li>Can you provide me references for an ideal workflow used in some well known data-science or graphics web applications using Django and parallel computations. This could help me revise my current framework.</li>
</ol>
<p>Below is some relevant information</p>
<ul>
<li><p>views.py looks like</p>
<pre><code> import core.ray_trace

def get_heatmap(request):
list_of_inputs = request.POST.get('inputs')
# parallel computation of list happens inside raytrace
response_data = ray_trace(list_of_inputs)
return JsonResponse(response_data, status=200)
</code></pre>
</li>
<li><p>Apache config for my project</p>
<pre><code> WSGIDaemonProcess sp python-path=/home/ubuntu/_/spaceplanning python-home=/home/ubuntu/_/venv processes=3 threads=10 request-timeout=600
WSGIProcessGroup sp
WSGIScriptAlias / /home/ubuntu/_/sp/spAPI/wsgi.py
</code></pre>
</li>
<li><p><code>ps aux | grep apache</code>
<a href="https://i.sstatic.net/6cjlkABM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6cjlkABM.png" alt="enter image description here" /></a></p>
</li>
<li><p>Error message and Traceback: <code>concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.</code>
<a href="https://i.sstatic.net/19TIVyZ3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/19TIVyZ3.png" alt="enter image description here" /></a></p>
</li>
</ul>
<p>I read switching my frame work to Celery or gunicorn server could help but honestly I could not catch on the intricates of the details provided.</p>
<p>Finally, are there any good web hosting services catering for data science and graphics applications that I could just use to unburden myself from hosting?</p>
 

Latest posts

Top