OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

How to fully release GPU memory used in function

  • Thread starter Thread starter clemisch
  • Start date Start date
C

clemisch

Guest
I'm using cupy in a function that receives a numpy array, shoves it on the GPU, does some operations on it and returns a cp.asnumpy copy of it.

The problem: The memory is not freed after the function (as seen in ndidia-smi).

I know about the caching and re-using of memory done by cupy. However, this seems to work only per-user. When multiple users are computing on the same GPU-server, they are limited by the cached memory of other users.

I also tried calling cp._default_memory_pool.free_all_blocks() inside the function at the end. This seems to have no effect. Importing cupy in the main code and calling free_all_blocks "manually" works, but I'd like to encapsulate the GPU stuff in the function, not visible to the user.

Can you fully release GPU memory used inside a function so that it's usable by other users?



Minimal example:

Main module:

Code:
# dont import cupy here, only numpy
import numpy as np

# module in which cupy is imported and used
from memory_test_module import test_function

# host array
arr = np.arange(1000000)

# out is also on host, gpu stuff happens in test_function
out = test_function(arr)

# GPU memory is not released here, unless manually:
import cupy as cp
cp._default_memory_pool.free_all_blocks()

Function module:

Code:
import cupy as cp

def test_function(arr):
    arr_gpu = cp.array(arr)
    arr_gpu += 1
    out_host = cp.asnumpy(arr_gpu)

    # this has no effect
    cp._default_memory_pool.free_all_blocks()

    return out_host
<p>I'm using <code>cupy</code> in a function that receives a <code>numpy</code> array, shoves it on the GPU, does some operations on it and returns a <code>cp.asnumpy</code> copy of it. </p>

<p><strong>The problem</strong>: The memory is not freed after the function (as seen in <code>ndidia-smi</code>). </p>

<p>I know about the caching and re-using of memory done by <code>cupy</code>. However, this seems to work only per-user. When multiple users are computing on the same GPU-server, they are limited by the cached memory of other users. </p>

<p>I also tried calling <code>cp._default_memory_pool.free_all_blocks()</code> inside the function at the end. This seems to have no effect. Importing <code>cupy</code> in the main code and calling <code>free_all_blocks</code> "manually" works, but I'd like to encapsulate the GPU stuff in the function, not visible to the user.</p>

<p><strong>Can you fully release GPU memory used inside a function so that it's usable by other users?</strong></p>

<hr>

<p><strong>Minimal example:</strong></p>

<p>Main module:</p>

<pre><code># dont import cupy here, only numpy
import numpy as np

# module in which cupy is imported and used
from memory_test_module import test_function

# host array
arr = np.arange(1000000)

# out is also on host, gpu stuff happens in test_function
out = test_function(arr)

# GPU memory is not released here, unless manually:
import cupy as cp
cp._default_memory_pool.free_all_blocks()
</code></pre>

<p>Function module:</p>

<pre><code>import cupy as cp

def test_function(arr):
arr_gpu = cp.array(arr)
arr_gpu += 1
out_host = cp.asnumpy(arr_gpu)

# this has no effect
cp._default_memory_pool.free_all_blocks()

return out_host
</code></pre>
 
Top