OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Memory use of pickling/unpickling enum in Python

  • Thread starter Thread starter Matthieu Dartiailh
  • Start date Start date
M

Matthieu Dartiailh

Guest
While investigating the memory use of an application that exchanges message through PyZMQ using tracemalloc, I noticed a weird behavior of pickling and unpickling enums: memory looks like it is leaking.

This seems impossible to me so I am guessing I am measuring something in an incorrect manner. Could someone let me know if this a real leak or, if it is not, how I should be measuring memory usage to avoid seeing such behavior ?

I reduced my test to:

Code:
import enum
import gc
import os
import pickle
import tracemalloc
from uuid import SafeUUID


class A:
    pass


@enum.unique
class E(enum.Enum):
    A = 1


obj = A()
tracemalloc.start()
for obj in (A(), E.A, SafeUUID.safe):
    print(obj)
    for j in range(10):
        tracemalloc.clear_traces()
        i = 0
        while True:
            s = pickle.dumps(obj)
            o = pickle.loads(s)

            del s, o

            if i == 100:
                gc.collect()
                origin = tracemalloc.take_snapshot().filter_traces(
                    [tracemalloc.Filter(True, __file__)]
                )
            elif i > 200:
                gc.collect()
                snap = tracemalloc.take_snapshot().filter_traces(
                    [tracemalloc.Filter(True, __file__)]
                )
                print(str(snap.compare_to(origin, "lineno")[0]).rsplit(os.sep, 1)[-1])
                break
            i += 1
    print()

The exact output is noisy but I can get things similar to the following, in which pickling a class instance displays no variation, pickling a local enum see some memory increase, and pickling an enum defined in a different module can result in large memory swings.

Code:
<__main__.A object at 0x000002840A56B150>
enum-pickle-leak.py:27: size=1520 B (+1520 B), count=1 (+1), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B

E.A
enum-pickle-leak.py:26: size=3640 B (+112 B), count=65 (+2), average=56 B
enum-pickle-leak.py:27: size=4040 B (+168 B), count=46 (+3), average=88 B
enum-pickle-leak.py:27: size=7288 B (+224 B), count=104 (+4), average=70 B
enum-pickle-leak.py:27: size=2528 B (+168 B), count=19 (+3), average=133 B
enum-pickle-leak.py:26: size=392 B (+112 B), count=7 (+2), average=56 B
enum-pickle-leak.py:26: size=168 B (+112 B), count=3 (+2), average=56 B
enum-pickle-leak.py:27: size=4600 B (+112 B), count=56 (+2), average=82 B
enum-pickle-leak.py:27: size=1800 B (+168 B), count=6 (+3), average=300 B
enum-pickle-leak.py:27: size=1744 B (+112 B), count=5 (+2), average=349 B
enum-pickle-leak.py:27: size=3312 B (+112 B), count=33 (+2), average=100 B

SafeUUID.safe
enum-pickle-leak.py:27: size=16.6 KiB (+10.3 KiB), count=283 (+193), average=60 B
enum-pickle-leak.py:27: size=12.9 KiB (+1098 B), count=214 (+20), average=62 B
enum-pickle-leak.py:27: size=12.3 KiB (+494 B), count=202 (+9), average=62 B
enum-pickle-leak.py:27: size=5539 B (+440 B), count=74 (+8), average=75 B
enum-pickle-leak.py:27: size=2678 B (+659 B), count=22 (+12), average=122 B
enum-pickle-leak.py:27: size=3343 B (+887 B), count=34 (+16), average=98 B
enum-pickle-leak.py:27: size=3377 B (+599 B), count=35 (+11), average=96 B
enum-pickle-leak.py:27: size=9367 B (+551 B), count=144 (+10), average=65 B
enum-pickle-leak.py:27: size=2834 B (+434 B), count=25 (+8), average=113 B
enum-pickle-leak.py:27: size=3005 B (+771 B), count=28 (+14), average=107 B
<p>While investigating the memory use of an application that exchanges message through PyZMQ using tracemalloc, I noticed a weird behavior of pickling and unpickling enums: memory looks like it is leaking.</p>
<p>This seems impossible to me so I am guessing I am measuring something in an incorrect manner. Could someone let me know if this a real leak or, if it is not, how I should be measuring memory usage to avoid seeing such behavior ?</p>
<p>I reduced my test to:</p>
<pre><code>import enum
import gc
import os
import pickle
import tracemalloc
from uuid import SafeUUID


class A:
pass


@enum.unique
class E(enum.Enum):
A = 1


obj = A()
tracemalloc.start()
for obj in (A(), E.A, SafeUUID.safe):
print(obj)
for j in range(10):
tracemalloc.clear_traces()
i = 0
while True:
s = pickle.dumps(obj)
o = pickle.loads(s)

del s, o

if i == 100:
gc.collect()
origin = tracemalloc.take_snapshot().filter_traces(
[tracemalloc.Filter(True, __file__)]
)
elif i > 200:
gc.collect()
snap = tracemalloc.take_snapshot().filter_traces(
[tracemalloc.Filter(True, __file__)]
)
print(str(snap.compare_to(origin, "lineno")[0]).rsplit(os.sep, 1)[-1])
break
i += 1
print()
</code></pre>
<p>The exact output is noisy but I can get things similar to the following, in which pickling a class instance displays no variation, pickling a local enum see some memory increase, and pickling an enum defined in a different module can result in large memory swings.</p>
<pre><code><__main__.A object at 0x000002840A56B150>
enum-pickle-leak.py:27: size=1520 B (+1520 B), count=1 (+1), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B

E.A
enum-pickle-leak.py:26: size=3640 B (+112 B), count=65 (+2), average=56 B
enum-pickle-leak.py:27: size=4040 B (+168 B), count=46 (+3), average=88 B
enum-pickle-leak.py:27: size=7288 B (+224 B), count=104 (+4), average=70 B
enum-pickle-leak.py:27: size=2528 B (+168 B), count=19 (+3), average=133 B
enum-pickle-leak.py:26: size=392 B (+112 B), count=7 (+2), average=56 B
enum-pickle-leak.py:26: size=168 B (+112 B), count=3 (+2), average=56 B
enum-pickle-leak.py:27: size=4600 B (+112 B), count=56 (+2), average=82 B
enum-pickle-leak.py:27: size=1800 B (+168 B), count=6 (+3), average=300 B
enum-pickle-leak.py:27: size=1744 B (+112 B), count=5 (+2), average=349 B
enum-pickle-leak.py:27: size=3312 B (+112 B), count=33 (+2), average=100 B

SafeUUID.safe
enum-pickle-leak.py:27: size=16.6 KiB (+10.3 KiB), count=283 (+193), average=60 B
enum-pickle-leak.py:27: size=12.9 KiB (+1098 B), count=214 (+20), average=62 B
enum-pickle-leak.py:27: size=12.3 KiB (+494 B), count=202 (+9), average=62 B
enum-pickle-leak.py:27: size=5539 B (+440 B), count=74 (+8), average=75 B
enum-pickle-leak.py:27: size=2678 B (+659 B), count=22 (+12), average=122 B
enum-pickle-leak.py:27: size=3343 B (+887 B), count=34 (+16), average=98 B
enum-pickle-leak.py:27: size=3377 B (+599 B), count=35 (+11), average=96 B
enum-pickle-leak.py:27: size=9367 B (+551 B), count=144 (+10), average=65 B
enum-pickle-leak.py:27: size=2834 B (+434 B), count=25 (+8), average=113 B
enum-pickle-leak.py:27: size=3005 B (+771 B), count=28 (+14), average=107 B
</code></pre>
 

Online statistics

Members online
0
Guests online
5
Total visitors
5
Top