Python メモリリーク
Pythonのメモリリークについて調べたことを記載してきます。
メモリリークとは
メモリリークとはGCが行われたのにまだ解放されていないメモリがあることを指しています。
PythonのGCの動きとしては生成したオブジェクトがどこにも参照されなくなった時にメモリを解放します。
なのでメモリリークは、プログラム内でオブジェクトがもう使われていないのにそのオブジェクトへの参照が残ってしまっている時に発生します。
参照が残ってしまう例
Pythonのdelは一般的には要素を削除しつつメモリを解放してくれる文法ですが、以下の例のように他で参照しているとメモリの解放を行なってくれません。
class Member:
def __init__(self, _id, name):
self._id = _id
self.name = name
def run(_):
members = {}
members['M001'] = Member('M001', 'Taro')
names = {}
names['Taro'] = members['M001']
print('---- before ----')
print('members:', dict(members))
print('names:', dict(names))
del members['M001']
print('---- after ----')
print('members:', dict(members))
print('names:', dict(names))
実行結果
---- before ----
members: {'M001': <memory.Member object at 0x1027919d0>}
names: {'Taro': <memory.Member object at 0x1027919d0>}
---- after ----
members: {}
names: {'Taro': <memory.Member object at 0x1027919d0>}
メモリリーク例
クラス変数に生成したインスタンスがインスタンス内で呼び出し元のインスタンスを保持しているとき
コード例
class ClassB:
def __init__(self):
self.__values = {}
@property
def values(self):
return self.__values
def add(self, instance, name):
self.__values[instance] = name
class ClassA:
class_b = ClassB()
def add(self, name):
self.class_b.add(self, name)
def run():
class_a = ClassA()
class_a.add('class A')
print('values:', dict(class_a.class_b.values))
def loop():
for _ in range(3):
run()
values: {<memory.ClassA object at 0x106971ca0>: 'class A'}
values: {<memory.ClassA object at 0x106971ca0>: 'class A', <memory.ClassA object at 0x106971a00>: 'class A'}
values: {<memory.ClassA object at 0x106971ca0>: 'class A', <memory.ClassA object at 0x106971a00>: 'class A', <memory.ClassA object at 0x1069a1550>: 'class A'}
結果を見ると、run関数が呼ばれるたびにClassBのvaluesの中のclass_aが増えていくのが分かります。
本来はrun関数の終了とともにインスタンス化したclass_aのメモリは解放して欲しいのですが、このようにして使われないのに残ってしまいます。
解決策としてはweakrefモジュールのWeakKeyDictionaryを使うことです。
from weakref import WeakKeyDictionary
class ClassB:
def __init__(self):
self.__values = WeakKeyDictionary()
@property
def values(self):
return self.__values
def add(self, instance, name):
self.__values[instance] = name
class ClassA:
class_b = ClassB()
def add(self, name):
self.class_b.add(self, name)
実行結果
values: {<memory.ClassA object at 0x10eac2ca0>: 'class A'}
values: {<memory.ClassA object at 0x10eac2ca0>: 'class A'}
values: {<memory.ClassA object at 0x10eac2ca0>: 'class A'}
他のメモリリークの例としてはファイルやネットワークのopen関数の後にcloseの呼び忘れなどがあるみたいです。
メモリを計測する方法
1 tracemalloc
実際に先程の処理をtracemallocで見てみます。
import tracemalloc
import memory_test
tracemalloc.start(10)
current_snap = tracemalloc.take_snapshot()
for _ in range(3):
memory_test.run()
snap = tracemalloc.take_snapshot()
stats = snap.compare_to(current_snap, 'lineno')
for stat in stats:
if 'memory_test.py' in str(stat):
print(stat)
current_snap = next_snap
実行結果
61行目がClassAをインスタンス化している箇所になります。
（WeakKeyDictionaryを使わない場合)
---- before ----
values: {}
---- after ----
values: {<memory_test.ClassA object at 0x10809f9d0>: 'class A'}
/dev/private/test/memory_test.py:66: size=64 B (+64 B), count=1 (+1), average=64 B
/dev/private/test/memory_test.py:63: size=64 B (+64 B), count=1 (+1), average=64 B
/dev/private/test/memory_test.py:61: size=48 B (+48 B), count=1 (+1), average=48 B
/dev/private/test/memory_test.py:39: size=2256 B (+0 B), count=9 (+0), average=251 B
/dev/private/test/memory_test.py:53: size=1808 B (+0 B), count=10 (+0), average=181 B
/dev/private/test/memory_test.py:46: size=208 B (+0 B), count=2 (+0), average=104 B
/dev/private/test/memory_test.py:43: size=168 B (+0 B), count=3 (+0), average=56 B
/dev/private/test/memory_test.py:69: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:60: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:56: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:49: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:41: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:54: size=48 B (+0 B), count=1 (+0), average=48 B
---- before ----
values: {<memory_test.ClassA object at 0x10809f9d0>: 'class A'}
---- after ----
values: {<memory_test.ClassA object at 0x10809f9d0>: 'class A', <memory_test.ClassA object at 0x10840bfd0>: 'class A'}
/dev/private/test/memory_test.py:53: size=1744 B (-64 B), count=9 (-1), average=194 B
/dev/private/test/memory_test.py:66: size=112 B (+48 B), count=2 (+1), average=56 B
/dev/private/test/memory_test.py:61: size=96 B (+48 B), count=2 (+1), average=48 B
/dev/private/test/memory_test.py:39: size=2256 B (+0 B), count=9 (+0), average=251 B
/dev/private/test/memory_test.py:46: size=208 B (+0 B), count=2 (+0), average=104 B
/dev/private/test/memory_test.py:43: size=168 B (+0 B), count=3 (+0), average=56 B
/dev/private/test/memory_test.py:69: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:60: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:56: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:49: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:41: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:63: size=64 B (+0 B), count=1 (+0), average=64 B
/dev/private/test/memory_test.py:54: size=48 B (+0 B), count=1 (+0), average=48 B
---- before ----
values: {<memory_test.ClassA object at 0x10809f9d0>: 'class A', <memory_test.ClassA object at 0x10840bfd0>: 'class A'}
---- after ----
values: {<memory_test.ClassA object at 0x10809f9d0>: 'class A', <memory_test.ClassA object at 0x10840bfd0>: 'class A', <memory_test.ClassA object at 0x1083c9c10>: 'class A'}
/dev/private/test/memory_test.py:61: size=144 B (+48 B), count=3 (+1), average=48 B
/dev/private/test/memory_test.py:39: size=2256 B (+0 B), count=9 (+0), average=251 B
/dev/private/test/memory_test.py:53: size=1744 B (+0 B), count=9 (+0), average=194 B
/dev/private/test/memory_test.py:46: size=208 B (+0 B), count=2 (+0), average=104 B
/dev/private/test/memory_test.py:43: size=168 B (+0 B), count=3 (+0), average=56 B
/dev/private/test/memory_test.py:69: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:60: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:56: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:49: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:41: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:66: size=112 B (+0 B), count=2 (+0), average=56 B
/dev/private/test/memory_test.py:63: size=64 B (+0 B), count=1 (+0), average=64 B
/dev/private/test/memory_test.py:54: size=48 B (+0 B), count=1 (+0), average=48 B
（WeakKeyDictionaryを使った場合）
---- before ----
values: {}
---- after ----
values: {<memory_test.ClassA object at 0x10d8d49d0>: 'class A'}
/dev/private/test/memory_test.py:66: size=488 B (+488 B), count=2 (+2), average=244 B
/dev/private/test/memory_test.py:50: size=448 B (+448 B), count=1 (+1), average=448 B
/dev/private/test/memory_test.py:57: size=432 B (+432 B), count=1 (+1), average=432 B
/dev/private/test/memory_test.py:39: size=2256 B (+0 B), count=9 (+0), average=251 B
/dev/private/test/memory_test.py:53: size=1808 B (+0 B), count=10 (+0), average=181 B
/dev/private/test/memory_test.py:46: size=208 B (+0 B), count=2 (+0), average=104 B
/dev/private/test/memory_test.py:42: size=152 B (+0 B), count=3 (+0), average=51 B
/dev/private/test/memory_test.py:69: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:60: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:56: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:49: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:41: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:54: size=48 B (+0 B), count=1 (+0), average=48 B
---- before ----
values: {}
---- after ----
values: {<memory_test.ClassA object at 0x10dc41fa0>: 'class A'}
/dev/private/test/memory_test.py:53: size=1744 B (-64 B), count=9 (-1), average=194 B
/dev/private/test/memory_test.py:66: size=536 B (+48 B), count=3 (+1), average=179 B
/dev/private/test/memory_test.py:39: size=2256 B (+0 B), count=9 (+0), average=251 B
/dev/private/test/memory_test.py:50: size=448 B (+0 B), count=1 (+0), average=448 B
/dev/private/test/memory_test.py:57: size=432 B (+0 B), count=1 (+0), average=432 B
/dev/private/test/memory_test.py:46: size=208 B (+0 B), count=2 (+0), average=104 B
/dev/private/test/memory_test.py:42: size=152 B (+0 B), count=3 (+0), average=51 B
/dev/private/test/memory_test.py:69: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:60: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:56: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:49: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:41: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:54: size=48 B (+0 B), count=1 (+0), average=48 B
---- before ----
values: {}
---- after ----
values: {<memory_test.ClassA object at 0x10d8d49d0>: 'class A'}
/dev/private/test/memory_test.py:66: size=488 B (-48 B), count=2 (-1), average=244 B
/dev/private/test/memory_test.py:39: size=2256 B (+0 B), count=9 (+0), average=251 B
/dev/private/test/memory_test.py:53: size=1744 B (+0 B), count=9 (+0), average=194 B
/dev/private/test/memory_test.py:50: size=448 B (+0 B), count=1 (+0), average=448 B
/dev/private/test/memory_test.py:57: size=432 B (+0 B), count=1 (+0), average=432 B
/dev/private/test/memory_test.py:46: size=208 B (+0 B), count=2 (+0), average=104 B
/dev/private/test/memory_test.py:42: size=152 B (+0 B), count=3 (+0), average=51 B
/dev/private/test/memory_test.py:69: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:60: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:56: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:49: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:41: size=136 B (+0 B), count=1 (+0), average=136 B
/dev/private/test/memory_test.py:54: size=48 B (+0 B), count=1 (+0), average=48 B
2 objgraph
import objgraph
import memory_test
for _ in range(3):
memory_test.run()
objgraph.show_growth()
実行結果
（WeakKeyDictionaryを使わない場合)
---- before ----
values: {}
---- after ----
values: {<memory.ClassA object at 0x104bf6ca0>: 'class A'}
function 2326 +2326
wrapper_descriptor 1080 +1080
dict 1077 +1077
tuple 868 +868
builtin_function_or_method 792 +792
weakref 761 +761
method_descriptor 735 +735
getset_descriptor 411 +411
member_descriptor 318 +318
type 317 +317
---- before ----
values: {<memory.ClassA object at 0x104bf6ca0>: 'class A'}
---- after ----
values: {<memory.ClassA object at 0x104bf6ca0>: 'class A', <memory.ClassA object at 0x104bf6a00>: 'class A'}
ClassA 2 +1
---- before ----
values: {<memory.ClassA object at 0x104bf6ca0>: 'class A', <memory.ClassA object at 0x104bf6a00>: 'class A'}
---- after ----
values: {<memory.ClassA object at 0x104bf6ca0>: 'class A', <memory.ClassA object at 0x104bf6a00>: 'class A', <memory.ClassA object at 0x104c26550>: 'class A'}
ClassA 3 +1
（WeakKeyDictionaryを使った場合）
---- before ----
values: {}
---- after ----
values: {<memory.ClassA object at 0x10eb3dca0>: 'class A'}
function 2327 +2327
wrapper_descriptor 1080 +1080
dict 1077 +1077
tuple 869 +869
builtin_function_or_method 792 +792
weakref 762 +762
method_descriptor 735 +735
getset_descriptor 411 +411
member_descriptor 318 +318
type 317 +317
---- before ----
values: {}
---- after ----
values: {<memory.ClassA object at 0x10eb3dca0>: 'class A'}
---- before ----
values: {}
---- after ----
values: {<memory.ClassA object at 0x10eb3dca0>: 'class A'}
3 Pyrasite
Pyrasiteはコードを触らずに計測ができるライブラリです。 Pyrasiteとobjgraphのように組み合わせて使います。
参考
Effective Python Brett Slatkin著