Memory leak for simple get call

[dima-kov]

🐞 Describe the bug

I have faced with memory leak in my program: after loading 100k URLs the process eats additional 300mb of RAM. I had extracted this piece of code, that reproduces memory issue (below) and ran it with mprof for 5 min for loading 4k urls:

💡 To Reproduce

Use env:

aiohttp==4.0.0a1
async-timeout==3.0.1
attrs==19.3.0
chardet==3.0.4
cycler==0.10.0
idna==2.9
kiwisolver==1.1.0
matplotlib==3.2.0
memory-profiler==0.57.0
multidict==4.7.5
numpy==1.18.1
psutil==5.7.0
pyparsing==2.4.6
python-dateutil==2.8.1
six==1.14.0
typing-extensions==3.7.4.1
uvloop==0.14.0
yarl==1.4.2

Tested with different versions of aiohttp (>3.5)

Code:

import asyncio
import uvloop
import aiohttp

url = 'https://docs.mapbox.com/mapbox-gl-js/assets/earthquakes.geojson'

load_times = 4000


def get_to_load():
    global load_times
    if load_times > 0:
        load_times -= 1
        return url
    return None


async def fetch(session, url):
    response = await session.get(url)
    return await response.text()


async def load(session, worker_id):
    to_load = get_to_load()
    while to_load is not None:
        r = await fetch(session, to_load)
        print('Done', worker_id)
        await asyncio.sleep(0.2)
        to_load = get_to_load()


async def main(workers_num=90):
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[load(session, i) for i in range(workers_num)])


asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
loop = asyncio.get_event_loop()

asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
asyncio.run(asyncio.sleep(3))
asyncio.run(main())
asyncio.run(asyncio.sleep(5))

3. Run for 5 min with profiler.

💡 Expected behavior
Keep memory usage near straight during the whole program run.

📋 Your version of the Python
Python 3.7.5 (Also tested on 3.8)

📋 Your version of the aiohttp/yarl/multidict distributions

• aiohttp:

Name: aiohttp
Version: 4.0.0a1

• yarn:

Name: yarl
Version: 1.4.2

• multidict

Name: multidict
Version: 4.7.5

📋 Additional context
I have read all issues related to memory leaks, but have not found any appropriate solution. Here are what I tried:

• update all packages
• aiohttp 4.0
• python 3.8
• uvloop implementation instead of asyncio
• single aiohttp.ClientSession instead of new on every request (actually, this gave a bit increase)

📋 Enviroment

• OSX

ProductVersion: 10.15.1
BuildVersion:   19B88

• also ran in docker (linux) - the same

[KKomarov]

such session helps
aiohttp.ClientSession(cookie_jar=aiohttp.DummyCookieJar())
also your request objects not closed exlicitly

[socketpair]

async def fetch (session, url):
    async with session.get(...) as x:
        return await x.text()

[dima-kov]

such session helps
aiohttp.ClientSession(cookie_jar=aiohttp.DummyCookieJar())
also your request objects not closed exlicitly

@KKomarov but why they are not closed explicitly? According to docs, as I understood, calling .text() should be enough?

[dima-kov]

@KKomarov, Dummy cookies have not helped

[dima-kov]

@socketpair , here is a result. Also, not increase

[KKomarov]

@dima-kov closing responses helped for me, memory is about 100mb. maybe it's closing after calling text but not immediately.

[dima-kov]

@KKomarov , could you share your code, please? I tried with this and failed:

import asyncio
import uvloop
import aiohttp

url = 'https://docs.mapbox.com/mapbox-gl-js/assets/earthquakes.geojson'
load_times = 4000


def get_to_load():
    global load_times
    if load_times > 0:
        load_times -= 1
        return url
    return None


async def fetch(session, url):
    response = await session.get(url)
    t = await response.text()
    response.close()
    return t


async def load(session, worker_id):
    to_load = get_to_load()
    print(f'start {worker_id}')
    while to_load is not None:
        r = await fetch(session, to_load)
        print('Done', worker_id)
        to_load = get_to_load()


async def main(workers_num=90):
    async with aiohttp.ClientSession(cookie_jar=aiohttp.DummyCookieJar()) as session:
        await asyncio.gather(*[load(session, i) for i in range(workers_num)])


asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
asyncio.run(asyncio.sleep(3))
asyncio.run(main())
asyncio.run(asyncio.sleep(5))

[KKomarov]

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

[hh-h]

this is a graph for this code

import asyncio
import aiohttp

url = 'https://docs.mapbox.com/mapbox-gl-js/assets/earthquakes.geojson'
load_times = 4000


def get_to_load():
    global load_times
    if load_times > 0:
        load_times -= 1
        return url
    return None


async def fetch(session, url):
    async with session.get(url) as r:
        return await r.text()


async def load(session, worker_id):
    to_load = get_to_load()
    print(f'start {worker_id}')
    while to_load is not None:
        await fetch(session, to_load)
        print('Done', worker_id)
        to_load = get_to_load()


async def main(workers_num=90):
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[load(session, i) for i in range(workers_num)])


asyncio.run(asyncio.sleep(3))
asyncio.run(main())
asyncio.run(asyncio.sleep(5))

but I suggest to rewrite it to something like this code

import asyncio
import aiohttp
from typing import Coroutine

URL = 'https://docs.mapbox.com/mapbox-gl-js/assets/earthquakes.geojson'
MAX_LOAD_TIMES = 500


async def fetch(session: aiohttp.ClientSession, url: str):
    async with session.get(url) as r:
        return await r.text()


async def limited(semaphore: asyncio.Semaphore, coro: Coroutine) -> None:
    async with semaphore:
        await coro


async def main(workers_num=90):
    await asyncio.sleep(3)
    semaphore = asyncio.Semaphore(workers_num)
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[limited(semaphore, fetch(session, URL)) for _ in range(MAX_LOAD_TIMES)])
    await asyncio.sleep(5)

asyncio.run(main())

graph (decreased load_times to 500)

[socketpair]

A week ago I faced almost the same problem. Seems Python makes huge memory fragmentation. So in order to account memory you should consider two methods:

1. number of anoymous pages of the process
2. summ of arguments of all malloc()s that were not free()d

Probably, due to enormous memory fragmentation point 1 will grow and point 2 will not.

I'm not sure, but possibly you need to use tracemalloc module

[Luis310C]

this bug has not been fixed?

[socketpair]

No, not fixed. Do you have sequence how to reproduce ?

[hf-kklein]

relates to #4833