Http client Memory Leak

[imkonsowa]

I'm using laravel 7.4.0 and with php 7.4.10

Currently working on a project which dispatches about 10K job daily using laravel queue database driver. The problem is the worker is consuming about 2GB of our server ram and the below error occurs frequently

PHP Fatal error:  Allowed memory size of 33554432 bytes exhausted (tried to allocate 1347584 bytes) in /home/cowpay/public_html/vendor/guzzlehttp/psr7/src/Stream.php on line 95

In Stream.php line 95:
                                                                               
  Allowed memory size of 33554432 bytes exhausted (tried to allocate 1347584   
  bytes)

I'm using laravel Http client and it throws that error.

Any help here will be appreciated

[ob1y2k]

experiencing same issue when using : Storage::disk('s3')->get('path/to/file.ext')

workaround is to set this above the code :

ini_set('memory_limit', '2048M');
Storage::disk('s3')->get('path/to/file.ext')

but this is not memory efficient.

any thoughts? something like Storage::putFileAs(...) for get() method could help

[georgeboot]

Consider streaming your download, instead of loading the entire file into memory.

[davesic]

Consider streaming your download, instead of loading the entire file into memory.

and?

[mikeydevelops]

I do not think this is framework related. An alternative if something is not clearing right. You could schedule queue:restart every 10 minutes or so. I assume you are using supervisor for your workers, so they would automatically start up again and continue running your jobs.

But you should investigate what is the reason for the high memory usage. If you are working with large files, as @georgeboot suggested you should consider streaming the downloads.

[henzeb]

You could try issue a gc_collect_cycle(), and if that does the trick configure the garbage collector correctly.