process.OpenFiles() does not scale well and consumes ton of CPU

[Dainerx]

Describe the bug

I have an issue with fetching open files for a process. I am using https://github.com/shirou/gopsutil. Calling process.OpenFiles() which consumes a lot of CPU, profiling link: https://pprof.me/171f81b809d99641cf6a32fee8d78eba/?profileType=profile%253Acpu%253Ananoseconds%253Acpu%253Ananoseconds%253Adelta&color_by=filename.

To Reproduce

Run any program and call process.OpenFiles(). Make sure to call it on envs with around 1k processes and for each process around 200 fds.

Expected behavior

Expected not to stress the CPU that much. I had the same code with python where I call https://github.com/giampaolo/psutil/blob/d4b37d9628e634c00a2990e488b1cc38ea6a41b5/psutil/_pslinux.py#L2225. And it's not hogging that much CPU, I don't understand though because both code they perform the same thing: read the directory under /proc//fd and then resolve the links. Apparently os.ReadLink is very costly.

Environment (please complete the following information):

• Windows: [paste the result of ver]
• Linux: [paste contents of /etc/os-release and the result of uname -a]
• Mac OS: [paste the result of sw_vers and uname -a
• FreeBSD: [paste the result of freebsd-version -k -r -u and uname -a]
• OpenBSD: [paste the result of uname -a]

NAME="Red Hat Enterprise Linux"
VERSION="9.4 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"

[csbo98]

@Lomanic @Dainerx Hi，anybody is handling this problem? if not，I want to try.

[Dainerx]

@csbo98 I am not handling this issue. I had to write my own function for OpenFiles per PID with tweaks.

As we heavily rely on this library, I am very interested to review any proposal and test its performance against my use case. One thing that can help improving performance is making OpenFiles accept a predicate that will filter on files depending on the permission: OpenFiles(WithWritePermissions)

[csbo98]

ok, Thanks. I will think a better method. If done, I will propose it here.