Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

process.OpenFiles() does not scale well and consumes ton of CPU #1781

Open
1 of 5 tasks
Dainerx opened this issue Jan 22, 2025 · 0 comments
Open
1 of 5 tasks

process.OpenFiles() does not scale well and consumes ton of CPU #1781

Dainerx opened this issue Jan 22, 2025 · 0 comments

Comments

@Dainerx
Copy link

Dainerx commented Jan 22, 2025

Describe the bug

I have an issue with fetching open files for a process. I am using https://github.com/shirou/gopsutil. Calling process.OpenFiles() which consumes a lot of CPU, profiling link: https://pprof.me/171f81b809d99641cf6a32fee8d78eba/?profileType=profile%253Acpu%253Ananoseconds%253Acpu%253Ananoseconds%253Adelta&color_by=filename.

To Reproduce

Run any program and call process.OpenFiles(). Make sure to call it on envs with around 1k processes and for each process around 200 fds.

Expected behavior

Expected not to stress the CPU that much. I had the same code with python where I call https://github.com/giampaolo/psutil/blob/d4b37d9628e634c00a2990e488b1cc38ea6a41b5/psutil/_pslinux.py#L2225. And it's not hogging that much CPU, I don't understand though because both code they perform the same thing: read the directory under /proc//fd and then resolve the links. Apparently os.ReadLink is very costly.

Environment (please complete the following information):

  • Windows: [paste the result of ver]
  • Linux: [paste contents of /etc/os-release and the result of uname -a]
  • Mac OS: [paste the result of sw_vers and uname -a
  • FreeBSD: [paste the result of freebsd-version -k -r -u and uname -a]
  • OpenBSD: [paste the result of uname -a]

NAME="Red Hat Enterprise Linux"
VERSION="9.4 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants