Skip to content

Use more efficient ListPager API. #10398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

fasaxc
Copy link
Member

@fasaxc fasaxc commented May 12, 2025

Description

Use ListPager.EachListItem instead of ListPager.List. Allows for incremental conversion page-by-page instead of loading up all pages into an upstream-format List object and then converting en-masse.

Noticed the more efficient API when doing a code read for a different issue.

Related issues/PRs

Todos

  • Tests
  • Documentation
  • Release note

Release Note

Calico now uses less RAM when resyncing with the API server.  Objects are converted to Calico format in pages instead of buffering all upstream Kubernetes objects pre-conversion.

Reminder for the reviewer

Make sure that this PR has the correct labels and milestone set.

Every PR needs one docs-* label.

  • docs-pr-required: This change requires a change to the documentation that has not been completed yet.
  • docs-completed: This change has all necessary documentation completed.
  • docs-not-required: This change has no user-facing impact and requires no docs.

Every PR needs one release-note-* label.

  • release-note-required: This PR has user-facing changes. Most PRs should have this label.
  • release-note-not-required: This PR has no user-facing changes.

Other optional labels:

  • cherry-pick-candidate: This PR should be cherry-picked to an earlier release. For bug fixes only.
  • needs-operator-pr: This PR is related to install and requires a corresponding change to the operator.

@marvin-tigera marvin-tigera added this to the Calico v3.31.0 milestone May 12, 2025
@marvin-tigera marvin-tigera added release-note-required Change has user-facing impact (no matter how small) docs-pr-required Change is not yet documented labels May 12, 2025
@fasaxc fasaxc added docs-not-required Docs not required for this change and removed docs-pr-required Change is not yet documented labels May 12, 2025
@fasaxc fasaxc force-pushed the incremental-page-read branch from bab1f06 to b439d2c Compare May 13, 2025 09:22
@fasaxc fasaxc changed the title Use more effiicent ListPager API. Use more efficient ListPager API. May 14, 2025
Use ListPager.EachListItem instead of ListPager.List.  Allows for
incremental conversion page-by-page instead of loading up all pages
into an upstream-format List object and then converting en-masse.
@fasaxc fasaxc force-pushed the incremental-page-read branch from b439d2c to 924a1c7 Compare May 14, 2025 14:25
Fixes panic due to new check that all pages lists find a revision.
Copy link

This PR is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stale Issues without recent activity label Jul 14, 2025
@fasaxc fasaxc removed the stale Issues without recent activity label Jul 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required Docs not required for this change release-note-required Change has user-facing impact (no matter how small)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants