Skip to content

SNOW-2173966 introduce server DoP cap #2375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sfc-gh-zyao
Copy link
Contributor

Description

  • introduce server DoP cap to connector
    • Currently sproc in fact use serial processing when it comes to file transfer agent execution. We introduce this server DoP cap to allow server side control of such behaviors
    • for Snowflake reviewers, more details are covered in this doc

Testing

  • Unit tests for the new change
  • Existing tests to make sure this change does not break anything
  • Manual testing to show that this change works for sproc
    Screenshot 2025-06-24 at 6 05 33 PM

Please answer these questions before submitting your pull requests. Thanks!

  1. What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes #NNNN

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am modifying authorization mechanisms
    • I am adding new credentials
    • I am modifying OCSP code
    • I am adding a new dependency
  3. Please describe how your code solves the related issue.

    Please write a short description of how your code change solves the related issue.

  4. (Optional) PR for stored-proc connector:

@sfc-gh-zyao sfc-gh-zyao requested a review from sfc-gh-sfan June 25, 2025 01:10
@@ -1086,6 +1088,11 @@ def execute(
use_s3_regional_url=self._connection.enable_stage_s3_privatelink_for_us_east_1,
iobound_tpe_limit=self._connection.iobound_tpe_limit,
unsafe_file_write=self._connection.unsafe_file_write,
snowflake_server_dop_cap_for_file_transfer=getattr(
self.connection,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, why is this not self._connection like above?

Comment on lines +1854 to +1858
snowflake_server_dop_cap_for_file_transfer=getattr(
self.connection,
f"_{_VARIABLE_NAME_SERVER_DOP_CAP_FOR_FILE_TRANSFER}",
_DEFAULT_VALUE_SERVER_DOP_CAP_FOR_FILE_TRANSFER,
),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used multiple times. Maybe we could create helper function?

It might be okay to expose this as a connection property as the name seems legit. This can just be called _snowflake_max_parallel_for_file_transfer?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants