Skip to content

Snow 2043816 kerberos proxy auth python #2359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 63 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
8be242b
SNOW-2043816: Added Http_interceptor logic for stages and snowflake api
sfc-gh-fpawlowski May 29, 2025
ca679da
[UCOMMIT THIS] SNOW-2043816: marked places of requests
sfc-gh-fpawlowski May 31, 2025
e6b1743
SNOW-2043816: Base of injection
sfc-gh-fpawlowski May 31, 2025
8334ea3
SNOW-2043816: Base for injection
sfc-gh-fpawlowski Jun 1, 2025
5289147
SNOW-2043816: Customization tests before running
sfc-gh-fpawlowski Jun 1, 2025
d48b168
SNOW-2043816: Customization tests running
sfc-gh-fpawlowski Jun 2, 2025
df6a48c
SNOW-2043816: Customization tests running - case sensitive adding hea…
sfc-gh-fpawlowski Jun 2, 2025
3e665e4
SNOW-2043816: Customization tests running all
sfc-gh-fpawlowski Jun 2, 2025
17347b3
SNOW-2043816: connection integ stash
sfc-gh-fpawlowski Jun 15, 2025
f18cec8
SNOW-2043816: Interceptors test for wiremock and integration
sfc-gh-fpawlowski Jun 15, 2025
924f252
SNOW-2043816: Added shared fixtures
sfc-gh-fpawlowski Jun 15, 2025
8cbeabc
SNOW-2043816: SEparate file
sfc-gh-fpawlowski Jun 15, 2025
c4ba926
SNOW-2043816: network py
sfc-gh-fpawlowski Jun 16, 2025
0f483a7
SNOW-2043816: storage py
sfc-gh-fpawlowski Jun 16, 2025
ddda565
SNOW-2043816: Imports fixed
sfc-gh-fpawlowski Jun 16, 2025
b3b54bd
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 16, 2025
ac2f4c5
SNOW-2043816: Chunks support added
sfc-gh-fpawlowski Jun 17, 2025
2cfe847
SNOW-2043816: Description added
sfc-gh-fpawlowski Jun 17, 2025
4085634
SNOW-2043816: Added imports for old driver
sfc-gh-fpawlowski Jun 17, 2025
64f0616
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 17, 2025
3eb2a6b
SNOW-2043816: Clean up
sfc-gh-fpawlowski Jun 17, 2025
9df7441
SNOW-2043816: test_http_interceptor fixed
sfc-gh-fpawlowski Jun 17, 2025
695bea4
SNOW-2043816: test_http_interceptor fixed
sfc-gh-fpawlowski Jun 17, 2025
ba88e41
SNOW-2043816: Adapter approach
sfc-gh-fpawlowski Jun 17, 2025
5bde0f8
SNOW-2043816: olddriver tests
sfc-gh-fpawlowski Jun 17, 2025
5381453
SNOW-2043816: Removed old test for attribute called request_intercept…
sfc-gh-fpawlowski Jun 17, 2025
f852437
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 18, 2025
b54e2f6
SNOW-2043816: Check if integ test would work
sfc-gh-fpawlowski Jun 22, 2025
514e65c
SNOW-2043816: Cloud agnostic chunks
sfc-gh-fpawlowski Jun 22, 2025
4e512ad
SNOW-2043816: Cloud agnostic chunks - azure
sfc-gh-fpawlowski Jun 22, 2025
52edb51
SNOW-2043816: Cloud agnostic chunks - azure
sfc-gh-fpawlowski Jun 22, 2025
70e07ce
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 22, 2025
885f758
SNOW-2043816: cloud
sfc-gh-fpawlowski Jun 22, 2025
b242da7
SNOW-2043816: multipart download
sfc-gh-fpawlowski Jun 22, 2025
16932f3
SNOW-2043816: removed wiremock for files
sfc-gh-fpawlowski Jun 22, 2025
122e814
SNOW-2043816: optional accelerate
sfc-gh-fpawlowski Jun 23, 2025
c8eeb5c
SNOW-2043816: connection
sfc-gh-fpawlowski Jun 23, 2025
a2b27ec
SNOW-2043816: network
sfc-gh-fpawlowski Jun 23, 2025
ed1dbf5
SNOW-2043816: files end fix
sfc-gh-fpawlowski Jun 23, 2025
30b5aaf
SNOW-2043816: first removed comments
sfc-gh-fpawlowski Jun 23, 2025
b8abe5c
SNOW-2043816: cleanup
sfc-gh-fpawlowski Jun 23, 2025
3be55aa
SNOW-2043816: chunks tried fix
sfc-gh-fpawlowski Jun 23, 2025
613b014
SNOW-2043816: chunks tried fix
sfc-gh-fpawlowski Jun 23, 2025
cdd70bb
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 23, 2025
0f58476
SNOW-2043816: gcp not support multipart
sfc-gh-fpawlowski Jun 23, 2025
8083a0e
SNOW-2043816: oldd fixed
sfc-gh-fpawlowski Jun 23, 2025
09247e7
SNOW-2043816: oldd fixed
sfc-gh-fpawlowski Jun 23, 2025
2744d44
SNOW-2043816: oldd fixed
sfc-gh-fpawlowski Jun 23, 2025
2637a42
SNOW-2043816: put update
sfc-gh-fpawlowski Jun 23, 2025
dd0ae4f
SNOW-2043816: put update
sfc-gh-fpawlowski Jun 23, 2025
9bdf73a
SNOW-2043816: fix parallelism
sfc-gh-fpawlowski Jun 24, 2025
4fecf43
SNOW-2043816: cloud differences fixde
sfc-gh-fpawlowski Jun 24, 2025
8362eb3
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 24, 2025
bb05162
SNOW-2043816: Refactored Http util behavior
sfc-gh-fpawlowski Jun 24, 2025
bf428db
SNOW-2043816: Check azure issues
sfc-gh-fpawlowski Jun 24, 2025
ead93c1
SNOW-2043816: fix azure issues
sfc-gh-fpawlowski Jun 24, 2025
2672e22
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 24, 2025
e7f6e36
SNOW-2043816: recheck
sfc-gh-fpawlowski Jun 24, 2025
99afcc0
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 24, 2025
89d5a45
SNOW-2043816: merge fix
sfc-gh-fpawlowski Jun 24, 2025
f83b4f0
SNOW-2043816: fix azure
sfc-gh-fpawlowski Jun 24, 2025
828ed88
Merge branch 'main' into SNOW-2043816-Kerberos-Proxy-Auth-Python
sfc-gh-fpawlowski Jun 24, 2025
e1f149e
SNOW-2043816: different puts for multipart and normal
sfc-gh-fpawlowski Jun 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions DESCRIPTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
- Bumped numpy dependency from <2.1.0 to <=2.2.4
- Added Windows support for Python 3.13.
- Add `bulk_upload_chunks` parameter to `write_pandas` function. Setting this parameter to True changes the behaviour of write_pandas function to first write all the data chunks to the local disk and then perform the wildcard upload of the chunks folder to the stage. In default behaviour the chunks are being saved, uploaded and deleted one by one.
- Add `headers_customizers` parameter to the `connect` function. Setting this parameter allows enriching outgoing request headers using a list of customizers. Only header enrichment is supported — modifying query parameters or overwriting the existing headers is not allowed.


- v3.15.1(May 20, 2025)
Expand Down
124 changes: 120 additions & 4 deletions src/snowflake/connector/connection.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,16 @@
from logging import getLogger
from threading import Lock
from types import TracebackType
from typing import Any, Callable, Generator, Iterable, Iterator, NamedTuple, Sequence
from typing import (
Any,
Callable,
Generator,
Iterable,
Iterator,
MutableSequence,
NamedTuple,
Sequence,
)
from uuid import UUID

from cryptography.hazmat.backends import default_backend
Expand Down Expand Up @@ -100,6 +109,11 @@
ER_NOT_IMPLICITY_SNOWFLAKE_DATATYPE,
)
from .errors import DatabaseError, Error, OperationalError, ProgrammingError
from .http_interceptor import (
HeadersCustomizer,
HeadersCustomizerInterceptor,
HttpInterceptor,
)
from .log_configuration import EasyLoggingConfigPython
from .network import (
DEFAULT_AUTHENTICATOR,
Expand All @@ -113,6 +127,9 @@
REQUEST_ID,
USR_PWD_MFA_AUTHENTICATOR,
WORKLOAD_IDENTITY_AUTHENTICATOR,
InterceptingAdapter,
InterceptingAdapterFactory,
ProxySupportAdapter,
ReauthenticationRequest,
SnowflakeRestful,
)
Expand Down Expand Up @@ -364,6 +381,10 @@ def _get_private_bytes_from_file(
True,
bool,
), # SNOW-XXXXX: remove the check_arrow_conversion_error_on_every_column flag
"headers_customizers": (
None,
(type(None), MutableSequence[HeadersCustomizer]),
),
}

APPLICATION_RE = re.compile(r"[\w\d_]+")
Expand Down Expand Up @@ -445,6 +466,7 @@ class SnowflakeConnection:
token_file_path: The file path of the token file. If both token and token_file_path are provided, the token in token_file_path will be used.
unsafe_file_write: When true, files downloaded by GET will be saved with 644 permissions. Otherwise, files will be saved with safe - owner-only permissions: 600.
check_arrow_conversion_error_on_every_column: When true, the error check after the conversion from arrow to python types will happen for every column in the row. This is a new behaviour which fixes the bug that caused the type errors to trigger silently when occurring at any place other than last column in a row. To revert the previous (faulty) behaviour, please set this flag to false.
headers_customizer: List of headers customizers (HeadersCustomizer class). Setting this parameter allows enriching outgoing request headers using a list of customizers. Only header enrichment is supported — modifying query parameters or overwriting the existing headers is not allowed.
"""

OCSP_ENV_LOCK = Lock()
Expand Down Expand Up @@ -854,7 +876,71 @@ def check_arrow_conversion_error_on_every_column(self) -> bool:
def check_arrow_conversion_error_on_every_column(self, value: bool) -> bool:
self._check_arrow_conversion_error_on_every_column = value

@property
def request_interceptors(self) -> MutableSequence[HttpInterceptor]:
return self._request_interceptors

@property
def headers_customizers(self) -> MutableSequence[HeadersCustomizer]:
return self._headers_customizers

@headers_customizers.setter
def headers_customizers(self, value: MutableSequence[HeadersCustomizer]) -> None:
self._headers_customizers = value
# TODO: zrobic to jako dict po typach interceptorów
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left-overs - removing in progress.

# TODO: remove all dto to Info
# TODO: add tests
request_interceptors = self._create_interceptor_for_headers_customizers(value)
self._request_interceptors = (
[
request_interceptors,
]
if request_interceptors
else []
)

def add_headers_customizer(
self, new_customizer: HeadersCustomizer
) -> SnowflakeConnection:
"""
Builder method to add a single headers customizer to the list of headers customizers TODO: finish this descr.
"""
if new_customizer in self._headers_customizers or not new_customizer:
return self

self._headers_customizers.append(new_customizer)
# TODO: czy nie powinien to brac ostatniego header interceptora i do neigo append header customizer
self._request_interceptors.append(
HeadersCustomizerInterceptor([new_customizer])
)
return self

def clear_headers_customizers(self) -> None:
"""
Builder method to add a single headers customizer to the list of headers customizers TODO: finish this descr.
"""
self._headers_customizers.clear()
self._request_interceptors[:] = [
interceptor
for interceptor in self._request_interceptors
if not isinstance(interceptor, HeadersCustomizerInterceptor)
]
# TODO: below or always rely on connection's attributes (properties outside)- this way we can only clear here
# self._rest.headers_customizers.clear()
# self._rest._request_interceptors.clear()

@staticmethod
def _create_interceptor_for_headers_customizers(
headers_customizers: MutableSequence[HeadersCustomizer],
) -> HeadersCustomizerInterceptor | None:
return (
HeadersCustomizerInterceptor(headers_customizers)
if headers_customizers
else None
)

def connect(self, **kwargs) -> None:
...
"""Establishes connection to Snowflake."""
logger.debug("connect")
if len(kwargs) > 0:
Expand Down Expand Up @@ -1144,6 +1230,7 @@ def __open_connection(self):
):
raise TypeError("auth_class must be a child class of AuthByKeyPair")
# TODO: add telemetry for custom auth
# TODO: why this is hre xd
self.auth_class = self.auth_class
elif self._authenticator == DEFAULT_AUTHENTICATOR:
self.auth_class = AuthByDefault(
Expand Down Expand Up @@ -1382,10 +1469,39 @@ def __config(self, **kwargs):
if "host" not in kwargs:
self._host = construct_hostname(kwargs.get("region"), self._account)

if "unsafe_file_write" in kwargs:
self._unsafe_file_write = kwargs["unsafe_file_write"]
self._unsafe_file_write = kwargs.get("unsafe_file_write", False)

self._headers_customizers = kwargs.get("headers_customizers", [])
# # TODO: rethink how adding customizer should work - create new interceptor or append to existing - or just dont do this scenario xd
# TODO: rethink if we should store here in self interceptors or customizers or what
if self._headers_customizers:
header_customizer_interceptor = (
self._create_interceptor_for_headers_customizers(
self._headers_customizers
)
)
request_interceptors = (
[
header_customizer_interceptor,
]
if header_customizer_interceptor
else []
)
InterceptingAdapterFactory.register_for_connection(
connection=self,
adapter_cls=InterceptingAdapter,
interceptors=request_interceptors,
)
else:
self._unsafe_file_write = False
InterceptingAdapterFactory.register_for_connection(
connection=self, adapter_cls=ProxySupportAdapter
)
self._request_interceptors = []

if self._headers_customizers:
logger.info(
f"{len(self._headers_customizers)} custom headers customizers were provided. Requests will be enriched according to the defined conditions."
)

logger.info(
f"Connecting to {_DOMAIN_NAME_MAP.get(extract_top_level_domain_from_hostname(self._host), 'GLOBAL')} Snowflake domain"
Expand Down
3 changes: 3 additions & 0 deletions src/snowflake/connector/file_transfer_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,7 @@ class TransferMetadata:
chunks_in_queue: int = 0


#
class SnowflakeFileTransferAgent:
"""Snowflake File Transfer Agent provides cloud provider independent implementation for putting/getting files."""

Expand Down Expand Up @@ -670,12 +671,14 @@ def _create_file_transfer_client(
self, meta: SnowflakeFileMeta
) -> SnowflakeStorageClient:
if self._stage_location_type == LOCAL_FS:
#
return SnowflakeLocalStorageClient(
meta,
self._stage_info,
4 * megabyte,
unsafe_file_write=self._unsafe_file_write,
)
# clients
elif self._stage_location_type == AZURE_FS:
return SnowflakeAzureRestClient(
meta,
Expand Down
Loading
Loading