feat(source-file): Add custom http proxy support #62451
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add proxy_url field to HTTPS provider configuration in spec.json - Add ca_certificate field with airbyte_secret=true and multiline=true - Implement proxy support in URLFile._open() method using smart_open transport_params - Implement certificate handling using temporary files and requests verify parameter - Add comprehensive unit tests for proxy and certificate functionality - Maintain backward compatibility with existing configurations Co-Authored-By: AJ Steers <[email protected]>
- Increment version from 0.5.33 to 0.6.0 in pyproject.toml and metadata.yaml - Apply pre-commit formatting fixes (license headers, code style) - Addresses CI failures for version increment and format checks Co-Authored-By: AJ Steers <[email protected]>
- Add changelog entry for HTTP proxy URL and custom CA certificate support - Addresses CI pre-release check failure for missing changelog entry Co-Authored-By: AJ Steers <[email protected]>
- Add comprehensive proxy investigation plan document - Add mitmproxy interception script with logging - Add test configurations for proxy vs direct connection comparison - Resources for new Devin session to investigate why proxy settings appear ignored Co-Authored-By: AJ Steers <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
👋 Greetings, Contributor!Here are some helpful tips and reminders for your convenience. Helpful Resources
PR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
|
|
aaronsteers
changed the base branch from
devin/1749618047-add-http-proxy-support
to
master
aaronsteers
commented
Jun 30, 2025
airbyte-integrations/connectors/source-file/integration_tests/proxy_intercept_script.py
Outdated
Show resolved
Hide resolved
aaronsteers
commented
Jun 30, 2025
airbyte-integrations/connectors/source-file/integration_tests/proxy_intercept_script.py
Outdated
Show resolved
Hide resolved
aaronsteers
commented
Jun 30, 2025
airbyte-integrations/connectors/source-file/proxy_interception_script.py
Outdated
Show resolved
Hide resolved
aaronsteers
commented
Jun 30, 2025
airbyte-integrations/connectors/source-file/proxy_investigation_plan.md
Outdated
Show resolved
Hide resolved
aaronsteers
commented
Jun 30, 2025
airbyte-integrations/connectors/source-file/source_file/client.py
Outdated
Show resolved
Hide resolved
aaronsteers
commented
Jun 30, 2025
airbyte-integrations/connectors/source-file/source_file/client.py
Outdated
Show resolved
Hide resolved
aaronsteers
commented
Jun 30, 2025
aaronsteers
commented
Jun 30, 2025
airbyte-integrations/connectors/source-file/source_file/proxy.py
Outdated
Show resolved
Hide resolved
aaronsteers
commented
Jun 30, 2025
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…/add-custom-proxy-support
4 tasks
…le implementation (do not merge) (#62455) Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: AJ Steers <[email protected]>
|
/build-connector-images
|
There was a problem hiding this comment.
Pull Request Overview
Adds support for custom HTTP proxy configuration (URL and CA certificate) for the file source connector.
- Introduces a new
http_proxytop-level config block inspec.json - Implements helper routines in
proxy.pyto install CA certs and set proxy env vars - Wires up
Clientto accepthttp_proxy, updates tests, docs, metadata, and version
Reviewed Changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| docs/integrations/sources/file.md | Added v0.6.0 release entry for proxy support |
| airbyte-integrations/connectors/source-file/unit_tests/test_proxy_certificate_support.py | New unit tests for proxy URL and CA certificate handling |
| airbyte-integrations/connectors/source-file/source_file/spec.json | Added http_proxy schema block with proxy_url and proxy_ca_certificate |
| airbyte-integrations/connectors/source-file/source_file/proxy.py | New configure_custom_http_proxy, _install_ca_certificate, and constants |
| airbyte-integrations/connectors/source-file/source_file/client.py | Extended Client.__init__ to accept http_proxy and invoke proxy setup |
| airbyte-integrations/connectors/source-file/pyproject.toml | Bumped version to 0.6.0 and added ruff |
| airbyte-integrations/connectors/source-file/metadata.yaml | Updated dockerImageTag to 0.6.0 |
| airbyte-integrations/connectors/source-file/integration_tests/proxy_test_config.json.template | Initial proxy test config template |
| airbyte-integrations/connectors/source-file/integration_tests/proxy_intercept_script.py | MITM-proxy intercept script for integration testing |
Comments suppressed due to low confidence (3)
airbyte-integrations/connectors/source-file/source_file/client.py:263
- The
__init__signature was extended withhttp_proxybut the docstring (or inline comment) wasn't updated to describe this new parameter. Please add or update the constructor doc to explain how to passhttp_proxyand its format.
def __init__(
airbyte-integrations/connectors/source-file/source_file/proxy.py:96
- The docstring references a
connector_config_dict, but the function parameters arehttp_proxy_config,proxy_url, andca_cert_file_text. Please update the docstring to accurately describe these parameters and remove the outdated term.
) -> None:
docs/integrations/sources/file.md:301
- [nitpick] Consider adding a sample
http_proxyconfiguration snippet somewhere in this document so users know how to supplyproxy_urlandproxy_ca_certificatein their connector config.
| 0.6.0 | 2025-07-02 | [61521](https://github.com/airbytehq/airbyte/pull/61521) | Add support for custom Https proxy URL and custom Proxy CA certificate |
airbyte-integrations/connectors/source-file/integration_tests/proxy_test_config.json.template
Outdated
Show resolved
Hide resolved
…proxy_test_config.json.template Co-authored-by: Copilot <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Loom Demo:
https://www.loom.com/share/0d54508a490844999ec9a073f82f4504
What
Adds custom HTTP Proxy support.
Within the
configdictionary, we'll look for:{ ... # other config # Proxy config: "http_proxy": { # Parent element for custom HTTP Proxy configuration "proxy_url": "https://path.to.my/proxy", "proxy_ca_certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n" }Replaces:
TODO
Can this PR be safely reverted and rolled back?