Add a walkthrough of building a custom Python connector #36743

girarda · 2024-04-02T01:08:31Z

What

Write a Python source connector development walkthrough covering

reading a page of data
pagination
incremental
integration tests
rate limits
concurrency

This can replace two old tutorials

vercel · 2024-04-02T01:08:38Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 9, 2024 3:33am

natikgadzhi · 2024-04-02T02:12:03Z

Spicy take: should this replace one of the existing tutorials? I don't like having to maintain three.

girarda · 2024-04-05T20:56:10Z

docs/connector-development/README.md

@@ -58,7 +58,7 @@ and choose the relevant template by using the arrow keys. This will generate a n
 Search the generated directory for "TODO"s and follow them to implement your connector. For more detailed walkthroughs and instructions, follow the relevant tutorial:

 - [Speedrun: Building a HTTP source with the CDK](tutorials/cdk-speedrun.md)
- [Building a HTTP source with the CDK](tutorials/cdk-tutorial-python-http/getting-started.md)
+- [Building a HTTP source with the CDK](tutorials/custom-python-connector/0-getting-started.md)


update the path because the cdk-tutorial-python-http is deleted

Is it deleted in this PR? Make sure that you add redirects for all URLs that you delete — redirect to the new guide. That way we don't loose search traffic.

girarda · 2024-04-05T20:56:18Z

docs/connector-development/cdk-python/README.md

@@ -50,7 +50,7 @@ cd airbyte-integrations/connector-templates/generator
 Next, find all `TODO`s in the generated project directory. They're accompanied by comments explaining what you'll
 need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.

-Additionally, you can follow [this tutorial](../tutorials/cdk-tutorial-python-http/getting-started.md) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
+Additionally, you can follow [this tutorial](../tutorials/custom-python-connector/0-getting-started.md) for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.


update the path because the cdk-tutorial-python-http is deleted

girarda · 2024-04-05T20:56:21Z

docs/connector-development/cdk-python/README.md

@@ -72,7 +72,7 @@ Airbyte recommends using the CDK template generator to develop with the CDK. The

 For tips on useful Python knowledge, see the [Python Concepts](python-concepts.md) page.

-You can find a complete tutorial for implementing an HTTP source connector in [this tutorial](../tutorials/cdk-tutorial-python-http/getting-started.md)
+You can find a complete tutorial for implementing an HTTP source connector in [this tutorial](../tutorials/custom-python-connector/0-getting-started.md)


update the path because the cdk-tutorial-python-http is deleted

girarda · 2024-04-05T20:56:26Z

docs/connector-development/config-based/tutorial/4-reading-data.md

@@ -26,7 +26,7 @@ See the [catalog guide](https://docs.airbyte.com/understanding-airbyte/beginners

 Let's define the stream schema in `source-exchange-rates-tutorial/source_exchange_rates_tutorial/schemas/rates.json`

-You can download the JSON file describing the output schema with all currencies [here](../../tutorials/cdk-tutorial-python-http/exchange_rates_schema.json) for convenience and place it in `schemas/`.
+You can download the JSON file describing the output schema with all currencies [here](./exchange_rates_schema.json) for convenience and place it in `schemas/`.


update the path because the cdk-tutorial-python-http is deleted

girarda · 2024-04-05T20:56:48Z

docs/connector-development/config-based/tutorial/6-testing.md

@@ -35,7 +35,7 @@ airbyte-ci connectors --use-remote-secrets=false --name source-exchange-rates-tu

 ## Next steps:

-Next, we'll add the connector to the [Airbyte platform](https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/use-connector-in-airbyte).
+Next, we'll add the connector to the [Airbyte platform](https://docs.airbyte.com/operator-guides/using-custom-connectors).


I realized there's a better guide to using custom connectors

girarda · 2024-04-05T20:59:39Z

docs/connector-development/tutorials/custom-python-connector/2-reading-a-page.md

+```
+rm unit_tests/test_incremental_streams.py unit_tests/test_source.py unit_tests/test_streams.py 
+```
+Replace the content of `airbyte-integrations/connectors/source-survey-monkey-demo/source_survey_monkey_demo/source.py` with the following template:


I'm really not a fan of the code in the template. We can do a better job of encouraging composition instead of inheritance

girarda · 2024-04-05T21:01:40Z

docs/connector-development/tutorials/custom-python-connector/4-check-and-error-handling.md

+
+We'll do this by trying to read a single record from the stream, and fail the connector could not read any.
+```python
+    def check_connection(self, logger, config) -> Tuple[bool, any]:


I feel like the template should provide a sensible implementation by default instead of returning True, None. it's pretty useless

girarda · 2024-04-05T21:02:30Z

docs/connector-development/tutorials/custom-python-connector/6-incremental-reads.md

+
+The `get_updated_state` method is used to update the stream's state. We'll set its value to the maximum between the current state's value and the value extracted from the record.
+```python
+    def get_updated_state(self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]) -> Mapping[str, Any]:


I believe this won't be part of the best practices anymore since RFR won't work with connectors that use get_updated_state

girarda · 2024-04-05T21:04:50Z

docs/connector-development/tutorials/custom-python-connector/8-concurrency.md

+
+Let's update the source. The bulk of the change is changing its parent class to `ConcurrentSourceAdapter`, and updating its `__init__` method so it's properly initialized. This requires a little bit of boilerplate:
+```python
+class SourceSurveyMonkeyDemo(ConcurrentSourceAdapter):


I find the need for the adapter to be pretty sad. I think we're pretty close to being able to recommend using the concurrent cdk interfaces from the get go. We're mostly only missing the concept of a PaginatedRequester.

We did a PoC with Stripe back in October, but never pulled the trigger because the connector had too many issues

girarda · 2024-04-05T21:05:50Z

docs/connector-development/tutorials/custom-python-connector/8-concurrency.md

+                yield parent_record
+```
+
+This can be solved by implementing the connector using constructs from the concurrent CDK directly instead of wrapping synchronous streams in an adapter. This is left outside of the scope of this tutorial because no production connectors currently implement this.


this is a very unfortunate broken window. We never got to productionizing support for concurrent substreams.

girarda · 2024-04-05T21:13:17Z

Build is failing on vercel but passing locally. not sure what the issue is

natikgadzhi

In general, this is great — this will be an epic onboarding guide for Cristina!

A few things to work on:

Formatting
Make sure links we delete redirect to this guide
Make sure to delete speedrun as well
Make issues for further improvements

I will sit down with this later today to get a few changes in and debug why Vercel did not work.

natikgadzhi · 2024-04-08T23:50:28Z

docs/connector-development/README.md

@@ -58,7 +58,7 @@ and choose the relevant template by using the arrow keys. This will generate a n
 Search the generated directory for "TODO"s and follow them to implement your connector. For more detailed walkthroughs and instructions, follow the relevant tutorial:

 - [Speedrun: Building a HTTP source with the CDK](tutorials/cdk-speedrun.md)
- [Building a HTTP source with the CDK](tutorials/cdk-tutorial-python-http/getting-started.md)
+- [Building a HTTP source with the CDK](tutorials/custom-python-connector/0-getting-started.md)


Is it deleted in this PR? Make sure that you add redirects for all URLs that you delete — redirect to the new guide. That way we don't loose search traffic.

natikgadzhi · 2024-04-08T23:51:57Z

docs/connector-development/tutorials/cdk-speedrun.md

@@ -3,7 +3,7 @@
 ## CDK Speedrun \(HTTP API Source Creation Any Route\)

 This is a blazing fast guide to building an HTTP source connector. Think of it as the TL;DR version
-of [this tutorial.](cdk-tutorial-python-http/getting-started.md)
+of [this tutorial.](custom-python-connector/0-getting-started.md)

 If you are a visual learner and want to see a video version of this guide going over each part in


We should urge people to use Connector Builder and low-code for most connectors.

I would say, let's delete this guide as well and point it to the new one. Perhaps we can add a little note on the top of it to direct some users to lowcode?

natikgadzhi · 2024-04-08T23:53:34Z

docs/connector-development/tutorials/custom-python-connector/0-getting-started.md

+# Getting started
+This tutorial will walk you through the creation of a custom Airbyte connector implemented with the Python CDK. This tutorial assumes you're already familiar with Airbyte concept and you've already built a connector using the [Connector Builder](../../connector-builder-ui/tutorial.mdx).
+
+The Python CDK should be used to implement connectors that require features that are not yet available in the Connector Builder or in the low-code framework. You can use the [Connector Builder compatibility guide](../../connector-builder-ui/connector-builder-compatibility.md) to know whether it is suitable for your needs.


docs/connector-development/tutorials/custom-python-connector/0-getting-started.md

natikgadzhi · 2024-04-09T00:01:04Z

docs/connector-development/tutorials/custom-python-connector/0-getting-started.md

+The two approaches are equivalent for the Survey Monkey API, but as a rule of thumb, it is preferable to use the links provided by the API if it is available instead of reverse engineering the mechanism. This way, we don't need to modify the connector if the API changes their pagination mechanism, for instance, if they decide to implement server-side pagination.
+
+:::info
+When available, server-side pagination should be preferred over client-side pagination because it has lower risks of missing records if the collection is modified while the connector iterates.


Not super clear to junior devs what server-side vs client-side pagination is. You're talking about it enough. Server-side — links provided by API, client-side: manually changing page_number or offset.

docs/connector-development/tutorials/custom-python-connector/2-reading-a-page.md

docs/connector-development/tutorials/custom-python-connector/4-check-and-error-handling.md

docs/connector-development/tutorials/custom-python-connector/8-concurrency.md

…nto alex/python_tutorial

natikgadzhi

LGTM overall, I'll look into redirects and merge tonight

Draft tutorial

4d62617

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Apr 2, 2024

vercel bot had a problem deploying to Preview April 2, 2024 01:10 Failure

fix compilation

aaddb91

vercel bot deployed to Preview April 3, 2024 04:57 View deployment

update

c38e9dd

girarda mentioned this pull request Apr 4, 2024

Custom python connector tutorial #36832

Closed

girarda added 3 commits April 5, 2024 13:20

update

59df8c5

delete two old tutorials

a2e5502

info bubble

b1e5b40

vercel bot had a problem deploying to Preview April 5, 2024 21:04 Failure

girarda commented Apr 5, 2024

View reviewed changes

girarda marked this pull request as ready for review April 5, 2024 21:12

girarda requested review from natikgadzhi and katmarkham April 5, 2024 21:12

girarda requested a review from a team April 8, 2024 13:37

natikgadzhi reviewed Apr 9, 2024

View reviewed changes

natikgadzhi added 2 commits April 8, 2024 17:08

Apply suggestions from code review

d6d43d5

prettier

fd94818

vercel bot had a problem deploying to Preview April 9, 2024 00:15 Failure

fix broken json link

6bb220e

vercel bot deployed to Preview April 9, 2024 00:22 View deployment

girarda added 2 commits April 8, 2024 17:46

instructions to run prettier

2ba7efe

Merge branch 'alex/python_tutorial' of github.com:airbytehq/airbyte i…

2c06fd4

…nto alex/python_tutorial

vercel bot deployed to Preview April 9, 2024 00:57 View deployment

natikgadzhi approved these changes Apr 9, 2024

View reviewed changes

redirects for older guide

5c6d0c4

vercel bot had a problem deploying to Preview April 9, 2024 02:44 Failure

fix redirects

c6acf73

vercel bot deployed to Preview April 9, 2024 03:33 View deployment

natikgadzhi merged commit 87f051c into master Apr 9, 2024
25 of 26 checks passed

natikgadzhi deleted the alex/python_tutorial branch April 9, 2024 04:08

natikgadzhi mentioned this pull request Apr 10, 2024

docs: Change connector development overview to link to the new guide #36972

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a walkthrough of building a custom Python connector #36743

Add a walkthrough of building a custom Python connector #36743

girarda commented Apr 2, 2024 •

edited

Loading

vercel bot commented Apr 2, 2024 •

edited

Loading

natikgadzhi commented Apr 2, 2024

girarda Apr 5, 2024

natikgadzhi Apr 8, 2024

girarda Apr 5, 2024

girarda Apr 5, 2024

girarda Apr 5, 2024

girarda Apr 5, 2024

girarda Apr 5, 2024

girarda Apr 5, 2024

girarda Apr 5, 2024

girarda Apr 5, 2024

girarda Apr 5, 2024

girarda commented Apr 5, 2024

natikgadzhi left a comment

natikgadzhi Apr 8, 2024

natikgadzhi Apr 8, 2024

natikgadzhi Apr 8, 2024

natikgadzhi Apr 9, 2024

natikgadzhi left a comment

Add a walkthrough of building a custom Python connector #36743

Add a walkthrough of building a custom Python connector #36743

Conversation

girarda commented Apr 2, 2024 • edited Loading

What

vercel bot commented Apr 2, 2024 • edited Loading

natikgadzhi commented Apr 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

girarda commented Apr 5, 2024

natikgadzhi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natikgadzhi left a comment

Choose a reason for hiding this comment

girarda commented Apr 2, 2024 •

edited

Loading

vercel bot commented Apr 2, 2024 •

edited

Loading