Skip to content

✨ [source-github] Bump cdk version and enable RFR for all non-incremental streams #42966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Aug 9, 2024
Merged

✨ [source-github] Bump cdk version and enable RFR for all non-incremental streams #42966

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0d0a5d2
bump cdk version and enable rfr for all github non-incremental streams
brianjlai Aug 2, 2024
9b37607
bump cdk version with fix for live tests
brianjlai Aug 3, 2024
17b23ca
update version and docs
brianjlai Aug 4, 2024
f596631
fix contributor activity stream to work with RFR and pr feedback
brianjlai Aug 6, 2024
e71de5b
Merge branch 'master' into brian/github_enable_rfr_for_substreams
brianjlai Aug 6, 2024
11cb7e3
a little cleanup
brianjlai Aug 6, 2024
aae21ae
adding mock server tests and fix failing CAT for missing expected record
brianjlai Aug 8, 2024
1d919ca
Merge branch 'master' into brian/github_enable_rfr_for_substreams
brianjlai Aug 8, 2024
21098fe
awesome, more mismatched expected records for another stream workflow…
brianjlai Aug 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 23 additions & 23 deletions airbyte-integrations/connectors/source-github/poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-github/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ repository = "https://github.com/airbytehq/airbyte"
include = "source_github"

[tool.poetry.dependencies]
python = "^3.9,<3.12"
airbyte-cdk = "^3"
python = "^3.10,<3.12"
airbyte-cdk = "^4"
sgqlc = "==16.3"

[tool.poetry.scripts]
Expand Down
4 changes: 4 additions & 0 deletions airbyte-integrations/connectors/source-github/source_github/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from airbyte_cdk.models import AirbyteLogMessage, AirbyteMessage, Level, SyncMode
from airbyte_cdk.models import Type as MessageType
from airbyte_cdk.sources.streams.availability_strategy import AvailabilityStrategy
from airbyte_cdk.sources.streams.checkpoint.substream_resumable_full_refresh_cursor import SubstreamResumableFullRefreshCursor
from airbyte_cdk.sources.streams.core import CheckpointMixin, Stream
from airbyte_cdk.sources.streams.http import HttpStream
from airbyte_cdk.sources.streams.http.error_handlers import ErrorHandler, ErrorResolution, HttpStatusErrorHandler, ResponseAction
Expand Down Expand Up @@ -57,6 +58,9 @@ def __init__(self, api_url: str = "https://api.github.com", access_token_type: s
self.api_url = api_url
self.state = {}

if self.cursor_field or len(self.cursor_field) == 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused by this check - we use the substream RFR cursor if there is a cursor_field or if its length is 0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, sorry i should've commented, but it's supposed to be not self.cursor_field, I reversed it to test something on regression tests, but i'll change it back for final testing when i move it out of draft

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the self.supports_incremental property to be checked?

CDK Code

It could be simply as:

if not self.supports_incremental:
    ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep fair point, will change thanks for the suggestion!

self.cursor = SubstreamResumableFullRefreshCursor()

@property
def url_base(self) -> str:
return self.api_url
Expand Down
Loading