Skip to content

[low-code connectors] Rename decode_response reference to response #14877

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 191 commits into from
Jul 21, 2022
Merged

[low-code connectors] Rename decode_response reference to response #14877

merged 191 commits into from
Jul 21, 2022

Conversation

girarda
Copy link
Contributor

@girarda girarda commented Jul 20, 2022 •
edited
Loading

What

  • Rename references of "decoded_response" to "response" in interpolated strings
"{{ 'codase' in response }}"

How

Describe the solution

Recommended reading order

  1. airbyte-cdk/python/unit_tests/sources/declarative/test_factory.py
  2. airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/http_response_filter.py
  3. airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py
  4. airbyte-cdk/python/unit_tests/sources/declarative/requesters/error_handlers/test_default_error_handler.py
  5. airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/composite_error_handler.py
  6. airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/strategies/cursor_pagination_strategy.py
  7. airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_cursor_pagination_strategy.py
  8. airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/strategies/cursor_pagination_strategy.py
    9.airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_limit_paginator.py

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the connector is published, connector added to connector index as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here
Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here
Connector Generator
  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

girarda added 30 commits June 29, 2022 09:35
@girarda
checkout files from test branch
114daea
@girarda
read_incremental works
6d47a6d
@girarda
reset to master
dbff403
@girarda
remove dead code
24f54db
@girarda
comment
e3f8d50
@girarda
fix
cbe4e57
@girarda
Merge branch 'master' into alex/declarativeStreamIncrementalFix
ca44a97
@girarda
Add test
3d3f321
@girarda
comments
4ea9a2b
@girarda
utc
2f09e79
@girarda
format
67f174d
@girarda
small fix
096b54e
@girarda
Add test with rfc3339
db01257
@girarda
remove unused param
f097ee2
@girarda
Merge branch 'master' into alex/declarativeStreamIncrementalFix
b40eb69
@girarda
fix test
eae5bb5
@girarda
configurable state checkpointing
e1a59aa
@girarda
update test
f534b20
@girarda
start working on retrier
22a2296
@girarda
retry predicate
5ade34d
@girarda
return response status
eef3a2b
@girarda
look in error message
16c33a5
@girarda
cleanup test
355fdc4
@girarda
constant backoff strategy
2b4da45
@girarda
chain backoff strategy
e58b977
@girarda
chain retrier
94b0ab1
@girarda
Add to class types registry
d9d9004
@girarda
extract backoff time from header
6296403
@girarda
wait until
a97d730
@girarda
update
814088a
@github-actions github-actions bot added the CDK Connector Development Kit label Jul 20, 2022
@girarda girarda mentioned this pull request Jul 20, 2022
Merged
37 tasks
@girarda girarda changed the title [low-code connectors] Alex/renametoresponse [low-code connectors] Rename decode_response reference to response Jul 21, 2022
@girarda girarda marked this pull request as ready for review July 21, 2022 06:16
@girarda girarda requested review from a team, sherifnada and brianjlai July 21, 2022 06:16
@girarda girarda merged commit c98f196 into master Jul 21, 2022
@girarda girarda deleted the alex/renametoresponse branch July 21, 2022 09:58
mfsiega-airbyte pushed a commit that referenced this pull request Jul 21, 2022
@girarda @sherifnada
@blarghmatey @grubberr @Amruta-Ranade @bjgbeelen @marcosmarxm @gfeldman @ChristopheDuong @octavia-squidington-iii @YiyangLi @mfsiega-airbyte
[low-code connectors] Rename decode_response reference to response (#…
8a5536c 
…14877)

* checkout files from test branch

* read_incremental works

* reset to master

* remove dead code

* comment

* fix

* Add test

* comments

* utc

* format

* small fix

* Add test with rfc3339

* remove unused param

* fix test

* configurable state checkpointing

* update test

* start working on retrier

* retry predicate

* return response status

* look in error message

* cleanup test

* constant backoff strategy

* chain backoff strategy

* chain retrier

* Add to class types registry

* extract backoff time from header

* wait until

* update

* split file

* parse_records

* classmethod

* delete dead code

* comment

* comment

* comments

* fix

* test for instantiating chain retrier

* fix parsing

* cleanup

* fix

* reset

* never raise on http error

* remove print

* comment

* comment

* comment

* comment

* remove prints

* add declarative stream to registry

* start working on limit paginator

* support for offset pagination

* tests

* move limit value

* extract request option

* boilerplate

* page increment

* delete offset paginator

* update conditional paginator

* refactor and fix test

* fix test

* small fix

* Delete dead code

* Add docstrings

* quick fix

* exponential backoff

* fix test

* fix

* delete unused properties

* fix

* missing unit tests

* uppercase

* docstrings

* rename to success

* compare full request instead of just url

* renmae module

* rename test file

* rename interface

* rename default retrier

* rename to compositeerrorhandler

* fix missing renames

* move action to filter

* str -> minmaxdatetime

* small fixes

* plural

* add example

* handle header variations

* also fix wait time from

* allow using a regex to extract the value

* group()

* docstring

* add docs

* update comment

* docstrings

* fix tests

* rename param

* cleanup stop_condition

* cleanup

* Add examples

* interpolated pagination strategy

* dont need duplicate class

* docstrings

* more docstrings

* docstrings

* update comment

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py

Co-authored-by: Sherif A. Nada <[email protected]>

* version: Update Parquet library to latest release (#14502)

The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commit apache/parquet-java@c72862b

* merge

* 🎉 Source Github: improve schema for stream `pull_request_commits` added "null" (#14613)

Signed-off-by: Sergey Chvalyuk <[email protected]>

* Docs: Fixed broken links (#14622)

* fixing broken links

* more broken links

* source-hubspot: change mentioning of Mailchimp into HubSpot  doc (#14620)

* Helm Chart: Add external temporal option (#14597)

* conflict env configmap and chart lock

* reverting lock

* add eof lines and documentation on values yaml

* conflict json file

* rollback json

* solve conflict

* correct minio with new version

Co-authored-by: Guy Feldman <[email protected]>

* 🎉 Add YAML format to source-file reader (#14588)

* Add yaml reader

* Update docs

* Bumpversion of connector

* bump docs

* Update pyarrow dependency

* Upgrade pandas dependency

* auto-bump connector version

Co-authored-by: Octavia Squidington III <[email protected]>

* 🎉 Source Okta: add GroupMembers stream (#14380)

* add Group_Members stream to okta source

- Group_Members return a list of users, the same schema of Users stream.
- Create a shared schema users, and both group_members and users sechema use it as a reference.
- Add Group_Members stream to source connector

* add tests and fix logs schema

- fix the test error: None is not one of enums though the enum type includes both string and null, it comes from json schema validator
https://github.com/python-jsonschema/jsonschema/blob/ddb87afad8f5d5c40600b5ede0ab96e4d4bdf7d3/jsonschema/_validators.py#L279-L285
- change grouop_members to use id as the cursor field since `filter` is not supported in the query string
- fix the abnormal state test on logs stream, when since is abnormally large, until has to defined, an equal or a larger value
- remove logs stream from full sync test, because 2 full sync always has a gap -- at least a new log about users or groups api.

* last polish before submit the PR

- bump docker version
- update changelog
- add the right abnormal value for logs stream
- correct the sample catalog

* address comments::

- improve comments for until parameter under the logs stream
- add use_cache on groupMembers

* add use_cache to Group_Members

* change configured_catalog to test

* auto-bump connector version

Co-authored-by: marcosmarxm <[email protected]>
Co-authored-by: Octavia Squidington III <[email protected]>

* split test files

* renames

* missing unit test

* add missing unit tests

* rename

* assert isinstance

* start extracting to their own files

* use final instead of classmethod

* assert we retry 429 errors

* Add log

* replace asserts with valueexceptions

* delete superfluous print statement

* fix factory so we don't need to union everything with strings

* get class_name from type

* remove from class types registry

* process error handlers one at a time

* sort

* delete print statement

* comment

* comment

* format

* delete unused file

* comment

* interpolatedboolean

* comment

* not optional

* not optional

* unit tests

* fix request body data

* add test

* move file to right module

* update

* reset to master

* format

* rename to pass_by

* rename to page size

* fix

* add test

* fix body data

* delete extra newlines

* move to subpackage

* fix imports

* handle str body data

* simplify

* fix typing

* always return a map

* rename to inject_into

* only accept enum

* delete conditional paginator

* only return body data

* rename decoded response to response

* decoded_response -> response

Co-authored-by: Sherif A. Nada <[email protected]>
Co-authored-by: Tobias Macey <[email protected]>
Co-authored-by: Serhii Chvaliuk <[email protected]>
Co-authored-by: Amruta Ranade <[email protected]>
Co-authored-by: Bas Beelen <[email protected]>
Co-authored-by: Marcos Marx <[email protected]>
Co-authored-by: Guy Feldman <[email protected]>
Co-authored-by: Christophe Duong <[email protected]>
Co-authored-by: Octavia Squidington III <[email protected]>
Co-authored-by: Yiyang Li <[email protected]>
Co-authored-by: marcosmarxm <[email protected]>
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Jul 23, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sherifnada sherifnada sherifnada approved these changes

@brianjlai brianjlai brianjlai approved these changes

Assignees
No one assigned
Labels
CDK Connector Development Kit
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@girarda @sherifnada @brianjlai @blarghmatey @grubberr @Amruta-Ranade @bjgbeelen @marcosmarxm @ChristopheDuong @YiyangLi