[ISSUE #20044] generate pydantic models from handwritten schema (#20475)

* handwritten low code manifest example components

* add MinMaxDatetime to jsonschema

* add a basic gradle command to generate manifest components

* Add auth components to handwritten component schema

- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth

* Respect optional properties in DeclarativeOauth2Authenticator

* Fix `Dict[str, Any]` mapping in auth components

* add default error handler composite error handler and http response filter components

* [low code component schema] adding backoff strategies to schema

* [low code component schema] fix float types

* [low code component schema] add RecordFilter

* Remove `config` from auth components

* [low code component schema] add Interpolation (with pending question on 'type' not being defined)

* Add CartesianProductStreamSlicer & DatetimeStreamSlicer

* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer

* [low code component schema] add InterpolatedRequestOptionsProvider

* Add slicer components, and fix a couple of components after reviewing output

* [low code component schema] adding transformations and adding type to interpolators

* adding spec and a few small tweaks

* Add DefaultSchemaLoader

* [low code component schema] attempt on custom class

* Add descriptions for auth components

* add RequestOption

* remove interpolated objects from the schema in favor of strings only

* a few schema fixes and adding some custom pagination and stream slicer

* [low code component schema] fix CustomBackoffStrategy

* Add CustomRecordExtractor

* add some description and add additional properties

* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema

* [low code component schema] validating existing schemas

* [low code component schema] clean validation script

* add manifest transformer tests and a few tweaks to the schema

* Revert "[low code component schema] clean validation script"

This reverts commit 2408f41.

* Revert "[low code component schema] validating existing schemas"

This reverts commit 9d39977.

* [low code component schema] integrate validation script to gradle

* [low code component schema] updating validation script permissions

* remove a few model gen spike files and clean up comments

* default types should take parent type into account and a few schema changes

* [ISSUE #20044] generate pydantic models from handwritten schema

* [ISSUE #20044] code review

* [ISSUE #20044] re-generating declarative component schema files

Co-authored-by: brianjlai <[email protected]>
Co-authored-by: Catherine Noll <[email protected]>

Author: 3 people

.pre-commit-config.yaml

@@ -6,7 +6,8 @@ repos:
     rev: v0.8.8
     hooks:
       - id: licenseheaders
-        args: ["--tmpl=LICENSE_SHORT", "--ext=py", "-f"]
+        args:
+          ["--tmpl=LICENSE_SHORT", "--ext=py", "-x=**/models/__init__.py", "-f"]
   - repo: https://github.com/psf/black
     rev: 22.3.0
     hooks:

airbyte-cdk/python/README.md

@@ -66,6 +66,13 @@ pip install -e ".[dev]" # [dev] installs development-only dependencies
 * Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `.mypy.ini`.
 * The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it!
 
+##### Autogenerated files
+If the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run:
+```commandline
+SUB_BUILD=CONNECTORS_BASE ./gradlew format --scan --info --stacktrace
+```
+This will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on pre-commit to the others, you can run the appropriate generation command i.e. `./gradlew generateProtocolClassFiles` or `./gradlew generateComponentManifestClassFiles`.
+
 #### Testing
 
 All tests are located in the `unit_tests` directory. Run `pytest --cov=airbyte_cdk unit_tests/` to run them. This also presents a test coverage report.