Improve Authentication Error handling (#21087)

* refactor checks

* Add Auth Error Checks

* rebase from master

* some weird versioning issue

* 🪟🎉 Connector builder: Substream slicer and cartesian slicer (#20861)

* improve some types

* improve further

* clean up a bit more

* refactor loading state

* move loading state up

* remove isLoading references

* remove unused props and make fetch connector error work

* remove special component for name

* remove top level state for unifinished flows

* start removing uiwidget

* Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss

Co-authored-by: Tim Roes <[email protected]>

* remove undefined option for selected id

* remove unused prop

* fix types

* remove uiwidget state

* clean up

* adjust comment

* handle errors in a nice way

* do not respect default on oneOf fields

* rename to formblock

* reduce re-renders

* pass error to secure inputs

* simplify and improve styling

* align top

* code review

* remove comment

* review comments

* rename file

* be strict about boolean values

* add example

* track form error in error boundary

* review comments

* handle unexpected cases better

* enrich error with connector id

* 🪟🎉 Add copy stream button (#20577)

* add copy stream button

* review comments

* rename prop

* 🪟🎉 Connector builder: Integrate connector form for test input (#20385)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* fix small stuff

* add warning label

* review comments

* adjust translation

Co-authored-by: lmossman <[email protected]>

* use request_body_json instead of request_body_data

* :window: :art: Move `Add` button into the line of Connector Builder key value list fields (#20699)

* move add button into line

* add stories for empty with control, and content + control

* change button name to Control

* 🪟🎉 Connector builder: Allow defining inputs (#20431)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* handle stored form values that don't contain new fields properly

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* 🪟🎉 Connector builder authentication (#20645)

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* fix keys

* 🪟🎉 Connector builder: Session token and oauth authentication (#20712)

* session token and oauth authentication

* fill in session token variable

* typos

* make sure validation error does not go away

* 🪟🎉 Connector builder: Always validate inputs form (#20664)

* validate user input outside of form

* review comments

Co-authored-by: lmossman <[email protected]>

Co-authored-by: lmossman <[email protected]>

* fix merge conflict with dropdown prop being renamed to control

* [Connector Builder] Add paginator (#20698)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* handle stored form values that don't contain new fields properly

* session token and oauth authentication

* fill in session token variable

* fix merge of default values

* add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts

* add cursor and primary key fields to ui

* save

* add page size and token option inputs

* fixes after rebase

* add pagination

* fix pagination types

* handle empty field_name better

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* typos

* make sure validation error does not go away

* make primary key and cursor optional, and reorder

* save toggle group progress

* fix style of toggle label

* handle empty values better

* fix page size/token option field validation and rendering

* handle cursor pagination page size option correctly

Co-authored-by: Joe Reuter <[email protected]>

* [Connector Builder] Add stream slicer (#20748)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* handle stored form values that don't contain new fields properly

* session token and oauth authentication

* fill in session token variable

* fix merge of default values

* add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts

* add cursor and primary key fields to ui

* save

* add page size and token option inputs

* fixes after rebase

* add pagination

* fix pagination types

* handle empty field_name better

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* typos

* make sure validation error does not go away

* make primary key and cursor optional, and reorder

* save toggle group progress

* fix style of toggle label

* handle empty values better

* fix page size/token option field validation and rendering

* handle cursor pagination page size option correctly

* save stream slicer progress

* finish stream slicer

* fix stream slicer fields and validation

Co-authored-by: Joe Reuter <[email protected]>

* debounce form builder values update to reduce load

* 🪟🔧  Connector builder: use new lowcode manifest (#20715)

* use new manifest yaml

* Update airbyte-webapp/src/components/connectorBuilder/types.ts

Co-authored-by: Lake Mossman <[email protected]>

* use updated manifest types

Co-authored-by: Lake Mossman <[email protected]>

* add substream slicer

* add substream and cartesian slicer

* debounce validation as well

* akways show stream test button in error state if there are errors

* fix type of oauth input

* add validation schema for add stream form

* validate all views on test click

* add type to prevent console warning

* do not allow path for substream slicer request option

* do not show request option for substream slicer

* rewrite stream slice field tooltip

Co-authored-by: Tim Roes <[email protected]>
Co-authored-by: lmossman <[email protected]>

* 🪟🔧 Connector builder: Performance improvements (#20620)

* improve some types

* improve further

* clean up a bit more

* refactor loading state

* move loading state up

* remove isLoading references

* remove unused props and make fetch connector error work

* remove special component for name

* remove top level state for unifinished flows

* start removing uiwidget

* Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss

Co-authored-by: Tim Roes <[email protected]>

* remove undefined option for selected id

* remove unused prop

* fix types

* remove uiwidget state

* clean up

* adjust comment

* handle errors in a nice way

* do not respect default on oneOf fields

* rename to formblock

* reduce re-renders

* pass error to secure inputs

* simplify and improve styling

* align top

* code review

* remove comment

* review comments

* rename file

* be strict about boolean values

* add example

* track form error in error boundary

* review comments

* handle unexpected cases better

* speed up some bits

* more changes

* enrich error with connector id

* 🪟🎉 Add copy stream button (#20577)

* add copy stream button

* review comments

* rename prop

* 🪟🎉 Connector builder: Integrate connector form for test input (#20385)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* fix small stuff

* add warning label

* review comments

* adjust translation

Co-authored-by: lmossman <[email protected]>

* use request_body_json instead of request_body_data

* :window: :art: Move `Add` button into the line of Connector Builder key value list fields (#20699)

* move add button into line

* add stories for empty with control, and content + control

* change button name to Control

* 🪟🎉 Connector builder: Allow defining inputs (#20431)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* handle stored form values that don't contain new fields properly

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* 🪟🎉 Connector builder authentication (#20645)

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* fix keys

* 🪟🎉 Connector builder: Session token and oauth authentication (#20712)

* session token and oauth authentication

* fill in session token variable

* typos

* make sure validation error does not go away

* 🪟🎉 Connector builder: Always validate inputs form (#20664)

* validate user input outside of form

* review comments

Co-authored-by: lmossman <[email protected]>

Co-authored-by: lmossman <[email protected]>

* fix merge conflict with dropdown prop being renamed to control

* [Connector Builder] Add paginator (#20698)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* handle stored form values that don't contain new fields properly

* session token and oauth authentication

* fill in session token variable

* fix merge of default values

* add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts

* add cursor and primary key fields to ui

* save

* add page size and token option inputs

* fixes after rebase

* add pagination

* fix pagination types

* handle empty field_name better

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* typos

* make sure validation error does not go away

* make primary key and cursor optional, and reorder

* save toggle group progress

* fix style of toggle label

* handle empty values better

* fix page size/token option field validation and rendering

* handle cursor pagination page size option correctly

Co-authored-by: Joe Reuter <[email protected]>

* [Connector Builder] Add stream slicer (#20748)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* handle stored form values that don't contain new fields properly

* session token and oauth authentication

* fill in session token variable

* fix merge of default values

* add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts

* add cursor and primary key fields to ui

* save

* add page size and token option inputs

* fixes after rebase

* add pagination

* fix pagination types

* handle empty field_name better

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* typos

* make sure validation error does not go away

* make primary key and cursor optional, and reorder

* save toggle group progress

* fix style of toggle label

* handle empty values better

* fix page size/token option field validation and rendering

* handle cursor pagination page size option correctly

* save stream slicer progress

* finish stream slicer

* fix stream slicer fields and validation

Co-authored-by: Joe Reuter <[email protected]>

* debounce form builder values update to reduce load

* 🪟🔧  Connector builder: use new lowcode manifest (#20715)

* use new manifest yaml

* Update airbyte-webapp/src/components/connectorBuilder/types.ts

Co-authored-by: Lake Mossman <[email protected]>

* use updated manifest types

Co-authored-by: Lake Mossman <[email protected]>

* debounce validation as well

* akways show stream test button in error state if there are errors

* fix type of oauth input

* review comments

* fix more

* add validation schema for add stream form

* validate all views on test click

* add type to prevent console warning

* review comment

* make sure testing state and form state stay consistent

* improve builder errors

* remove test state from streamconfig view

* remove console log

* remove unnecessary positive index check

Co-authored-by: Tim Roes <[email protected]>
Co-authored-by: lmossman <[email protected]>

* Remove workspace helper from fetchConfigActivity (#21048)

* Remove workspace helper and replace with workspaceApi

* Publish new version of destination-redshift (#21083)

* Update changelog

* auto-bump connector version

Co-authored-by: Octavia Squidington III <[email protected]>

* Progress Bar Read APIs (#20937)

Follow up PR to #20787 . Make stats available to the read apis so these are available to the webapp.

After this, all that is left is writing these stats as the job progresses.

Add the required logic in JobHistoryHandler.java.

Took the chance to also rename our internal Attempt models field from id to attemptNumber to better reflect that the field stores not the row's database id, but the job's attempt number. Most of the files changes here are due to that rename.

* Rm temporal version (#21045)

* Rm temporal version

* Remove temporal version

* Update the replayed workflow

* Format

* Fix pmd

* 🪟🎉 Connector builder: Available inputs dropdown (#20983)

* improve some types

* improve further

* clean up a bit more

* refactor loading state

* move loading state up

* remove isLoading references

* remove unused props and make fetch connector error work

* remove special component for name

* remove top level state for unifinished flows

* start removing uiwidget

* Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss

Co-authored-by: Tim Roes <[email protected]>

* remove undefined option for selected id

* remove unused prop

* fix types

* remove uiwidget state

* clean up

* adjust comment

* handle errors in a nice way

* do not respect default on oneOf fields

* rename to formblock

* reduce re-renders

* pass error to secure inputs

* simplify and improve styling

* align top

* code review

* remove comment

* review comments

* rename file

* be strict about boolean values

* add example

* track form error in error boundary

* review comments

* handle unexpected cases better

* enrich error with connector id

* 🪟🎉 Add copy stream button (#20577)

* add copy stream button

* review comments

* rename prop

* 🪟🎉 Connector builder: Integrate connector form for test input (#20385)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* fix small stuff

* add warning label

* review comments

* adjust translation

Co-authored-by: lmossman <[email protected]>

* use request_body_json instead of request_body_data

* :window: :art: Move `Add` button into the line of Connector Builder key value list fields (#20699)

* move add button into line

* add stories for empty with control, and content + control

* change button name to Control

* 🪟🎉 Connector builder: Allow defining inputs (#20431)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* handle stored form values that don't contain new fields properly

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* 🪟🎉 Connector builder authentication (#20645)

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* fix keys

* 🪟🎉 Connector builder: Session token and oauth authentication (#20712)

* session token and oauth authentication

* fill in session token variable

* typos

* make sure validation error does not go away

* 🪟🎉 Connector builder: Always validate inputs form (#20664)

* validate user input outside of form

* review comments

Co-authored-by: lmossman <[email protected]>

Co-authored-by: lmossman <[email protected]>

* fix merge conflict with dropdown prop being renamed to control

* [Connector Builder] Add paginator (#20698)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* handle stored form values that don't contain new fields properly

* session token and oauth authentication

* fill in session token variable

* fix merge of default values

* add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts

* add cursor and primary key fields to ui

* save

* add page size and token option inputs

* fixes after rebase

* add pagination

* fix pagination types

* handle empty field_name better

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* typos

* make sure validation error does not go away

* make primary key and cursor optional, and reorder

* save toggle group progress

* fix style of toggle label

* handle empty values better

* fix page size/token option field validation and rendering

* handle cursor pagination page size option correctly

Co-authored-by: Joe Reuter <[email protected]>

* [Connector Builder] Add stream slicer (#20748)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* handle stored form values that don't contain new fields properly

* session token and oauth authentication

* fill in session token variable

* fix merge of default values

* add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts

* add cursor and primary key fields to ui

* save

* add page size and token option inputs

* fixes after rebase

* add pagination

* fix pagination types

* handle empty field_name better

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <[email protected]>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <[email protected]>

* inputs editing weirdness

* input form reset

* using the Label component

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* typos

* make sure validation error does not go away

* make primary key and cursor optional, and reorder

* save toggle group progress

* fix style of toggle label

* handle empty values better

* fix page size/token option field validation and rendering

* handle cursor pagination page size option correctly

* save stream slicer progress

* finish stream slicer

* fix stream slicer fields and validation

Co-authored-by: Joe Reuter <[email protected]>

* debounce form builder values update to reduce load

* 🪟🔧  Connector builder: use new lowcode manifest (#20715)

* use new manifest yaml

* Update airbyte-webapp/src/components/connectorBuilder/types.ts

Co-authored-by: Lake Mossman <[email protected]>

* use updated manifest types

Co-authored-by: Lake Mossman <[email protected]>

* debounce validation as well

* akways show stream test button in error state if there are errors

* fix type of oauth input

* available inputs dropdown

* add validation schema for add stream form

* validate all views on test click

* add type to prevent console warning

* improve styling

* make sure padding is set correctly

* make sure focus is set right

* comment

* use correct state hook

* add tooltip to user input button and fix wording for new user input option

Co-authored-by: Tim Roes <[email protected]>
Co-authored-by: lmossman <[email protected]>

* Fixed appstore docs link (#21098)

* fill in all default values on switch (#21059)

* 🐛Source Freshdesk: Fix schema types (#20416)

* Fix schema types

* Updated version

* Updated acceptance tests

* auto-bump connector version

Co-authored-by: Octavia Squidington III <[email protected]>

* CDK: Add schema inferrer class (#20941)

* fix stuff

* Update schema_inferrer.py

* Update schema_inferrer.py

* bump version

* review comments

* code style

* fix formatting

* improve tests

* Source Facebook Marketing: Docs update (#21105)

* Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (partial revert of #20662) (#21082)

* Revert "Source Amazon Ads: fix bug with handling: "Report date is too far in the past." (#20662)"

This reverts commit ec995959f7af3574cd7a6a5e63d2c2aaf3118183.

* fix

Signed-off-by: Sergey Chvalyuk <[email protected]>

* revert master

Signed-off-by: Sergey Chvalyuk <[email protected]>

* revert to master

Signed-off-by: Sergey Chvalyuk <[email protected]>

* bump 0.1.27

Signed-off-by: Sergey Chvalyuk <[email protected]>

* fix

Signed-off-by: Sergey Chvalyuk <[email protected]>

* amazon-ads.md updated

Signed-off-by: Sergey Chvalyuk <[email protected]>

* amazon-ads.md updated

Signed-off-by: Sergey Chvalyuk <[email protected]>

* auto-bump connector version

Signed-off-by: Sergey Chvalyuk <[email protected]>
Co-authored-by: Augustin <[email protected]>
Co-authored-by: Octavia Squidington III <[email protected]>
Co-authored-by: Topher Lubaway <[email protected]>

* Revert "Progress Bar Read APIs (#20937)" (#21115)

Breaks when there is no config present
https://github.com/airbytehq/airbyte/issues/21112
This reverts commit 3a2b0405c425562ba5fa8bfe3f5a4646c165591a.

* Remove unneeded margin top and bottom (#21111)

* feat(Platform): update actor configuration when receiving control messages from connectors during sync (#19811)

* track latest config message

* pass new config as part of outputs

* persist new config

* persist config as the messages come through, dont set output

* clean up old implementation

* accept control messages for destinations

* get api client from micronaut

* mask instance-wide oauth params when updating configs

* defaultreplicationworker tests

* formatting

* tests for source/destination handlers

* rm todo

* refactor test a bit to fix pmd

* fix pmd

* fix test

* add PersistConfigHelperTest

* update message tracker comment

* fix pmd

* format

* move ApiClientBeanFactory to commons-worker, use in container-orchestrator

* pull out config updating to separate methods

* add jitter

* rename PersistConfigHelper -> UpdateConnectorConfigHelper, docs

* fix exception type

* fmt

* move message type check into runnable

* formatting

* pass api client env vars to container orchestrator

* pass micronaut envs to container orchestrator

* print stacktrace for debugging

* different api host for container orchestrator

* fix default env var

* format

* fix errors after merge

* set source and destination actor id as part of the sync input

* fix: get destination definition

* fix null ptr

* remove "actor" from naming

* fix missing change from rename

* revert ContainerOrchestratorConfigBeanFactory changes

* inject sourceapi/destinationapi directly rather than airbyteapiclient

* UpdateConnectorConfigHelper -> ConnectorConfigUpdater

* rm log

* fix test

* dont fail on config update error

* pass id, not full config to runnables/accept control message

* add new config required for api client

* add test file

* fix test compatibility

* mount data plane credentials secret to container orchestrator (#20724)

* mount data plane credentials secret to container orchestrator

* rm copy-pasta

* properly handle empty strings

* set env vars like before

* use the right config vars

* 🐛Source Looker: Fix schema transformation issue (#20182)

* Fix schema transformation issue

* Updated PR number

* Added unittest

* Add a test case that uses the recursion

* Unhide from cloud

* Bumed seed version

* [ISSUE #20322] add datetime_granularity logic to DatetimeStreamSlicer… (#20717)

* [ISSUE #20322] add datetime_granularity logic to DatetimeStreamSlicer and migrate duration to ISO8601

* [ISSUE #20322] fix tests

* [ISSUE #20322] code review based on clnoll's comments and fixed tests

* [ISSUE #20322] fix flake8 error

* [ISSUE #20322] fix source tests

* [ISSUE #20322] fixing yet another error in source

* [ISSUE #20322] code review

* [ISSUE #20322] adding new sources using datetime slicer

* [ISSUE #20322] fixing source-datascope and increasing version

* [ISSUE #20322] regenerate component schema

* [ISSUE #20322] fixing source-datascope

* [ISSUE #20322] extra field error

* Source Slack: update schema; data from openapi spec (#20767)

* Source Slack: update schema; data from openapi spec

* Source Slack: Docs update

* Source Slack: Update schema

* Source Slack: Update schema (timestamp returned in POSIX time)

* Source Slack: Disable backward compatibility test for discovery

* auto-bump connector version

Co-authored-by: Octavia Squidington III <[email protected]>

* make sure name is rendered in first position (#21101)

* Source Github: Raise Error if no organizations or repos are available (#21084)

* Source Github: Raise Error if no organizations or repos are available

* Source GitHub: Update docs; bump version

* Source GitHub: Add test

* Source GitHub: Exclude empty streams

* auto-bump connector version

Co-authored-by: Octavia Squidington III <[email protected]>

* Convert the server to micronaut (#19194)

* Extract Operation API

* Extract scheduler API

* Format

* extract source api

* Extract source definition api

* Add path

* Extract State API

* extract webbackend api

* extract webbackend api

* extract workspace api

* Extract source definition specification api

* Remove configuration API

* tmp

* Checkstyle

* tmp

* tmp

* Inject but don't resolve Bean

* tmp

* Tmp

* fix build

* TMP

* Tmp

* Clean up

* better thread pool

* Change port to 8080

* Fix port

* Rm unused

* Cors filter

* Format

* rename

* Tmp

* Config based

* Rm health controller ref

* tmp

* Pool size

* Mock healthcheck

* Revert "Mock healthcheck"

This reverts commit 466677636bda96bed7bbfd354a59abc38ff1e315.

* Revert "Revert "Mock healthcheck""

This reverts commit 267094ae149f3c077bdba6f71646d99655ce81a0.

* Restore health check

* Tmp

* format

* Rm deprecated

* Fix PMD

* Tmp

* Fix proxy test

* Remove useless annotation

* set auto commit as false

* Clean up and PR comments

* Bmoric/convert attempt micronaut (#19847)

* tmp

* Fix build

* tmp

* Tmp

* tmp

* tmp

* Tmp

* tmp

* tmp

* Clean up

* Comments and banner

* Non related files

* rm tmp

* Fix build

* Format

* Hit the micronaut server directly

* micronaut OperationApiController (#20270)

* micronaut OperationApiController

* pass micronaut client to OperationApi

* Bmoric/convert connection micronaut (#20211)

* tmp

* Fix build

* tmp

* Tmp

* tmp

* tmp

* Tmp

* tmp

* tmp

* Clean up

* tmp

* Convert Connection Api Controller

* PR Comments

* convert openapiapicontroller to micronaut (#20258)

* convert openapiapicontroller to micronaut

* merge health/openapi locations into one entry

* Fix build

* Format

* Remove media type

* Format

Co-authored-by: Cole Snodgrass <[email protected]>

* Bmoric/convert destination controller micronaut (#20269)

* tmp

* Fix build

* tmp

* Tmp

* tmp

* tmp

* Tmp

* tmp

* tmp

* Clean up

* tmp

* Convert Connection Api Controller

* Tmp

* PR Comments

* convert openapiapicontroller to micronaut (#20258)

* convert openapiapicontroller to micronaut

* merge health/openapi locations into one entry

* Fix bean

* Add JsonSchemaValidator as a Bean

* Fix build

* Format

* Format

* Test fix

* Pr comments

* Remove media type

* Format

* Remove media type

* Format

* format

* Add missing airbyte api client

Co-authored-by: Cole Snodgrass <[email protected]>

* Bmoric/convert destination definition controller micronaut (#20277)

* tmp

* Fix build

* tmp

* Tmp

* tmp

* tmp

* Tmp

* tmp

* tmp

* Clean up

* tmp

* Convert Connection Api Controller

* Tmp

* PR Comments

* convert openapiapicontroller to micronaut (#20258)

* convert openapiapicontroller to micronaut

* merge health/openapi locations into one entry

* Fix bean

* Add JsonSchemaValidator as a Bean

* Fix build

* Format

* Format

* Test fix

* Pr comments

* Remove media type

* Format

* Remove media type

* Format

* Remove media type

* Format

* api client

* missing annotation

* format

Co-authored-by: Cole Snodgrass <[email protected]>

* convert StateApiController to Micronaut (#20329)

* convert to micronaut

* nginx updates

* format

* Move dest oauth to micronaut (#20318)

* tmp

* Fix build

* tmp

* Tmp

* tmp

* tmp

* Tmp

* tmp

* tmp

* Clean up

* tmp

* Convert Connection Api Controller

* PR Comments

* convert openapiapicontroller to micronaut (#20258)

* convert openapiapicontroller to micronaut

* merge health/openapi locations into one entry

* Fix build

* Format

* Remove media type

* Format

* Move dest oauth to micronaut

* Pr comments

* format

Co-authored-by: Cole Snodgrass <[email protected]>

* Bmoric/convert source micronaut (#20334)

* tmp

* Fix build

* tmp

* Tmp

* tmp

* tmp

* Tmp

* tmp

* tmp

* Clean up

* tmp

* Convert Connection Api Controller

* PR Comments

* convert openapiapicontroller to micronaut (#20258)

* convert openapiapicontroller to micronaut

* merge health/openapi locations into one entry

* Fix build

* Format

* Remove media type

* Format

* Tmp

* tmp

* Build

* missing bean

* format

Co-authored-by: Cole Snodgrass <[email protected]>

* Migrate to micronaut (#20339)

* Migrate source to micronaut

* convert SchedulerApiController to Micronaut (#20337)

* wip; SchedulerApiController

* remove @Named

* remove @Singleton

* add back todo message

* Bmoric/convert source definition micronaut (#20338)

* tmp

* Fix build

* tmp

* Tmp

* tmp

* tmp

* Tmp

* tmp

* tmp

* Clean up

* tmp

* Convert Connection Api Controller

* PR Comments

* convert openapiapicontroller to micronaut (#20258)

* convert openapiapicontroller to micronaut

* merge health/openapi locations into one entry

* Fix build

* Format

* Remove media type

* Format

* Tmp

* tmp

* Build

* missing bean

* Tmp

* Add Beans

* fix Bean

* Add passthrough

* Clean up

* Missing path

* FIx typo

* Fix conflicts

* for mat

Co-authored-by: Cole Snodgrass <[email protected]>

* update SourceOauthApiController to Micronaut (#20386)

* convert SourceOauthApiController to Micronaut

* remove SourceOauthApi reference

* convert WorkspaceApiController to micronaut (#20214)

…

Author: 58 people

airbyte-config/init/src/main/resources/seed/destination_definitions.yaml

@@ -40,7 +40,7 @@
 - name: BigQuery
   destinationDefinitionId: 22f6c74f-5699-40ff-833c-4a879ea40133
   dockerRepository: airbyte/destination-bigquery
-  dockerImageTag: 1.2.11
+  dockerImageTag: 1.2.12
   documentationUrl: https://docs.airbyte.com/integrations/destinations/bigquery
   icon: bigquery.svg
   normalizationConfig:
@@ -139,7 +139,7 @@
 - name: Google Cloud Storage (GCS)
   destinationDefinitionId: ca8f6566-e555-4b40-943a-545bf123117a
   dockerRepository: airbyte/destination-gcs
-  dockerImageTag: 0.2.12
+  dockerImageTag: 0.2.13
   documentationUrl: https://docs.airbyte.com/integrations/destinations/gcs
   icon: googlecloudstorage.svg
   resourceRequirements:
@@ -290,7 +290,7 @@
 - name: Redshift
   destinationDefinitionId: f7a7d195-377f-cf5b-70a5-be6b819019dc
   dockerRepository: airbyte/destination-redshift
-  dockerImageTag: 0.3.53
+  dockerImageTag: 0.3.54
   documentationUrl: https://docs.airbyte.com/integrations/destinations/redshift
   icon: redshift.svg
   normalizationConfig:
@@ -321,7 +321,7 @@
 - name: S3
   destinationDefinitionId: 4816b78f-1489-44c1-9060-4b19d5fa9362
   dockerRepository: airbyte/destination-s3
-  dockerImageTag: 0.3.18
+  dockerImageTag: 0.3.19
   documentationUrl: https://docs.airbyte.com/integrations/destinations/s3
   icon: s3.svg
   resourceRequirements:
@@ -348,7 +348,7 @@
 - name: Snowflake
   destinationDefinitionId: 424892c4-daac-4491-b35d-c6688ba547ba
   dockerRepository: airbyte/destination-snowflake
-  dockerImageTag: 0.4.43
+  dockerImageTag: 0.4.44
   documentationUrl: https://docs.airbyte.com/integrations/destinations/snowflake
   icon: snowflake.svg
   normalizationConfig:

airbyte-config/init/src/main/resources/seed/destination_specs.yaml

@@ -621,7 +621,7 @@
     supported_destination_sync_modes:
     - "overwrite"
     - "append"
-- dockerImage: "airbyte/destination-bigquery:1.2.11"
+- dockerImage: "airbyte/destination-bigquery:1.2.12"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/destinations/bigquery"
     connectionSpecification:
@@ -2325,7 +2325,7 @@
     supported_destination_sync_modes:
     - "overwrite"
     - "append"
-- dockerImage: "airbyte/destination-gcs:0.2.12"
+- dockerImage: "airbyte/destination-gcs:0.2.13"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/destinations/gcs"
     connectionSpecification:
@@ -5123,7 +5123,7 @@
     supported_destination_sync_modes:
     - "overwrite"
     - "append"
-- dockerImage: "airbyte/destination-redshift:0.3.53"
+- dockerImage: "airbyte/destination-redshift:0.3.54"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/destinations/redshift"
     connectionSpecification:
@@ -5492,7 +5492,7 @@
     supported_destination_sync_modes:
     - "append"
     - "overwrite"
-- dockerImage: "airbyte/destination-s3:0.3.18"
+- dockerImage: "airbyte/destination-s3:0.3.19"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/destinations/s3"
     connectionSpecification:
@@ -6109,7 +6109,7 @@
     supported_destination_sync_modes:
     - "overwrite"
     - "append"
-- dockerImage: "airbyte/destination-snowflake:0.4.43"
+- dockerImage: "airbyte/destination-snowflake:0.4.44"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/destinations/snowflake"
     connectionSpecification:

airbyte-integrations/bases/base-java-s3/src/main/java/io/airbyte/integrations/destination/s3/BlobStorageOperations.java

@@ -22,9 +22,9 @@ protected BlobStorageOperations() {
   public abstract String getBucketObjectPath(String namespace, String streamName, DateTime writeDatetime, String customFormat);
 
   /**
-   * Create a storage object where to store data in the destination for a @param objectPath
+   * Ensure that the bucket specified in the config exists
    */
-  public abstract void createBucketObjectIfNotExists(String objectPath) throws Exception;
+  public abstract void createBucketIfNotExists() throws Exception;
 
   /**
    * Upload the data files into the storage area.

airbyte-integrations/bases/base-java-s3/src/main/java/io/airbyte/integrations/destination/s3/S3BaseChecks.java

@@ -120,7 +120,7 @@ private static void attemptWriteAndDeleteS3Object(final S3StorageOperations stor
     final var bucketPath = s3Config.getBucketPath();
 
     if (!Strings.isNullOrEmpty(bucketPath)) {
-      storageOperations.createBucketObjectIfNotExists(bucketPath);
+      storageOperations.createBucketIfNotExists();
     }
     s3.putObject(s3Bucket, outputTableName, "check-content");
     testIAMUserHasListObjectPermission(s3, s3Bucket);

airbyte-integrations/bases/base-java-s3/src/main/java/io/airbyte/integrations/destination/s3/S3StorageOperations.java

@@ -8,19 +8,22 @@
 
 import alex.mojaki.s3upload.StreamTransferManager;
 import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.AmazonS3Exception;
 import com.amazonaws.services.s3.model.DeleteObjectsRequest;
 import com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion;
 import com.amazonaws.services.s3.model.ListObjectsRequest;
 import com.amazonaws.services.s3.model.ObjectListing;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ImmutableMap;
+import io.airbyte.commons.exceptions.ConfigErrorException;
 import io.airbyte.commons.string.Strings;
 import io.airbyte.integrations.destination.NamingConventionTransformer;
 import io.airbyte.integrations.destination.record_buffer.SerializableBuffer;
 import io.airbyte.integrations.destination.s3.template.S3FilenameTemplateManager;
 import io.airbyte.integrations.destination.s3.template.S3FilenameTemplateParameterObject;
 import io.airbyte.integrations.destination.s3.util.StreamTransferManagerFactory;
+import io.airbyte.integrations.util.ConnectorExceptionUtil;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
@@ -95,23 +98,15 @@ public String getBucketObjectPath(final String namespace, final String streamNam
 
   /**
    * Create a directory object at the specified location. Creates the bucket if necessary.
-   *
-   * @param objectPath The directory to create. Must be a nonempty string.
    */
   @Override
-  public void createBucketObjectIfNotExists(final String objectPath) {
+  public void createBucketIfNotExists() {
     final String bucket = s3Config.getBucketName();
-    final String folderPath = objectPath.endsWith("/") ? objectPath : objectPath + "/";
     if (!doesBucketExist(bucket)) {
       LOGGER.info("Bucket {} does not exist; creating...", bucket);
       s3Client.createBucket(bucket);
       LOGGER.info("Bucket {} has been created.", bucket);
     }
-    if (!s3Client.doesObjectExist(bucket, folderPath)) {
-      LOGGER.info("Storage Object {}/{} does not exist in bucket; creating...", bucket, objectPath);
-      s3Client.putObject(bucket, folderPath, "");
-      LOGGER.info("Storage Object {}/{} has been created in bucket.", bucket, objectPath);
-    }
   }
 
   protected boolean doesBucketExist(final String bucket) {
@@ -138,7 +133,18 @@ public String uploadRecordsToBucket(final SerializableBuffer recordsData,
         exceptionsThrown.add(e);
       }
     }
-    throw new RuntimeException(String.format("Exceptions thrown while uploading records into storage: %s", Strings.join(exceptionsThrown, "\n")));
+    // Verifying that ALL exceptions are authentication related before assuming this is a configuration
+    // issue
+    // reduces risk of misidentifying errors or reporting a transient error.
+    final boolean areAllExceptionsAuthExceptions = exceptionsThrown.stream().filter(e -> e instanceof AmazonS3Exception)
+        .map(s3e -> ((AmazonS3Exception) s3e).getStatusCode())
+        .filter(ConnectorExceptionUtil.HTTP_AUTHENTICATION_ERROR_CODES::contains)
+        .count() == exceptionsThrown.size();
+    if (areAllExceptionsAuthExceptions) {
+      throw new ConfigErrorException(exceptionsThrown.get(0).getMessage(), exceptionsThrown.get(0));
+    } else {
+      throw new RuntimeException(String.format("Exceptions thrown while uploading records into storage: %s", Strings.join(exceptionsThrown, "\n")));
+    }
   }
 
   /**

airbyte-integrations/bases/base-java/src/main/java/io/airbyte/integrations/util/ConnectorExceptionUtil.java

@@ -4,6 +4,7 @@
 
 package io.airbyte.integrations.util;
 
+import com.google.common.collect.ImmutableList;
 import io.airbyte.commons.exceptions.ConfigErrorException;
 import io.airbyte.commons.exceptions.ConnectionErrorException;
 import io.airbyte.integrations.base.errors.messages.ErrorMessage;
@@ -22,6 +23,8 @@ public class ConnectorExceptionUtil {
   static final String RECOVERY_CONNECTION_ERROR_MESSAGE =
       "We're having issues syncing from a Postgres replica that is configured as a hot standby server. " +
           "Please see https://docs.airbyte.com/integrations/sources/postgres/#sync-data-from-postgres-hot-standby-server for options and workarounds";
+
+  public static final List<Integer> HTTP_AUTHENTICATION_ERROR_CODES = ImmutableList.of(401, 403);
   private static final List<Predicate<Throwable>> configErrorPredicates =
       List.of(getConfigErrorPredicate(), getConnectionErrorPredicate(),
           isRecoveryConnectionExceptionPredicate(), isUnknownColumnInFieldListException());

airbyte-integrations/connectors/destination-bigquery-denormalized/Dockerfile

@@ -17,5 +17,5 @@ ENV ENABLE_SENTRY true
 
 COPY --from=build /airbyte /airbyte
 
-LABEL io.airbyte.version=1.2.11
+LABEL io.airbyte.version=1.2.12
 LABEL io.airbyte.name=airbyte/destination-bigquery-denormalized

airbyte-integrations/connectors/destination-bigquery/Dockerfile

@@ -17,5 +17,5 @@ ENV ENABLE_SENTRY true
 
 COPY --from=build /airbyte /airbyte
 
-LABEL io.airbyte.version=1.2.11
+LABEL io.airbyte.version=1.2.12
 LABEL io.airbyte.name=airbyte/destination-bigquery

airbyte-integrations/connectors/destination-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryGcsOperations.java

@@ -13,11 +13,14 @@
 import com.google.cloud.bigquery.LoadJobConfiguration;
 import com.google.cloud.bigquery.Schema;
 import com.google.cloud.bigquery.TableId;
+import com.google.common.collect.ImmutableList;
+import io.airbyte.commons.exceptions.ConfigErrorException;
 import io.airbyte.integrations.destination.StandardNameTransformer;
 import io.airbyte.integrations.destination.bigquery.uploader.AbstractBigQueryUploader;
 import io.airbyte.integrations.destination.gcs.GcsDestinationConfig;
 import io.airbyte.integrations.destination.gcs.GcsStorageOperations;
 import io.airbyte.integrations.destination.record_buffer.SerializableBuffer;
+import io.airbyte.integrations.util.ConnectorExceptionUtil;
 import io.airbyte.protocol.models.v0.DestinationSyncMode;
 import java.util.HashSet;
 import java.util.List;
@@ -84,7 +87,15 @@ public String getStagingFullPath(final String datasetId, final String stream) {
   public void createSchemaIfNotExists(final String datasetId, final String datasetLocation) {
     if (!existingSchemas.contains(datasetId)) {
       LOGGER.info("Creating dataset {}", datasetId);
-      BigQueryUtils.getOrCreateDataset(bigQuery, datasetId, datasetLocation);
+      try {
+        BigQueryUtils.getOrCreateDataset(bigQuery, datasetId, datasetLocation);
+      } catch (BigQueryException e) {
+        if (ConnectorExceptionUtil.HTTP_AUTHENTICATION_ERROR_CODES.contains(e.getCode())) {
+          throw new ConfigErrorException(e.getMessage(), e);
+        } else {
+          throw e;
+        }
+      }
       existingSchemas.add(datasetId);
     }
   }
@@ -99,7 +110,7 @@ public void createTmpTableIfNotExists(final TableId tmpTableId, final Schema tab
   public void createStageIfNotExists(final String datasetId, final String stream) {
     final String objectPath = getStagingFullPath(datasetId, stream);
     LOGGER.info("Creating staging path for stream {} (dataset {}): {}", stream, datasetId, objectPath);
-    gcsStorageOperations.createBucketObjectIfNotExists(objectPath);
+    gcsStorageOperations.createBucketIfNotExists();
   }
 
   @Override

airbyte-integrations/connectors/destination-bigquery/src/test-integration/java/io/airbyte/integrations/destination/bigquery/BigQueryDestinationTest.java

@@ -84,8 +84,10 @@ class BigQueryDestinationTest {
   protected static final Path CREDENTIALS_WITH_GCS_STAGING_PATH =
       Path.of("secrets/credentials-gcs-staging.json");
 
-  protected static final Path[] ALL_PATHS = {CREDENTIALS_WITH_GCS_STAGING_PATH, CREDENTIALS_BAD_PROJECT_PATH, CREDENTIALS_NO_DATASET_CREATION_PATH,
+
+  protected static final Path[] ALL_PATHS = {CREDENTIALS_STANDARD_INSERT_PATH, CREDENTIALS_BAD_PROJECT_PATH, CREDENTIALS_NO_DATASET_CREATION_PATH,
     CREDENTIALS_NO_EDIT_PUBLIC_SCHEMA_ROLE_PATH, CREDENTIALS_NON_BILLABLE_PROJECT_PATH, CREDENTIALS_WITH_GCS_STAGING_PATH};
+
   private static final Logger LOGGER = LoggerFactory.getLogger(BigQueryDestinationTest.class);
   private static final String DATASET_NAME_PREFIX = "bq_dest_integration_test";
 

airbyte-integrations/connectors/destination-gcs/Dockerfile

@@ -16,5 +16,5 @@ ENV APPLICATION destination-gcs
 
 COPY --from=build /airbyte /airbyte
 
-LABEL io.airbyte.version=0.2.12
+LABEL io.airbyte.version=0.2.13
 LABEL io.airbyte.name=airbyte/destination-gcs

airbyte-integrations/connectors/destination-redshift/Dockerfile

@@ -16,5 +16,5 @@ ENV APPLICATION destination-redshift
 
 COPY --from=build /airbyte /airbyte
 
-LABEL io.airbyte.version=0.3.53
+LABEL io.airbyte.version=0.3.54
 LABEL io.airbyte.name=airbyte/destination-redshift

airbyte-integrations/connectors/destination-redshift/src/main/java/io/airbyte/integrations/destination/redshift/operations/RedshiftS3StagingSqlOperations.java

@@ -77,7 +77,7 @@ public String getStagingPath(UUID connectionId, String namespace, String streamN
   public void createStageIfNotExists(JdbcDatabase database, String stageName) throws Exception {
     final String bucketPath = s3Config.getBucketPath();
     final String prefix = bucketPath.isEmpty() ? "" : bucketPath + (bucketPath.endsWith("/") ? "" : "/");
-    s3StorageOperations.createBucketObjectIfNotExists(prefix + stageName);
+    s3StorageOperations.createBucketIfNotExists();
   }
 
   @Override

airbyte-integrations/connectors/destination-s3/Dockerfile

@@ -40,5 +40,5 @@ RUN /bin/bash -c 'set -e && \
        echo "unknown arch" ;\
     fi'
 
-LABEL io.airbyte.version=0.3.18
+LABEL io.airbyte.version=0.3.19
 LABEL io.airbyte.name=airbyte/destination-s3

airbyte-integrations/connectors/destination-snowflake/Dockerfile

@@ -20,5 +20,5 @@ RUN tar xf ${APPLICATION}.tar --strip-components=1
 
 ENV ENABLE_SENTRY true
 
-LABEL io.airbyte.version=0.4.43
+LABEL io.airbyte.version=0.4.44
 LABEL io.airbyte.name=airbyte/destination-snowflake

airbyte-integrations/connectors/destination-snowflake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StagingSqlOperations.java

@@ -82,7 +82,7 @@ public String uploadRecordsToStage(final JdbcDatabase database,
 
   @Override
   public void createStageIfNotExists(final JdbcDatabase database, final String stageName) {
-    s3StorageOperations.createBucketObjectIfNotExists(stageName);
+    s3StorageOperations.createBucketIfNotExists();
   }
 
   @Override

docs/integrations/destinations/bigquery.md

@@ -136,6 +136,7 @@ Now that you have set up the BigQuery destination connector, check out the follo
 
 | Version | Date       | Pull Request                                              | Subject                                                                                                                  |
 |:--------|:-----------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------|
+| 1.2.12  | 2023-01-18 | [#21087](https://github.com/airbytehq/airbyte/pull/21087) | Wrap Authentication Errors as Config Exceptions                                                                          |
 | 1.2.11  | 2023-01-18 | [#21144](https://github.com/airbytehq/airbyte/pull/21144) | Added explicit error message if sync fails due to a config issue                                                         |
 | 1.2.9   | 2022-12-14 | [#20501](https://github.com/airbytehq/airbyte/pull/20501) | Report GCS staging failures that occur during connection check                                                           |
 | 1.2.8   | 2022-11-22 | [#19489](https://github.com/airbytehq/airbyte/pull/19489) | Added non-billable projects handle to check connection stage                                                             |