Skip to content

Fix catalog conversion issue in new update endpoint #14763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 15, 2022

Conversation

lmossman
Copy link
Contributor

What

I discovered a bug in the new update endpoint that was causing one of my CDC acceptance tests to fail.

The issue is when the new update endpoint tries to convert the existing ConfiguredAirbyteCatalog that was stored in the db to an api AirbyteCatalog to then compare with the catalog provided in the update input.

This code first converts the configured catalog to a "protocol" AirbyteCatalog, and then converts that to the "api" AirbyteCatalog on the next line.

The problem is that the CatalogHelpers.configuredCatalogToCatalog() method loses the stream configurations in the conversion, as it just grabs the "stream" half of the ConfiguredAirbyteStream object for each stream:

Then, the CatalogConverter.toApi() method just fills in a default configuration for each stream, and that is then used in the getConfigurationDiff comparison. This is bad because it is always comparing the update endpoint input configuration with some default configuration of each stream, instead of comparing the actual configuration in the database.

How

This PR fixes the issue by just using the proper toApi() endpoint in the CatalogConverter class to convert the configured catalog directly to an "api" AirbyteCatalog which preserves the stream configurations, just skipping the bad intermediate convert.

I also updated the "no streams to reset" unit test to validate that it is actually calling getConfigurationDiff on the correct catalogs (and I verified that this caused the test to fail before I added my fix).

@github-actions github-actions bot added area/platform issues related to the platform area/server labels Jul 15, 2022
@lmossman lmossman requested review from alovew and benmoriceau July 15, 2022 19:48
@lmossman lmossman merged commit d035b03 into master Jul 15, 2022
@lmossman lmossman deleted the lmossman/fix-catalog-conversion branch July 15, 2022 21:02
@lmossman lmossman mentioned this pull request Jul 17, 2022
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform area/server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants