Fix catalog conversion issue in new update endpoint #14763
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
I discovered a bug in the new update endpoint that was causing one of my CDC acceptance tests to fail.
The issue is when the new update endpoint tries to convert the existing
ConfiguredAirbyteCatalog
that was stored in the db to an apiAirbyteCatalog
to then compare with the catalog provided in the update input.This code first converts the configured catalog to a "protocol" AirbyteCatalog, and then converts that to the "api" AirbyteCatalog on the next line.
The problem is that the
CatalogHelpers.configuredCatalogToCatalog()
method loses the stream configurations in the conversion, as it just grabs the "stream" half of the ConfiguredAirbyteStream object for each stream:airbyte/airbyte-protocol/protocol-models/src/main/java/io/airbyte/protocol/models/CatalogHelpers.java
Line 82 in 5edf3fe
Then, the
CatalogConverter.toApi()
method just fills in a default configuration for each stream, and that is then used in thegetConfigurationDiff
comparison. This is bad because it is always comparing the update endpoint input configuration with some default configuration of each stream, instead of comparing the actual configuration in the database.How
This PR fixes the issue by just using the proper
toApi()
endpoint in the CatalogConverter class to convert the configured catalog directly to an "api" AirbyteCatalog which preserves the stream configurations, just skipping the bad intermediate convert.I also updated the "no streams to reset" unit test to validate that it is actually calling
getConfigurationDiff
on the correct catalogs (and I verified that this caused the test to fail before I added my fix).