Change the settings page in the UI to display the union of the schema and the configured schema #9902
Labels
area/platform
issues related to the platform
team/compose
team/platform-move
type/enhancement
New feature or request
user-impact
Epic: #9895
Tell us about the problem you're trying to solve
Currently the UI gets the catalog for a source from
sources/discover_schema
. It is then used to create a connectionconnections/create
.The configured catalog can be lossy. For example, if the catalog from discover_schema has stream1 and stream2, but then the connection only configures stream1, then the user will never be able to see that stream again in the UI (unless they click force refresh schema). This is very confusing. Our goal is not forget this information. Thus, when loading the settings page, we should use the information page and call
sources/discover_schema
. With the new caching behavior in this endpoint (#9901), this will be cheap. We can then use that catalog in the settings page and then mask the actual configuration on top of it. If needed, we can do this on the backend first. We should discuss the right interface here.The text was updated successfully, but these errors were encountered: