Description
Epic: #9895
Tell us about the problem you're trying to solve
Currently the UI gets the catalog for a source from sources/discover_schema
. It is then used to create a connection connections/create
.
The configured catalog can be lossy. For example, if the catalog from discover_schema has stream1 and stream2, but then the connection only configures stream1, then the user will never be able to see that stream again in the UI (unless they click force refresh schema). This is very confusing. Our goal is not forget this information. Thus, when loading the settings page, we should use the information page and call sources/discover_schema
. With the new caching behavior in this endpoint (#9901), this will be cheap. We can then use that catalog in the settings page and then mask the actual configuration on top of it. If needed, we can do this on the backend first. We should discuss the right interface here.