Create schema to persist discover schema output #9896
Labels
area/platform
issues related to the platform
team/compose
team/platform-move
type/enhancement
New feature or request
epic: #9895
Tell us about the problem you're trying to solve
We want to keep track of the output of discover schema call. See the epic for the full context.
We want the persistence of this information to support the following features:
Describe the solution you’d like
Add 2 tables:
actor_catalog - this table primarily stores a raw catalog
catalog_hash - hash of catalog for easier comparisons
actor_catalog_fetch_event
Update connection table to add a column
The goal of using 2 tables here is to avoid storing multiple copies of the same catalog outputs which can be massive. We know they tend not to change.
Describe the alternative you’ve considered or used
We could use a single table but we would likely end up with tons of duplicate catalogs. The objects are huge so they are hard to tell apart, so we would need some separate mechanism to do that anyway.
The text was updated successfully, but these errors were encountered: