InfoSchema V2

[tiancaiamao]

Enhancement

When we have millions of tables, the memory footprint for meta data is not negligible.
Some users use TiDB to provide services to their tenant, so there could be a huge amount of tables.
We need to develop a kind of caching mechanism to ensure that even in the extreme cases, the meta data do not cause TiDB OOM.

In the new design, the table.Table will not be persisted in memory, insted, they will be cached.

Development tasks

• add design document
• internal test plan document (link)
• introduce the core data struct for InfoSchemaV2
• introduce @@tidb_infoschema_cache_size switch to control the behaviour #51256
• apply diff for infoschema v2 #51680
• support snapshot infoschema for historial read and stale read #51679
• handle autoid when table is evicted #52150
• support meta data cache size limit #52237
• do not expose Tables in model.DBInfo #52234
• When many tables under a DB, dashboard keyvis feature will cost a lot of resource #54281
• observability: about the jitter when cache miss
  • display refill cache latency to slow query log
  • display refill cache latency to trace statement
  • display refill cache latency to explain analyze etc