feat(api): append cooments of column and table for metadata api #812
Conversation
There was a problem hiding this comment.
Thanks @grieve54706 working on this.
It seems that this PR doesn't implement for BigQuery to get the comment of tables or columns. Is there any plan for it?
| DataSource.mssql: MSSQLMetadata, | ||
| DataSource.mysql: MySQLMetadata, | ||
| DataSource.postgres: PostgresMetadata, | ||
| DataSource.snowflake: SnowflakeMetadata, |
There was a problem hiding this comment.
It seems Snowflake metadata is no yet supported. Maybe we can remove it from the mapping and throw NoImplementedError if someone uses it.
| SELECT | ||
| t.table_catalog, | ||
| t.table_schema, | ||
| t.table_name, | ||
| c.column_name, | ||
| c.data_type, | ||
| c.is_nullable, | ||
| c.column_comment | ||
| FROM | ||
| information_schema.tables AS t | ||
| INNER JOIN | ||
| information_schema.columns AS c | ||
| ON t.table_catalog = c.table_catalog | ||
| AND t.table_schema = c.table_schema | ||
| AND t.table_name = c.table_name | ||
| WHERE t.table_schema = '{schema}' |
There was a problem hiding this comment.
I didn't try it but maybe we can merge it with the table comments query? Something like
SELECT
t.table_catalog,
t.table_schema,
t.table_name,
c.column_name,
c.data_type,
c.is_nullable,
c.column_comment,
tc.comment
FROM
information_schema.tables AS t
INNER JOIN
information_schema.columns AS c
ON t.table_catalog = c.table_catalog
AND t.table_schema = c.table_schema
AND t.table_name = c.table_name
INNER JOIN
system.metadata.table_comments AS tc
ON t.table_schema = tc.schema_name
AND t.table_name = tc.table_name
WHERE t.table_schema = '{schema}'There was a problem hiding this comment.
No. Trino does not implement getTableHandle for SystemTablesMetadata. It will fail when executing.
After implementing it again, it worked. Thank you for the nice idea.
| text(""" | ||
| EXEC sys.sp_addextendedproperty | ||
| @name = N'MS_Description', | ||
| @value = N'This is a table comment', | ||
| @level0type = N'SCHEMA', @level0name = 'dbo', | ||
| @level1type = N'TABLE', @level1name = 'orders'; | ||
| """) | ||
| ) | ||
| conn.execute( | ||
| text(""" | ||
| EXEC sys.sp_addextendedproperty | ||
| @name = N'MS_Description', | ||
| @value = N'This is a comment', | ||
| @level0type = N'SCHEMA', @level0name = 'dbo', | ||
| @level1type = N'TABLE', @level1name = 'orders', | ||
| @level2type = N'COLUMN', @level2name = 'o_comment'; | ||
| """) |
Already implemented this feature for BigQuery. |
| @@ -97,7 +115,7 @@ def test_query(trino: TrinoContainer): | |||
| "1996-01-02", | |||
| "1_370", | |||
| "2024-01-01 23:59:59.000000", | |||
| "2024-01-01 23:59:59.000000 UTC", | |||
| "2024-01-01 23:59:59.000000", | |||
There was a problem hiding this comment.
We didn't change the MDL and input data. Why does this value change? 🤔
There was a problem hiding this comment.
We changed the expression before, but we did not run CI for trino and did not change the excepted (#782)
|
There are some conflicts here. @grieve54706 |
we changed expression but did not update excepted on #782
grieve54706
force-pushed
the
feature/metadata-api
branch
from
9364cdd to
53247f0
Compare
* feat(clickhouse): append comments of column and table * feat(clickhouse): append comments of column and table * feat(mssql): append comments of column and table * feat(mysql): append comments of column and table * feat(postgres): append comments of column and table * fix(trino): wait trino container ready to avoid 'nodes is empty' from Trino * feat(trino): append comments of column and table * chore(test): sort func and rename func * test(snowflake): mark not implemented feature * refactor(metadata): tidy up * chore(metadata): remove unimplemented data source * fix(trino): fix connection * chore(trino): merge two statement * test(trino): update excepted we changed expression but did not update excepted on #782
This pull request includes several changes aimed at improving the metadata handling for various database connections in the
ibis-serverproject. The most important changes include removing unnecessaryjsonimports, updating methods to useto_dictinstead ofto_json, adding connection initialization in constructors, and enhancing the SQL queries to fetch additional metadata.Codebase Simplification:
jsonimports across multiple filesMethod Updates:
to_dictinstead ofto_jsonfor converting SQL query results to dictionaries.Connection Initialization:
ClickHouseMetadata,MSSQLMetadata,MySQLMetadata, andPostgresMetadataclasses.SQL Query Enhancements:
Factory Pattern:
MetadataFactoryto use a mapping dictionary for cleaner and more maintainable code.These changes collectively enhance the maintainability, readability, and functionality of the metadata handling code in the
ibis-serverproject.