Skip to content

column_type override not being honoured for dbt seed #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sdairs opened this issue Apr 6, 2022 · 4 comments
Closed

column_type override not being honoured for dbt seed #17

sdairs opened this issue Apr 6, 2022 · 4 comments
Assignees
Labels
bug Something isn't working

Comments

@sdairs
Copy link
Collaborator

sdairs commented Apr 6, 2022

create a seed with 2 cols

country_code,population
ABW,106766
AFE,677243299
AFG,38928341

Running dbt seed fails due to type INT being inferred, but some values are too large for INT and require BIGINT.

12:56:16  Completed with 1 error and 0 warnings:
12:56:16  
12:56:16  Runtime Error in seed ref__populations (seeds/reference/ref__populations.csv)
12:56:16    AnalysisException: Possible loss of precision for target table 'dbt_demo_reference.ref__populations'.
12:56:16    Expression '3332105361' (type: BIGINT) would need to be cast to INT for column 'population'

Create a schema.yml for the seed to overrise the inferred type to BIGINT

version: 2

seeds:
  - name: ref__populations
    description: Worldwide population data by country
    config:
      column_types:
        population: BIGINT

Use dbt seed --full-refresh to ensure the new schema is used.

12:40:25  Runtime Error in seed ref__populations (seeds/reference/ref__populations.csv)
12:40:25    AnalysisException: Target table 'dbt_demo_reference.ref__populations' is incompatible with source expressions.
12:40:25    Expression ''106766'' (type: STRING) is not compatible with column 'population' (type: BIGINT)
12:40:25 

Fails as the over ride is not honoured.

Sounds similar to dbt-labs/dbt-spark#139 seen on dbt-spark

@sdairs sdairs added the bug Something isn't working label Apr 6, 2022
@tovganesh
Copy link
Contributor

tovganesh commented Apr 6, 2022

Did some debugging, apparently the table is correctly created, but some issue with the insert:

`create table s3test.ref__populations (country_code string,population BIGINT)

      insert into s3test.ref__populations (country_code, population) values
      (%s,%s),(%s,%s),(%s,%s)
      
      ['ABW', '106766', 'AFE', '677243299', 'AFG', '38928341']

`

tovganesh added a commit that referenced this issue Apr 6, 2022
See: Issue#17 #17
Internal Ticket: https://jira.cloudera.com/browse/DBT-78

Testplan: Please follow the steps in Issue#17
@tovganesh
Copy link
Contributor

tovganesh commented Apr 6, 2022

#18

@sdairs can you verify if the above PR works at your end? This is mostly straight out of spark adapters' solution for similar issue

@sdairs
Copy link
Collaborator Author

sdairs commented Apr 7, 2022

Thanks @tovganesh - works perfectly

tovganesh added a commit that referenced this issue Apr 8, 2022
See: Issue#17 #17
Internal Ticket: https://jira.cloudera.com/browse/DBT-78

Testplan: Please follow the steps in Issue#17
@sdairs
Copy link
Collaborator Author

sdairs commented Apr 8, 2022

Fix has been merged

@sdairs sdairs closed this as completed Apr 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants