[CT-3438] `UserConfig` in `profiles.yml` -> `flags` in `dbt_project.yml`

[jtcohen6]

Housekeeping

• I am a maintainer of dbt-core

Short description

We are deprecating config as a thing that is settable in profiles.yml. Instead, we will support flags in dbt_project.yml

For dbt-core v1.8, we should see if the user has defined config in profiles.yml.

• If they have, continue to respect it, and raise a deprecation warning.
• If they haven't, great. Instead, we will be reading flags from dbt_project.yml. The two are mutually exclusive.

In a future release, we will remove UserConfig from profiles.yml entirely.

Acceptance criteria

In dbt_project.yml, it will look like:

# dbt_project.yml
flags:
  send_anonymous_usage_stats: True

Going forward, some flags will be settable in:

1. "Project flags" (dbt_project.yml) only. These flags define behaviors tied to the source code in a project, and should remain consistent across all users, environments, and invocations of that project.
2. CLI option / env var only. These flags are highly specific to the running invocation, and don't make sense to define in version-controlled code. Several of these are file system paths, such as --project-dir or --log-path. Some want to vary across commands, such as --full-refresh. Environment variables are best suited to flags that want different settings in different environments, such as turning DBT_FAIL_FAST on in dev/CI but not in prod.
3. Both places. In this case, a project wants to set a default behavior, but it may be appropriate to override it for different environments/invocations.

If set in both places, the inheritance hierarchy remains the same:

1. CLI var (handled by click)
2. Env var (handled by click)
3. User config Project flags

Implementation Notes

• We should rename the internal UserConfig to ProjectFlags
• User config is read when Flags is instantiated here. This is probably still the right place to do it, but the logic now becomes:
  • read_project_flags:
    • tries to load a user config from profiles.yml. If found, read it and raise a deprecation warning
    • if not found, tries to load user config from dbt_project.yml (can be a minimal yaml parse for just the 'flags' key - no jinja parsing necessary)
    • otherwise, use default user config settings (Nones are appropriate to ensure flag precedence works as expected for flags that can be set via CLI or env var)
• this will also need to be updated in core.dbt.flags.set_from_args

Testing:

• let's add a unit test that ensures that: every attribute of ProjectFlags that has a corresponding click option in params.py should be set to None by default (except for anon user tracking). Going forward, flags can have non-None defaults if they do not have a corresponding CLI option/env var. These will be used to control backwards incompatible interface or behaviour changes.

Impact to Other Teams

N/A

Will backports be required?

no :)

Context

Implications of deprecation

Why was this called UserConfig before? Well... hypothetically, if Michelle wants colorized logs, and Jeremy doesn't, they should be able to set user-specific preferences that apply across all their projects, without checking those preferences into those projects' version control (because profiles.yml lives in the HOME directory on each machine). While we are removing UserConfig from profiles.yml, it will remain possible to set user-specific preferences by setting environment variables (in .bashrc, .zshrc, etc).

[jtcohen6]

From chatting with @gshank: Let's see what the complexity will be for implementation a "soft" deprecation. If it's a lot of additional work, we should instead opt for a "hard" deprecation in v1.8 (with a lot of prior notice), i.e. stop reading config from profiles.yml entirely.