Skip to content

File CDK: migrate S3 configs  #29435

Closed
Closed
@clnoll

Description

@clnoll

Proposed solution

Steps:

  1. Modify S3 config to support both:
    • make dataset, path_pattern and provider not required and airbyte_hidden=True to all V3 fields
    • update order so that it does not clash with V4 This does not seem relevant
    • have SourceFilesAbstract.spec add all source_s3.v4.config.Config.schema required to connectionSpecification and consider those fields when reading the config
  2. Have the read_config method emit an AirbyteControlMessage with the new config
  3. Deploy

Expected result:

  • There are no V3 configurations anymore

User impact:

  • Once deployed, if a user wants to edit an already existing config before a new sync has been executed, he will have to re-enter all the configuration hence we should make sure the time between those steps is as little a possible
  • If the user updates from <3.1.X to V4, the migration result is undetermined. Based on @flash1293 in most cases, it should be fine as the spec hasn't changed much between v1.X and now but we have only tested with clearly V3 configs

Acceptance criteria

  • End state:
    • migration is still in place and will migrate configs. Eventually, we could remove it
    • Spec visible to the user is V4 spec. V3 specific fields are hidden
  • Update the breaking change to mention that some fields might be empty at the beginning (if a sync has not been executed yet)

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions