Add airbyte-ci command: migrate-to-manifest-only

[ChristoGrab]

What

This PR adds a new airbyte-ci command: migrate-to-manifest-only. This command converts a valid low-code connector to manifest-only. It does not accept any additional arguments.

Resolves #8849

How

The command is broken up into three distinct steps:

1. First, we validate the following on a given connector:

• The connector is low-code
• The connector is not already tagged as manifest-only
• Only the following files exist in the connector's source folder: manifest.yaml, run.py, __init__.py and source.py
• The connector is a subclass of YamlDeclarativeSource
• The connector does not override the streams method (ie, it is not a hybrid low-code/Python connector)

If any of these checks fail, the migration is safely skipped

2. If the connector is a valid candidate, the second step strips it down to only essential files:

• Moves manifest.yaml to the root level of the directory
• Moves any non-inline specs to manifest.yaml
• Deletes all non-essential files and folders (everything but manifest.yaml, metadata.yaml, icon.svg, and any valid acceptance/integration test files)

3. The third step updates the remaining files to complete the migration:

• Points the connector's acceptance test config to the root-level manifest
• Adds the language:manifest-only tag to the connector's metadata and points to the latest source-declarative-manifest base image
• Generates an updated README.md file

If a failure occurs during the actual migration, the command will revert any changes to the connector's directory.

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 7, 2024 4:51pm

[natikgadzhi]

Good start!

[natikgadzhi]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (6d26096)

[natikgadzhi]

Please update the versions, and merge this, I don't want the code to sit for too long.

Make a follow-up PR (or a commit in this one) that parses the manifest and detects if $parameters are there and stops.

Other than that, we can run the wave.

[natikgadzhi]

@ChristoGrab new problems: source-activecampaign publishing fails in a funny way:

• gh run: https://github.com/airbytehq/airbyte/actions/runs/10223174041/job/28289079809
• report: https://ff9a4c2d768d00641202104c799ad0b0e60577efb4c7e966b4b3ff3-apidata.googleusercontent.com/download/storage/v1/b/airbyte-ci-reports-multi/o/airbyte-ci%2Fconnectors%2Fpublish%2Fmaster%2Fmaster%2F1722640435%2F3be2e97d8743ab3059be00625342ce672e43e898%2Fsource-activecampaign%2F0.2.0%2Foutput.html?jk=AYBlUPCRQofAIVI5Oxi3D45VdEgGuS7LCu5UrKWXRSDOPth8rHfDYaCv-C7RI3adKyaGWhd9V7Cy9emkFtr2IMp9AkqjHfcaRVVqZqqTARSiwEsIHT8chbmyO7YkqocZB-2LOxqV8cKVDCa10ouRqx-YOeCMsZLbuMAZeb4a7OZ70iUvJGl0hCnZN9W3HwPxSSR9jaR8nVinuE5h4bBy8PN9ZYLmVRwbUDYRozGpp3wTiiz6LDQ_VdN62eSnewiBoCSzrXyYzHEMr4n5r9FPX9BWlR6zpmdLiCykG5Jpp8xe3lNxofyt53OvDA4r90qLfh6tvnipKX3jPRMtTU0lDU9w3cS2t_NL6_151IIg9k67TnN_u_I-Q99-TQcTaJ8zNhVe0o8cB5ACYu7YnspDZ8dcjV77hTsNsAVwYw4IsnC5KY1fQ6zLSU-AOkz5_LmME8hfMDq79-BMCjBH3-Ua61nJYJbtqsxhmQJ8HhNUbCwckm2TRvwvSdl4WZYl5DPHgjxon85foU9chZvTHHY_sNzzW_WE1p1cp9mHt4LZ6iZ3gtt0YJLKMdt-Ca8jgBkhDfa-wp9ojg8zJ0ifIso1wYp4y-pPdzRIrIDwqHJ1sjCLLyD1AmYdotcUldGHura1D6J023v2FPRI7vsNqMCK-XHgzzmbuRYi8RTaONynqZfpEnevHSHHElkgbA38_NyRiuYYEOMkLijX_cKew2JdeQDl1rn1TVHJ-r8A8y0zXhHlgaXnDxJIQ-hVALl0ikR2lCuzIsB3A3WuMZRpP5UrBPtCSsArg6TcDZG75aInbOTeGkbnSI24PP6RPfRpJbAbpOX6LqTrUJ14RsTFt5-nJuxcOn2hlpVEzhr8cGz6fnuFFoF8KCScv0HpW3MNjmFT55xeeR5KdzFjPR3vhBj4AQZoe31Gdfv4eAKQeBJR5J7AmA7mNn2H-dGRKLwdEZYNWv_cPNSFmqm5lMD5irh9fXNZW0oYk5X6LaVvMZ6wQcooAi67ikBm7CtwOMyH1R78UfbfuMhhINjQAMWKZQBVTY72u99lGYZK8El7T920RPiiLlK-EI0Ux8QjURxV-tfXmN8r7IqLg7Xy7CRQauzbkpDlHgV0JBZL5LSPGACK3-keQTELZZdkHhASyKMqhdqbhKYgOoNPlvFfG6di4IIuEFzsamNZuqCJGa7-PgHEFReiUZd60b4qivwD0q7xd_v0GqwN9km04qoGBELGaGBxgsZyLso-R0LrwioXHHyLeArm34cXiXxENmUXs5RuKEPlp16Nbl_08nQuDPwkBl5_YNC5HniuGuu5CfNIC8Sm9jWjHf4asFRVtzScDkupxjPTYvixQszZ2zJZFSfbie4F76zOn9JhRHJ_Mf_Jlg&isca=1

Basically, it says that if it's a pypi connector, it must have python language tag. The answer to this is to remove disable the pypi entry in metadata if it exists when converting.

Long-term, this creates a little problem: there are packages for connectors on pypi that we are going to pull from pypi and start publishing as manifest-only. We should have a way to tell pypi users about it.

[ChristoGrab]

@natikgadzhi Hey, just coming back from the long weekend! Heard on the pypi tagging, I'll add logic to disable the tag and get this merged 👍

[natikgadzhi]

/format-fix

Format-fix job started... Check job output.

❌ Job failed.

[natikgadzhi]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (ccc30a7)

[natikgadzhi]

Using this to migrate source-hibob and source-northpass-lms that got merged yesterday ;)