🐛 octavia-cli: specific release job for octavia #11517

alafanechere · 2022-03-29T21:12:53Z

What

The current release workflow does not work for Octavia because:

Octavia CLI builds needs Python 3.8
The current release workflow uses Python 3.7
The self-hosted runner docker version does not have the docker buildx command to build multi-arch images, which is required for Octavia.

How

1. Common `bump_version.sh` script

I move the version bumping logic to a separate script which is re-used by the three jobs:

releaseAirbyte job, called from release_version.sh
releaseOctavia job, called from release_version_octavia.sh
createPullRequest job, called from a dedicated step
There is no harm in bumping three times in these three jobs because they all check out the same branch and only the createPullRequest job commits the changes. Other jobs require version bumping to tag the images after build.

2. Specific Octavia release job

Use ubuntu-latest runner, on which docker buildx is available
Install and use python 3.8 on this job
Run release_version_octavia.sh which is bumping, building and publishing octavia

3. Create a final job to open the bump PR

This job runs after releaseAirbyte and releaseOctavia:

Checkout Airbyte repo
Bump version
Create and open the bump version PR

It works 🎉

https://github.com/airbytehq/airbyte/pull/11587/files

The docker images were also successfully pushed to the DockerHub for Octavia, but I deleted the tags.
Did not run the platform build and publish yet, for safety mainly 😄

🚨User impact

The usual release process should not be impacted.

git-phu · 2022-03-29T22:57:33Z

merged in #11518 to get the release workflow working again on master branch

…/airbytehq/airbyte into augustin/octavia-cli/fix-release

supertopher

I made this comments before you changed some stuff so ¯_(ツ)_/¯

but looks great overall

supertopher · 2022-03-30T15:50:59Z

tools/bin/bump_version.sh

+pip install bumpversion
+bumpversion "$PART_TO_BUMP"
+
+NEW_VERSION=$(grep -w VERSION .env | cut -d"=" -f2)


nit: these grep lines can get scary as the env file changes in the future

Suggested change

NEW_VERSION=$(grep -w VERSION .env | cut -d"=" -f2)

# source and . are the same command

source .env

new_version=$VERSION

by POSIX convention that is largely unfollowed ALL_CAPS variables are for use outside of the file, and lower case variables are for internal use... no one follows this though so ¯\(ツ)/¯

my suggestion will annoy set -u as .env has alot of unset things

I can retrieve the new version from the bumpversion execution, but I wanted to use the same grep that we use for PREV_VERSION for consistency. Thanks for the suggestion! Do you think the grep regex should be more strict for safety?

greps are fine, I've just been bitten in the past.

They are brittle to future changes and it's not usually clear when a break happens caused by changes to something like ENV variable naming

supertopher · 2022-03-30T15:53:44Z

tools/bin/release_version.sh

-SUB_BUILD=PLATFORM ./gradlew publish
-VERSION=$NEW_VERSION GIT_REVISION=$GIT_REVISION docker-compose -f docker-compose.build.yaml push
-echo "Completed building and publishing..."
+source ./tools/bin/bump_version.sh


source seems a little confusing here, are we simply intending to execute bump version, or is there something I am missing?

I'd like to make the NEW_VERSION, PREV_VERSION and GIT_REVISION variable available in this script, these are defined in ./tools/bin/bump_version.sh this why I sourced the script

you will get those variables. However you will also execute all the code. I made a screenshot example. Just like most import statements you get the variables and you execute the code

Shell in it's heart wants executables to be TINY so you can put them all together and do cool stuff. If you want half a script it's probably too large, break it up and call it from both places if it works better

tools/bin/release_version_octavia.sh

Co-authored-by: Topher Lubaway <[email protected]>

supertopher

still approving

supertopher · 2022-03-30T17:00:18Z

tools/bin/release_version.sh

-SUB_BUILD=PLATFORM ./gradlew publish
-VERSION=$NEW_VERSION GIT_REVISION=$GIT_REVISION docker-compose -f docker-compose.build.yaml push
-echo "Completed building and publishing..."
+source ./tools/bin/bump_version.sh


you will get those variables. However you will also execute all the code. I made a screenshot example. Just like most import statements you get the variables and you execute the code

Shell in it's heart wants executables to be TINY so you can put them all together and do cool stuff. If you want half a script it's probably too large, break it up and call it from both places if it works better

supertopher · 2022-03-30T17:01:32Z

tools/bin/bump_version.sh

+pip install bumpversion
+bumpversion "$PART_TO_BUMP"
+
+NEW_VERSION=$(grep -w VERSION .env | cut -d"=" -f2)


greps are fine, I've just been bitten in the past.

They are brittle to future changes and it's not usually clear when a break happens caused by changes to something like ENV variable naming

alafanechere added 3 commits March 29, 2022 23:10

specific release step for octavia

bb938fe

specific release step for octavia

fb85af0

specific release step for octavia

f940207

git-phu temporarily deployed to more-secrets March 29, 2022 21:20 Inactive

fix typo

88120ec

git-phu temporarily deployed to more-secrets March 29, 2022 21:37 Inactive

clean venv

80fb22f

alafanechere self-assigned this Mar 30, 2022

alafanechere added 3 commits March 30, 2022 09:15

Merge branch 'master' into augustin/octavia-cli/fix-release

7d81669

disable push

b409a5b

Merge branch 'augustin/octavia-cli/fix-release' of https://github.com…

f56b478

…/airbytehq/airbyte into augustin/octavia-cli/fix-release

alafanechere temporarily deployed to more-secrets March 30, 2022 07:20 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 07:29 Inactive

check if plaftorm build pollutes octavia build

2c2fabc

alafanechere temporarily deployed to more-secrets March 30, 2022 07:49 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 07:50 Inactive

alafanechere added 4 commits March 30, 2022 10:26

separte job for octavia release

2ebc4e3

custom bumpversion for octavia

43f347a

clean workflox

bf04b8e

clean release_version_octavia.sh

7741b77

alafanechere temporarily deployed to more-secrets March 30, 2022 08:46 Inactive

get new version from bumpversion output

6beccda

alafanechere temporarily deployed to more-secrets March 30, 2022 08:57 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 08:58 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 08:59 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 14:40 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 14:41 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 14:42 Inactive

re-enable build + publish for octavia

eabec42

alafanechere temporarily deployed to more-secrets March 30, 2022 14:46 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 14:47 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 14:48 Inactive

alafanechere marked this pull request as draft March 30, 2022 15:02

alafanechere added 3 commits March 30, 2022 17:21

source bump_version.sh

6218f67

source bump_version.sh

a8c0be0

readd login

28629ff

alafanechere temporarily deployed to more-secrets March 30, 2022 15:25 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 15:27 Inactive

alafanechere temporarily deployed to more-secrets March 30, 2022 15:40 Inactive

re-enable build and publish of platform

4661a36

alafanechere marked this pull request as ready for review March 30, 2022 15:43

alafanechere requested review from jrhizor and cgardens March 30, 2022 16:05

supertopher approved these changes Mar 30, 2022

View reviewed changes

alafanechere and others added 2 commits March 30, 2022 18:52

Update tools/bin/release_version_octavia.sh

a8feb00

Co-authored-by: Topher Lubaway <[email protected]>

Update tools/bin/release_version_octavia.sh

afd97ab

Co-authored-by: Topher Lubaway <[email protected]>

alafanechere temporarily deployed to more-secrets March 30, 2022 16:55 Inactive

supertopher approved these changes Mar 30, 2022

View reviewed changes

alafanechere merged commit 2d8f66f into master Mar 30, 2022

alafanechere deleted the augustin/octavia-cli/fix-release branch March 30, 2022 18:47

octavia-squidington-iii mentioned this pull request Mar 31, 2022

Bump Airbyte version from 0.35.62-alpha to 0.35.63-alpha #11624

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 octavia-cli: specific release job for octavia #11517

🐛 octavia-cli: specific release job for octavia #11517

alafanechere commented Mar 29, 2022 •

edited

Loading

git-phu commented Mar 29, 2022

supertopher left a comment

supertopher Mar 30, 2022 •

edited

Loading

alafanechere Mar 30, 2022

supertopher Mar 30, 2022

supertopher Mar 30, 2022

alafanechere Mar 30, 2022

supertopher Mar 30, 2022 •

edited

Loading

supertopher left a comment

supertopher Mar 30, 2022 •

edited

Loading

supertopher Mar 30, 2022

-NEW_VERSION=$(grep -w VERSION .env | cut -d"=" -f2)
+# source and . are the same command
+source .env
+new_version=$VERSION

🐛 octavia-cli: specific release job for octavia #11517

🐛 octavia-cli: specific release job for octavia #11517

Conversation

alafanechere commented Mar 29, 2022 • edited Loading

What

How

1. Common bump_version.sh script

2. Specific Octavia release job

3. Create a final job to open the bump PR

It works 🎉

🚨User impact

git-phu commented Mar 29, 2022

supertopher left a comment

Choose a reason for hiding this comment

supertopher Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

alafanechere Mar 30, 2022

Choose a reason for hiding this comment

supertopher Mar 30, 2022

Choose a reason for hiding this comment

supertopher Mar 30, 2022

Choose a reason for hiding this comment

alafanechere Mar 30, 2022

Choose a reason for hiding this comment

supertopher Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

supertopher left a comment

Choose a reason for hiding this comment

supertopher Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

supertopher Mar 30, 2022

Choose a reason for hiding this comment

alafanechere commented Mar 29, 2022 •

edited

Loading

1. Common `bump_version.sh` script

supertopher Mar 30, 2022 •

edited

Loading

supertopher Mar 30, 2022 •

edited

Loading

supertopher Mar 30, 2022 •

edited

Loading