Include shadowJar artifact to release workflow (#8) #106

GumpacG · 2023-07-17T20:06:55Z

Description

Modified release-drafter to include shadowJar to the release artifacts.
Modified JenkinsFile to publish shadowJar along with JDBC driver files to the artifacts bucket.

Issues Resolved

#98

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

* Test packaging tar with shadowJar Signed-off-by: Guian Gumpac <[email protected]> * Setting version as a variable Signed-off-by: Guian Gumpac <[email protected]> * Changed release-drafter to include shadowJar Signed-off-by: Guian Gumpac <[email protected]> * Bumped version to 1.4.0.1 Signed-off-by: Guian Gumpac <[email protected]> * Bumped version to 1.4.0.1 Signed-off-by: Guian Gumpac <[email protected]> * Simplified adding shadowJar to artifacts Signed-off-by: Guian Gumpac <[email protected]> * Reverted whitespace change Signed-off-by: Guian Gumpac <[email protected]> * Reverted whitespace change Signed-off-by: Guian Gumpac <[email protected]> * Create shadowJar as a separate artifact Signed-off-by: Guian Gumpac <[email protected]> * Fixed wrong version Signed-off-by: Guian Gumpac <[email protected]> * Added jenkins job to publish shadowJar to artifacts bucket Signed-off-by: Guian Gumpac <[email protected]> * Changed source and signing platform Signed-off-by: Guian Gumpac <[email protected]> --------- Signed-off-by: Guian Gumpac <[email protected]>

GumpacG · 2023-07-17T22:28:23Z

@prudhvigodithi and @gaiksaya could you take a look at the jenkinsfile change to ensure that I have provided the correct arguments as this is difficult to test? Thanks in advance.

gaiksaya · 2023-07-24T18:22:12Z

.github/workflows/release-drafter.yml

      - name: Draft a release
        uses: softprops/action-gh-release@v1
        with:
          draft: true
          generate_release_notes: true
          files: |
            artifacts.tar.gz
+            build/libs/opensearch-sql-jdbc-shadow-*.jar


Tha shadow jar need to be bundled within artifacts.tar.gz. The jenkins server will only download that 1 artifact and unzip it.
See how can you bundle all in one tar.

Thanks!

Thanks for responding @gaiksaya, my worry is that including the shadowJar in the artifacts.tar is that Jenkins will also upload it to Maven which not ideal as per Bit-Quill#8 (comment).

What we need is:

have maven include non-shadowJar driver files and exclude shadowJar

OpenSearch downloads page to include shadowJar and non-shadowJar driver files.

Would you know if it is possible to exclude a file from artifacts.tar from getting published to Maven?

Yes that is possible. What is published to maven is decided by these lines https://github.com/opensearch-project/sql-jdbc/pull/106/files#diff-619ac2b62b38c4694d7b69e89c001d78c5f853c5b39979c34d01012f778d26f0R12-R15

Right now if you unzip artifacts.tar.gz, these are the contents:

tree repository repository └── org └── opensearch └── driver └── opensearch-sql-jdbc ├── 1.4.0.0 │ ├── opensearch-sql-jdbc-1.4.0.0-javadoc.jar │ ├── opensearch-sql-jdbc-1.4.0.0-javadoc.jar.md5 │ ├── opensearch-sql-jdbc-1.4.0.0-javadoc.jar.sha1 │ ├── opensearch-sql-jdbc-1.4.0.0-sources.jar │ ├── opensearch-sql-jdbc-1.4.0.0-sources.jar.md5 │ ├── opensearch-sql-jdbc-1.4.0.0-sources.jar.sha1 │ ├── opensearch-sql-jdbc-1.4.0.0.jar │ ├── opensearch-sql-jdbc-1.4.0.0.jar.md5 │ ├── opensearch-sql-jdbc-1.4.0.0.jar.sha1 │ ├── opensearch-sql-jdbc-1.4.0.0.pom │ ├── opensearch-sql-jdbc-1.4.0.0.pom.md5 │ └── opensearch-sql-jdbc-1.4.0.0.pom.sha1 ├── maven-metadata.xml ├── maven-metadata.xml.md5 └── maven-metadata.xml.sha1

We can also zip the shadow jar outside repository folder.

Question: Do we want all the below files or just the jar?

ll build/libs total 12872 -rw-r--r-- 1 gaiksaya staff 261B Jul 26 11:08 opensearch-sql-jdbc-1.4.0.0-javadoc.jar -rw-r--r-- 1 gaiksaya staff 109K Jul 26 11:08 opensearch-sql-jdbc-1.4.0.0-sources.jar -rw-r--r-- 1 gaiksaya staff 192K Jul 26 11:08 opensearch-sql-jdbc-1.4.0.0.jar -rw-r--r-- 1 gaiksaya staff 5.9M Jul 26 11:08 opensearch-sql-jdbc-shadow-1.4.0.0.jar

Sorry, my assumptions were actually wrong. What we actually want is to have:

only non-shadowJar files in Maven

only shadowJar file in the artifacts bucket

I've updated the PR and with updated paths for the artifacts bucket. The new tree for artifacts is

artifacts ├── repository │ └── org │ └── opensearch │ └── driver │ └── opensearch-sql-jdbc │ ├── 1.4.0.0 │ │ ├── opensearch-sql-jdbc-1.4.0.0-javadoc.jar │ │ ├── opensearch-sql-jdbc-1.4.0.0-javadoc.jar.md5 │ │ ├── opensearch-sql-jdbc-1.4.0.0-javadoc.jar.sha1 │ │ ├── opensearch-sql-jdbc-1.4.0.0-sources.jar │ │ ├── opensearch-sql-jdbc-1.4.0.0-sources.jar.md5 │ │ ├── opensearch-sql-jdbc-1.4.0.0-sources.jar.sha1 │ │ ├── opensearch-sql-jdbc-1.4.0.0.jar │ │ ├── opensearch-sql-jdbc-1.4.0.0.jar.md5 │ │ ├── opensearch-sql-jdbc-1.4.0.0.jar.sha1 │ │ ├── opensearch-sql-jdbc-1.4.0.0.pom │ │ ├── opensearch-sql-jdbc-1.4.0.0.pom.md5 │ │ └── opensearch-sql-jdbc-1.4.0.0.pom.sha1 │ ├── maven-metadata.xml │ ├── maven-metadata.xml.md5 │ └── maven-metadata.xml.sha1 └── shadowJar └── opensearch-sql-jdbc-shadow-1.4.0.0.jar

Could you take a look at the jenkinsFile again to ensure that it is correct? In particular the role for publishing to the artifacts bucket.

Signed-off-by: Guian Gumpac <[email protected]>

.github/workflows/release-drafter.yml

Signed-off-by: Guian Gumpac <[email protected]>

jenkins/release.jenkinsFile

Signed-off-by: Guian Gumpac <[email protected]>

gaiksaya · 2023-07-26T21:28:36Z

Hi @GumpacG,

One important thing is signing the artifacts that are uploaded to artifacts.opensearch.org. We have 2 options:
Create a detached signature such as .sig or .asc file based on shadow jar (will create opensearch-sql-jdbc-shadow-1.4.0.0.jar.sig or opensearch-sql-jdbc-shadow-1.4.0.0.jar.asc)or use jar signer that signs the actual jar. (Adding @rishabh6788 here for visibility)
How would the maintainers like to proceed?

We are yet to integrate our jarsigner with the signing system and would like to take this opportunity to do the same.
Also, when is the next release for sql-jdbc planned?

GumpacG · 2023-07-26T21:36:28Z

I believe we want jarsigner. @Yury-Fridlyand could you confirm and also wondering if you have a time line for next release?

Yury-Fridlyand · 2023-07-27T16:35:24Z

Yes, we need a jarsigner.
Thank you @gaiksaya

GumpacG · 2023-07-27T16:55:18Z

Thanks @Yury-Fridlyand and @gaiksaya! Any more actions you need from me for this PR?

gaiksaya · 2023-07-28T21:44:19Z

Thanks @Yury-Fridlyand and @gaiksaya! Any more actions you need from me for this PR?

I would hold on to this PR until jarsigner is integrated by @rishabh6788 . Will come back and comment on this with necessary changes.
cc: @bbarani

GumpacG · 2023-07-28T21:55:22Z

Sounds good. Thanks!

GumpacG · 2023-07-31T16:59:16Z

Hi @rishabh6788, would you have a timeline for this?

prudhvigodithi · 2023-07-31T17:06:02Z

Hey quick thought, we are using jarsigner for the .taco files (related issues: opensearch-project/opensearch-build#2469), for .jar files should we use PGP, just like how we sigh the other jars. @rishabh6788 @gaiksaya @peterzhuamazon.

gaiksaya · 2023-07-31T17:25:45Z

Hey quick thought, we are using jarsigner for the .taco files (related issues: opensearch-project/opensearch-build#2469), for .jar files should we use PGP, just like how we sigh the other jars. @rishabh6788 @gaiksaya @peterzhuamazon.

That's because maven wants .asc files for publishing. In this case we are not publishing the shadow jar to maven but just to artifacts.opensearch.org. So instead of having 2 files, one the actual jar and other one signature we can try signing it with jar signer.

jenkins/release.jenkinsFile

gaiksaya · 2023-08-19T01:26:09Z

Thanks for the patience. The jar_signer is now integrated with jenkins.
Please update the lib to 5.6.1. (First line of jenkinsFile, not allowing me to add comment to unchanged line.)

lib = library(identifier: '[email protected]', retriever: modernSCM([
    $class: 'GitSCMSource',
    remote: 'https://github.com/opensearch-project/opensearch-build-libraries.git',
]))

Signed-off-by: acarbonetto <[email protected]>

Yury-Fridlyand · 2023-08-21T22:16:55Z

@gaiksaya can you re-review please?

gaiksaya

LGTM! Thank you for the patience.

GumpacG requested review from pjfitzgibbons, ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vamsimanohar, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey and acarbonetto as code owners July 17, 2023 20:06

Yury-Fridlyand requested a review from gaiksaya July 17, 2023 20:08

Yury-Fridlyand requested a review from prudhvigodithi July 17, 2023 22:48

gaiksaya reviewed Jul 24, 2023

View reviewed changes

Separated shadowJar from repository directory

1d270aa

Signed-off-by: Guian Gumpac <[email protected]>

gaiksaya reviewed Jul 26, 2023

View reviewed changes

.github/workflows/release-drafter.yml Outdated Show resolved Hide resolved

Fixed tar issue with *

43e1af0

Signed-off-by: Guian Gumpac <[email protected]>

gaiksaya reviewed Jul 26, 2023

View reviewed changes

jenkins/release.jenkinsFile Show resolved Hide resolved

jenkins/release.jenkinsFile Outdated Show resolved Hide resolved

jenkins/release.jenkinsFile Outdated Show resolved Hide resolved

Addressed PR comments

19bdcc1

Signed-off-by: Guian Gumpac <[email protected]>

acarbonetto mentioned this pull request Aug 1, 2023

[FEATURE] Update 1-click release process to publish shadow jar as well #98

Closed

Yury-Fridlyand linked an issue Aug 1, 2023 that may be closed by this pull request

[FEATURE] Update 1-click release process to publish shadow jar as well #98

Closed

gaiksaya reviewed Aug 19, 2023

View reviewed changes

jenkins/release.jenkinsFile Outdated Show resolved Hide resolved

Update jenkins version and release

452f603

Signed-off-by: acarbonetto <[email protected]>

acarbonetto requested a review from gaiksaya August 22, 2023 15:27

gaiksaya approved these changes Aug 22, 2023

View reviewed changes

acarbonetto approved these changes Aug 22, 2023

View reviewed changes

GumpacG merged commit 5a67284 into opensearch-project:main Aug 22, 2023

Yury-Fridlyand deleted the integ-shadow-release branch August 22, 2023 18:31

Include shadowJar artifact to release workflow (#8) #106

Include shadowJar artifact to release workflow (#8) #106

Uh oh!

Conversation

GumpacG commented Jul 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

GumpacG commented Jul 17, 2023

Uh oh!

gaiksaya Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GumpacG Jul 24, 2023

Choose a reason for hiding this comment

Uh oh!

gaiksaya Jul 26, 2023

Choose a reason for hiding this comment

Uh oh!

GumpacG Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaiksaya commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GumpacG commented Jul 26, 2023

Uh oh!

Yury-Fridlyand commented Jul 27, 2023

Uh oh!

GumpacG commented Jul 27, 2023

Uh oh!

gaiksaya commented Jul 28, 2023

Uh oh!

GumpacG commented Jul 28, 2023

Uh oh!

GumpacG commented Jul 31, 2023

Uh oh!

prudhvigodithi commented Jul 31, 2023

Uh oh!

gaiksaya commented Jul 31, 2023

Uh oh!

Uh oh!

gaiksaya commented Aug 19, 2023

Uh oh!

Yury-Fridlyand commented Aug 21, 2023

Uh oh!

gaiksaya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GumpacG commented Jul 17, 2023 •

edited

Loading

gaiksaya Jul 24, 2023 •

edited

Loading

GumpacG Jul 26, 2023 •

edited

Loading

gaiksaya commented Jul 26, 2023 •

edited

Loading