[Tests][BWC] add BWC tests to Jenkins

[kavilla]

Description

For OpenSearch and OpenSearch Dashboards. Also fix a bug with the test config
showing incorrectly.

Signed-off-by: Kawika Avilla [email protected]

Issues Resolved

#705

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #1795 (54cb671) into main (ee396c4) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1795      +/-   ##
============================================
+ Coverage     94.49%   94.52%   +0.03%     
- Complexity       18       19       +1     
============================================
  Files           174      176       +2     
  Lines          3612     3614       +2     
  Branches         27       27              
============================================
+ Hits           3413     3416       +3     
+ Misses          195      194       -1     
  Partials          4        4              

Impacted Files	Coverage Δ
src/test_workflow/bwc_test/bwc_test_suite.py	92.45% <ø> (ø)
tests/jenkins/jobs/RunBwcTestScript_Jenkinsfile	100.00% <100.00%> (ø)
...RunBwcTestScript_OpenSearch_Dashboards_Jenkinsfile	100.00% <100.00%> (ø)
src/jenkins/BuildManifest.groovy	95.12% <0.00%> (+2.43%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee396c4...54cb671. Read the comment docs.

[abhinavGupta16]

Can you please add tests for the new jobs? Thank you!

Refer this

[kavilla]

Can you please add tests for the new jobs? Thank you!

Refer this

@abhinavGupta16, could you provide a reference or help with how to handle the dockerAgent variable that is used for the agent. I keep getting something on the lines:

    <failure message="groovy.lang.MissingPropertyException: No such property: dockerAgent for class: bwc-test" type="groovy.lang.MissingPropertyException">groovy.lang.MissingPropertyException: No such property: dockerAgent for class: bwc-test

I created a binded variable and that doesn't work and tried to return a function. It would appear there is an open issue here: jenkinsci/JenkinsPipelineUnit#507 that actually seems the reason why I'm not able to do this.

[abhinavGupta16]

Can you please add tests for the new jobs? Thank you!
Refer this

@abhinavGupta16, could you provide a reference or help with how to handle the dockerAgent variable that is used for the agent. I keep getting something on the lines:

    <failure message="groovy.lang.MissingPropertyException: No such property: dockerAgent for class: bwc-test" type="groovy.lang.MissingPropertyException">groovy.lang.MissingPropertyException: No such property: dockerAgent for class: bwc-test

I created a binded variable and that doesn't work and tried to return a function. It would appear there is an open issue here: jenkinsci/JenkinsPipelineUnit#507 that actually seems the reason why I'm not able to do this.

I don't think we can override the variables declared locally in script. Adding a libtester for detectTestDockerImage() should help

[kavilla]

I don't think we can override the variables declared locally in script. Adding a libtester for detectTestDockerImage() should help

Yes I am referring the lib tester for detect test docker image like:

this.registerLibTester(new DetectTestDockerAgentLibTester('tests/jenkins/data/opensearch-dashboards-1.2.0-test.yml'))

but it still fails here

    stage('bwc-test') {
            agent {
                docker {
                    label AGENT_LABEL
                    image dockerAgent.image
                    args dockerAgent.args
                    alwaysPull true
                }
            }

because No such property: dockerAgent for class: bwc-test

[abhinavGupta16]

Yes I am referring the lib tester for detect test docker image like:

this.registerLibTester(new DetectTestDockerAgentLibTester('tests/jenkins/data/opensearch-dashboards-1.2.0-test.yml'))

but it still fails here

    stage('bwc-test') {
            agent {
                docker {
                    label AGENT_LABEL
                    image dockerAgent.image
                    args dockerAgent.args
                    alwaysPull true
                }
            }

because No such property: dockerAgent for class: bwc-test

Ah! I see. I believe we should make dockerAgent variable globally available or in a parent stage. Currently, it is out of scope for this stage, hence the failure.

[kavilla]

I believe we should make dockerAgent variable globally available or in a parent stage.

Within the test?

[abhinavGupta16]

I believe we should make dockerAgent variable globally available or in a parent stage.

Within the test?

In the job

[kavilla]

In the job

Will that create conflicts for every pipeline that uses a variable dockerAgent

[abhinavGupta16]

In the job

Will that create conflicts for every pipeline that uses a variable dockerAgent

It is still being used as a global variable in the job. dockerAgent is declared locally in a stage and within script block. It is then being used in docker block of a different stage. Unfortunately, jenkins allows this, but we shouldn't share local variables between stages

[kavilla]

Agent can't be none here due to fileExists

Error message from Jenkins : Required context class hudson.FilePath is missing Perhaps you forgot to surround the code with a step that provides this, such as: node

[kavilla]

@abhinavGupta16, was able to get the tests added. Also got it successfully running in Jenkins as well:

Results:

2022-03-23 07:32:52 INFO     Skipping bwc-tests for opensearch-observability, as it is currently not declared in the test manifest
2022-03-23 07:32:53 INFO     | alerting             | with-security        | PASS  |    0 |
2022-03-23 07:32:53 INFO     | alerting             | without-security     | PASS  |    0 |
2022-03-23 07:32:53 ERROR    | anomaly-detection    | with-security        | FAIL  |    1 |
2022-03-23 07:32:53 ERROR    | anomaly-detection    | without-security     | FAIL  |    1 |
2022-03-23 07:32:53 INFO     | asynchronous-search  | with-security        | PASS  |    0 |
2022-03-23 07:32:53 INFO     | asynchronous-search  | without-security     | PASS  |    0 |
2022-03-23 07:32:53 INFO     | dashboards-reports   | without-security     | PASS  |    0 |
2022-03-23 07:32:53 INFO     | index-management     | with-security        | PASS  |    0 |
2022-03-23 07:32:53 INFO     | index-management     | without-security     | PASS  |    0 |
2022-03-23 07:32:53 ERROR    | k-NN                 | with-security        | FAIL  |    1 |
2022-03-23 07:32:53 ERROR    | k-NN                 | without-security     | FAIL  |    1 |
2022-03-23 07:32:53 ERROR    | ml-commons           | with-security        | FAIL  |    1 |
2022-03-23 07:32:53 ERROR    | ml-commons           | without-security     | FAIL  |    1 |
2022-03-23 07:32:53 INFO     | sql                  | with-security        | PASS  |    0 |
2022-03-23 07:32:53 INFO     | sql                  | without-security     | PASS  |    0 |
[Pipeline] }

[abhinavGupta16]

Thanks for adding the tests.