Skip to content

Support subfolders for distribution types for build and assemble locally and in Jenkins #1807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 84 commits into from
Apr 8, 2022

Conversation

tianleh
Copy link
Member

@tianleh tianleh commented Mar 23, 2022

Signed-off-by: Tianle Huang [email protected]

Description

Support subfolders for distribution types. E.g

current builds/opensearch/dist/ -> builds/opensearch/tar/dist/
current dist/opensearch/ -> dist/opensearch/rpm/

Issues Resolved

Resolves #1800

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov-commenter
Copy link

codecov-commenter commented Mar 23, 2022

Codecov Report

Merging #1807 (ff50022) into main (4df94c4) will decrease coverage by 0.03%.
The diff coverage is 75.86%.

@@             Coverage Diff              @@
##               main    #1807      +/-   ##
============================================
- Coverage     94.60%   94.56%   -0.04%     
  Complexity       20       20              
============================================
  Files           178      179       +1     
  Lines          3633     3645      +12     
  Branches         27       29       +2     
============================================
+ Hits           3437     3447      +10     
  Misses          192      192              
- Partials          4        6       +2     
Impacted Files Coverage Δ
src/build_workflow/build_args.py 100.00% <ø> (ø)
src/build_workflow/builder_from_source.py 100.00% <ø> (ø)
...sts/jenkins/jobs/ArchiveAssembleUpload_Jenkinsfile 100.00% <ø> (ø)
tests/jenkins/jobs/BuildArchive_Jenkinsfile 100.00% <ø> (ø)
tests/jenkins/jobs/BuildAssembleUpload_Jenkinsfile 100.00% <ø> (ø)
...ts/jenkins/jobs/uploadMinSnapshotsToS3_Jenkinsfile 100.00% <ø> (ø)
src/jenkins/BuildManifest.groovy 86.00% <30.00%> (-11.57%) ⬇️
src/assemble_workflow/bundle_file_location.py 100.00% <100.00%> (ø)
src/assemble_workflow/bundle_location.py 92.30% <100.00%> (+0.64%) ⬆️
src/assemble_workflow/bundle_locations.py 100.00% <100.00%> (ø)
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4df94c4...ff50022. Read the comment docs.

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

Send for early feedback on the functional part. Local test, Jenkins test and unit tests will be added soon.

For now, I put the parameter DISTRIBUTION in the Jenkins input for ease of testing. I would suggest that we always build all supported distribution types together in one job. (Similar to x64 and arm64) Otherwise, I will also need to update the latest feature since different distributions could have a different latest build number.

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.1.1/latest/linux/x64/dist/opensearch/tar/manifest.yml

cc @peterzhuamazon @dblock

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

Local test for OSD:

Run this
./build.sh manifests/2.0.0/opensearch-dashboards-2.0.0.yml

The output is

2022-03-23 02:28:13 INFO     Created build manifest /usr/share/opensearch/workspace/opensearch-build/builds/opensearch-dashboards/tar/manifest.yml
2022-03-23 02:28:18 INFO     Done.

Verify that the tar subfolder is added.
Screen Shot 2022-03-22 at 7 33 23 PM

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

Local test for assembling OSD

Run ./assemble.sh builds/opensearch-dashboards/tar/manifest.yml

The output is

./assemble.sh builds/opensearch-dashboards/tar/manifest.yml

Verify the file structure
Screen Shot 2022-03-22 at 7 36 26 PM

Verify that the content of the manifest has the correct location information
Screen Shot 2022-03-22 at 7 36 58 PM

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

Local test for building OS

Run ./build.sh manifests/1.3.0/opensearch-1.3.0.yml

Output is

2022-03-23 02:38:48 INFO     Created build manifest /usr/share/opensearch/workspace/opensearch-build/builds/opensearch/tar/manifest.yml
2022-03-23 02:38:50 INFO     Done.

The file structure is
Screen Shot 2022-03-22 at 7 44 00 PM

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

Local test for assembling OS

Run ./assemble.sh builds/opensearch/tar/manifest.yml

Output is

2022-03-23 02:46:23 INFO     Published /usr/share/opensearch/workspace/opensearch-build/dist/opensearch/tar/opensearch-1.3.0-linux-x64.tar.gz.
2022-03-23 02:46:23 INFO     Done.

File structure
Screen Shot 2022-03-22 at 7 48 03 PM

Verify content of the manifest file about location
Screen Shot 2022-03-22 at 7 48 21 PM

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

RPM build for OSD 2.0.0 failed. will try to see if 1.3.0 work

  │ info $ fpm --force -t rpm --package /tmp/tmpnngo78v2/OpenSearch-Dashboards/target/NAME-2.0.0-ARCH.TYPE -s dir --name opensearch-dashboards --description Explore and visualize your OpenSearch data --version 2.0.0 --url https://opensearch.org --vendor OpenSearch --maintainer OpenSearch Dashboards Team <[email protected]> --license ASL 2.0 --conflicts opensearch-dashboards --after-install /tmp/tmpnngo78v2/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/package_scripts/post_install.sh --before-install /tmp/tmpnngo78v2/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/package_scripts/pre_install.sh --before-remove /tmp/tmpnngo78v2/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/package_scripts/pre_remove.sh --after-remove /tmp/tmpnngo78v2/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/package_scripts/post_remove.sh --config-files /etc/opensearch-dashboards/opensearch_dashboards.yml --template-value user=opensearch-dashboards --template-value group=opensearch-dashboards --template-value optimizeDir=/usr/share/opensearch-dashboards/optimize --template-value configDir=/etc/opensearch-dashboards --template-value pluginsDir=/usr/share/opensearch-dashboards/plugins --template-value dataDir=/var/lib/opensearch-dashboards --exclude usr/share/opensearch-dashboards/config --exclude usr/share/opensearch-dashboards/data --exclude usr/share/opensearch-dashboards/assets --architecture x64 --rpm-os linux /tmp/tmpnngo78v2/OpenSearch-Dashboards/build/opensearch-dashboards-2.0.0-linux-x64/=/usr/share/opensearch-dashboards/ /tmp/tmpnngo78v2/OpenSearch-Dashboards/build/opensearch-dashboards-2.0.0-linux-x64/config/=/etc/opensearch-dashboards/ /tmp/tmpnngo78v2/OpenSearch-Dashboards/build/opensearch-dashboards-2.0.0-linux-x64/data/=/var/lib/opensearch-dashboards/ /tmp/tmpnngo78v2/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/service_templates/sysv/=/ /tmp/tmpnngo78v2/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/service_templates/systemd/=/
   │ERROR failure 0 sec
   │ERROR Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
   │          at new NodeError (internal/errors.js:322:7)
   │          at Socket.onclose (internal/streams/end-of-stream.js:117:38)
   │          at Socket.emit (events.js:412:35)
   │          at Pipe.<anonymous> (net.js:686:12)

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2022-03-23 08:04:55 ERROR    Error building OpenSearch-Dashboards, retry with: ./build.sh manifests/2.0.0/opensearch-dashboards-2.0.0.yml --component OpenSearch-Dashboards
Traceback (most recent call last):
  File "./src/run_build.py", line 81, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 69, in main
    builder.build(build_recorder)
  File "/usr/share/opensearch/workspace/opensearch-build/src/build_workflow/builder_from_source.py", line 55, in build
    self.git_repo.execute(build_command)
  File "/usr/share/opensearch/workspace/opensearch-build/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /usr/share/opensearch/workspace/opensearch-build/scripts/components/OpenSearch-Dashboards/build.sh -v 2.0.0 -p linux -a x64 -d rpm -s false -o builds' returned non-zero exit status 1.

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

RPM build for OSD 1.3.0 also failed. We may need to fix the RPM build issue or skip the test given the logic flow is proved by tar build.

│ info $ fpm --force -t rpm --package /tmp/tmpend213sm/OpenSearch-Dashboards/target/NAME-1.3.0-ARCH.TYPE -s dir --name opensearch-dashboards --description Explore and visualize your OpenSearch data --version 1.3.0 --url https://opensearch.org/ --vendor OpenSearch --maintainer OpenSearch Dashboards Team <[email protected]> --license ASL 2.0 --conflicts opensearch-dashboards --after-install /tmp/tmpend213sm/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/package_scripts/post_install.sh --before-install /tmp/tmpend213sm/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/package_scripts/pre_install.sh --before-remove /tmp/tmpend213sm/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/package_scripts/pre_remove.sh --after-remove /tmp/tmpend213sm/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/package_scripts/post_remove.sh --config-files /etc/opensearch-dashboards/opensearch_dashboards.yml --template-value user=opensearch-dashboards --template-value group=opensearch-dashboards --template-value optimizeDir=/usr/share/opensearch-dashboards/optimize --template-value configDir=/etc/opensearch-dashboards --template-value pluginsDir=/usr/share/opensearch-dashboards/plugins --template-value dataDir=/var/lib/opensearch-dashboards --exclude usr/share/opensearch-dashboards/config --exclude usr/share/opensearch-dashboards/data --exclude usr/share/opensearch-dashboards/assets --architecture x64 --rpm-os linux /tmp/tmpend213sm/OpenSearch-Dashboards/build/opensearch-dashboards-1.3.0-linux-x64/=/usr/share/opensearch-dashboards/ /tmp/tmpend213sm/OpenSearch-Dashboards/build/opensearch-dashboards-1.3.0-linux-x64/config/=/etc/opensearch-dashboards/ /tmp/tmpend213sm/OpenSearch-Dashboards/build/opensearch-dashboards-1.3.0-linux-x64/data/=/var/lib/opensearch-dashboards/ /tmp/tmpend213sm/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/service_templates/sysv/=/ /tmp/tmpend213sm/OpenSearch-Dashboards/src/dev/build/tasks/os_packages/service_templates/systemd/=/
   │ERROR failure 0 sec
   │ERROR Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
   │          at Socket.onclose (internal/streams/end-of-stream.js:52:15)
   │          at Socket.emit (events.js:203:15)
   │          at Pipe._handle.close (net.js:607:12)

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2022-03-23 08:11:55 ERROR    Error building OpenSearch-Dashboards, retry with: ./build.sh manifests/1.3.0/opensearch-dashboards-1.3.0.yml --component OpenSearch-Dashboards
Traceback (most recent call last):
  File "./src/run_build.py", line 81, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 69, in main
    builder.build(build_recorder)
  File "/var/jenkins/workspace/Playground/tianleh-test/tianleh-osd-build-3-22/src/build_workflow/builder_from_source.py", line 55, in build
    self.git_repo.execute(build_command)
  File "/var/jenkins/workspace/Playground/tianleh-test/tianleh-osd-build-3-22/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /var/jenkins/workspace/Playground/tianleh-test/tianleh-osd-build-3-22/scripts/components/OpenSearch-Dashboards/build.sh -v 1.3.0 -p linux -a x64 -d rpm -s false -o builds' returned non-zero exit status 1.

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

RPM build for OS. will add RPM support

Not generating sha for /var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22/builds/opensearch/dist/opensearch-min-1.3.0-SNAPSHOT.x86_64.rpm with artifact Path /var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22/builds/opensearch/dist, doesn't match allowed types [.tar.gz, .zip]

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

RPM build for OS 2.0.0 failed

> Task :opensearch-job-scheduler-spi:compileJava FAILED
/tmp/tmp4dsppnd1/job-scheduler/spi/src/main/java/org/opensearch/jobscheduler/spi/utils/LockService.java:86: error: no suitable method found for mapping(String,String,XContentType)
                    .mapping(MAPPING_TYPE, lockMapping(), XContentType.JSON);
                    ^
    method CreateIndexRequest.mapping(String) is not applicable
      (actual and formal argument lists differ in length)
    method CreateIndexRequest.mapping(BytesReference,XContentType) is not applicable
      (actual and formal argument lists differ in length)
    method CreateIndexRequest.mapping(XContentBuilder) is not applicable
      (actual and formal argument lists differ in length)
    method CreateIndexRequest.mapping(Map<String,?>) is not applicable
      (actual and formal argument lists differ in length)
    method CreateIndexRequest.mapping(String,Map<String,?>) is not applicable
      (actual and formal argument lists differ in length)
1 error

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-job-scheduler-spi:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
> Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':opensearch-job-scheduler-spi:compileJava'.
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:145)
	at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:143)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:131)
	at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
	at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:74)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:402)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:389)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:382)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:368)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: org.gradle.api.internal.tasks.compile.CompilationFailedException: Compilation failed; see the compiler error output for details.
	at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:56)
	at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:39)
	at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.delegateAndHandleErrors(NormalizingJavaCompiler.java:97)
	at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.execute(NormalizingJavaCompiler.java:51)
	at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.execute(NormalizingJavaCompiler.java:37)
	at org.gradle.api.internal.tasks.compile.AnnotationProcessorDiscoveringCompiler.execute(AnnotationProcessorDiscoveringCompiler.java:51)
	at org.gradle.api.internal.tasks.compile.AnnotationProcessorDiscoveringCompiler.execute(AnnotationProcessorDiscoveringCompiler.java:37)
	at org.gradle.api.internal.tasks.compile.ModuleApplicationNameWritingCompiler.execute(ModuleApplicationNameWritingCompiler.java:46)
	at org.gradle.api.internal.tasks.compile.ModuleApplicationNameWritingCompiler.execute(ModuleApplicationNameWritingCompiler.java:36)
	at org.gradle.jvm.toolchain.internal.DefaultToolchainJavaCompiler.execute(DefaultToolchainJavaCompiler.java:57)
	at org.gradle.api.tasks.compile.JavaCompile.lambda$createToolchainCompiler$1(JavaCompile.java:232)
	at org.gradle.api.internal.tasks.compile.CleaningJavaCompiler.execute(CleaningJavaCompiler.java:53)
	at org.gradle.api.internal.tasks.compile.incremental.IncrementalCompilerFactory.lambda$createRebuildAllCompiler$0(IncrementalCompilerFactory.java:52)
	at org.gradle.api.internal.tasks.compile.incremental.SelectiveCompiler.execute(SelectiveCompiler.java:67)
	at org.gradle.api.internal.tasks.compile.incremental.SelectiveCompiler.execute(SelectiveCompiler.java:41)
	at org.gradle.api.internal.tasks.compile.incremental.IncrementalResultStoringCompiler.execute(IncrementalResultStoringCompiler.java:66)
	at org.gradle.api.internal.tasks.compile.incremental.IncrementalResultStoringCompiler.execute(IncrementalResultStoringCompiler.java:52)
	at org.gradle.api.internal.tasks.compile.CompileJavaBuildOperationReportingCompiler$2.call(CompileJavaBuildOperationReportingCompiler.java:59)
	at org.gradle.api.internal.tasks.compile.CompileJavaBuildOperationReportingCompiler$2.call(CompileJavaBuildOperationReportingCompiler.java:51)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
	at org.gradle.api.internal.tasks.compile.CompileJavaBuildOperationReportingCompiler.execute(CompileJavaBuildOperationReportingCompiler.java:51)
	at org.gradle.api.tasks.compile.JavaCompile.performCompilation(JavaCompile.java:279)
	at org.gradle.api.tasks.compile.JavaCompile.performIncrementalCompilation(JavaCompile.java:165)
	at org.gradle.api.tasks.compile.JavaCompile.compile(JavaCompile.java:146)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
	at org.gradle.api.internal.project.taskfactory.IncrementalInputsTaskAction.doExecute(IncrementalInputsTaskAction.java:32)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
	at org.gradle.api.internal.project.taskfactory.AbstractIncrementalTaskAction.execute(AbstractIncrementalTaskAction.java:25)
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
	at org.gradle.api.internal.tasks.execution.TaskExecution$2.run(TaskExecution.java:239)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
	at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:224)
	at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:207)
	at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:190)
	at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:168)
	at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:89)
	at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:40)
	at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:53)
	at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:50)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:50)
	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:40)
	at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:68)
	at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:38)
	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:48)
	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:36)
	at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
	at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
	at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
	at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:51)
	at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:29)
	at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:61)
	at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:42)
	at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:60)
	at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:27)
	at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:188)
	at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:75)
	at org.gradle.internal.Either$Right.fold(Either.java:175)
	at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
	at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:73)
	at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:48)
	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:38)
	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:27)
	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:109)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:56)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:56)
	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:38)
	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:73)
	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:44)
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:89)
	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:50)
	at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:114)
	at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:57)
	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:76)
	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:50)
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.lambda$execute$2(SkipEmptyWorkStep.java:93)
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:93)
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:34)
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
	at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:43)
	at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:31)
	at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:40)
	at org.gradle.api.internal.tasks.execution.TaskExecution$3.withWorkspace(TaskExecution.java:284)
	at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:40)
	at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
	at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
	at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
	at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:44)
	at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:33)
	at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:76)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:142)
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:131)
	at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
	at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:74)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:402)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:389)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:382)
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:368)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)


* Get more help at https://help.gradle.org/

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 5s
1 actionable task: 1 executed
2022-03-23 17:53:45 ERROR    Error building job-scheduler, retry with: ./build.sh manifests/2.0.0/opensearch-2.0.0.yml --component job-scheduler --snapshot
Traceback (most recent call last):
  File "./src/run_build.py", line 80, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 68, in main
    builder.build(build_recorder)
  File "/var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22/src/build_workflow/builder_from_source.py", line 55, in build
    self.git_repo.execute(build_command)
  File "/var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /tmp/tmp4dsppnd1/job-scheduler/scripts/build.sh -v 2.0.0 -q alpha1 -p linux -a arm64 -s true -o builds' returned non-zero exit status 1.

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

RPM assemble for OS 1.3.0 failed

+ ./assemble.sh builds/opensearch/rpm/manifest.yml --base-url https://ci.opensearch.org/ci/dbc/Playground/tianleh-test/tianle-opensearch-build-3-22/1.3.0/11/linux/x64
Installing dependencies in . ...
Installing dependencies from Pipfile.lock (16e10a)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Running ./src/run_assemble.py builds/opensearch/rpm/manifest.yml --base-url https://ci.opensearch.org/ci/dbc/Playground/tianleh-test/tianle-opensearch-build-3-22/1.3.0/11/linux/x64 ...
2022-03-23 18:13:17 INFO     Bundling OpenSearch (x64) on linux into /var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22@2/dist/opensearch/rpm ...
2022-03-23 18:13:18 INFO     Copied min bundle to /tmp/tmp10vf1fpb/opensearch-min-1.3.0.x86_64.rpm.
2022-03-23 18:13:18 INFO     Start creating distribution rpm for OpenSearch.
2022-03-23 18:13:18 INFO     Extracting dist into /tmp/tmp10vf1fpb.
2022-03-23 18:13:18 INFO     Convert rpm to cpio for extraction: /tmp/tmp10vf1fpb/opensearch-min-1.3.0.x86_64.rpm to /tmp/tmp10vf1fpb/opensearch-min-1.3.0.x86_64.cpio
2022-03-23 18:13:20 INFO     Extract cpio /tmp/tmp10vf1fpb/opensearch-min-1.3.0.x86_64.cpio content to /tmp/tmp10vf1fpb
Traceback (most recent call last):
  File "./src/run_assemble.py", line 58, in <module>
    sys.exit(main())
  File "./src/run_assemble.py", line 42, in main
    with Bundles.create(build_manifest, artifacts_dir, bundle_recorder, args.keep) as bundle:
  File "/var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22@2/src/assemble_workflow/bundles.py", line 31, in create
    return klass(build_manifest, artifacts_dir, bundle_recorder, keep)  # type: ignore[no-any-return, operator]
  File "/var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22@2/src/assemble_workflow/bundle.py", line 49, in __init__
    self.min_dist = self.__get_min_dist(build_manifest.components)
  File "/var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22@2/src/assemble_workflow/bundle.py", line 140, in __get_min_dist
    min_dist.extract(self.tmp_dir.name)
  File "/var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22@2/src/assemble_workflow/dist.py", line 73, in extract
    self.__extract__(dest)
  File "/var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22@2/src/assemble_workflow/dist.py", line 113, in __extract__
    BundleRpm(self.filename, self.path, self.min_path).extract(dest)
  File "/var/jenkins/workspace/Playground/tianleh-test/tianle-opensearch-build-3-22@2/src/assemble_workflow/bundle_rpm.py", line 51, in extract
    cwd=dest,
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cpio', '-imdv']' returned non-zero exit status 2.

@tianleh
Copy link
Member Author

tianleh commented Mar 23, 2022

For the test related logic, e.g integ and bwc, I plan to hardcode to tar in this PR. I create an issue to introduce distribution parameter to the test command #1812

@tianleh tianleh requested a review from seraphjiang March 23, 2022 20:06
@tianleh tianleh changed the title Support subfolders for distribution types for build and assemble Support subfolders for distribution types for build and assemble locally and in Jenkins Mar 25, 2022
@tianleh tianleh marked this pull request as ready for review March 26, 2022 06:43
@tianleh tianleh requested a review from a team as a code owner March 26, 2022 06:43
tianleh added 20 commits April 7, 2022 20:48
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Signed-off-by: Tianle Huang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] Add new subfolder to support more distributions
8 participants