Not able to execute Jenkins pipeline on the dynamic provisioned VM.

[mkjkec2005]

Jenkins and plugins versions report

Environment

  Jenkins: 2.426.3
OS: Linux - 5.14.0-503.15.1.el9_5.x86_64
Java: 17.0.13 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
active-directory:2.38
analysis-model-api:12.6.0
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.4-124.v31e2987e48f4
artifactory:4.0.8
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.119.v50285141b_7e1
aws-credentials:231.v08a_59f17d742
aws-java-sdk-ec2:1.12.767-467.vb_e93f0c614b_6
aws-java-sdk-minimal:1.12.767-467.vb_e93f0c614b_6
blueocean:1.27.16
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.16
blueocean-commons:1.27.16
blueocean-config:1.27.16
blueocean-core-js:1.27.16
blueocean-dashboard:1.27.16
blueocean-display-url:2.4.3
blueocean-events:1.27.16
blueocean-git-pipeline:1.27.16
blueocean-github-pipeline:1.27.16
blueocean-i18n:1.27.16
blueocean-jira:1.27.16
blueocean-jwt:1.27.16
blueocean-personalization:1.27.16
blueocean-pipeline-api-impl:1.27.16
blueocean-pipeline-editor:1.27.16
blueocean-pipeline-scm-api:1.27.16
blueocean-rest:1.27.16
blueocean-rest-impl:1.27.16
blueocean-web:1.27.16
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1178.v969d9eb_c728e
build-name-setter:2.4.3
build-timeout:1.32
build-user-vars-plugin:166.v52976843b_435
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.1
cloud-stats:336.v788e4055508b_
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-compress-api:1.26.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.12.0-126.v7b_5df3fa_5444
conditional-buildstep:1.4.3
config-driven-pipeline:1.3
config-file-provider:980.v88956a_a_5d6a_d
configuration-as-code:1836.vccda_4a_122a_a_e
copyartifact:757.v05365583a_455
credentials:1380.va_435002fa_924
credentials-binding:681.vf91669a_32e45
dashboard-view:2.517.v776a_b_811a_b_4e
data-tables-api:2.1.8-1
display-url-api:2.204.vf6fddd8a_8b_e9
docker-build-step:2.12
docker-commons:445.v6b_646c962a_94
docker-java-api:3.4.0-94.v65ced49b_a_7d5
docker-plugin:1.5
docker-workflow:580.vc0c340686b_54
durable-task:568.v8fb_5c57e8417
echarts-api:5.5.1-4
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:2.105
envinject:2.919.v009a_a_1067cd0
envinject-api:1.199.v3ce31253ed13
extended-choice-parameter:382.v5697b_32134e8
external-monitor-job:215.v2e88e894db_f8
favorite:2.221.v19ca_666b_62f5
font-awesome-api:6.6.0-2
forensics-api:2.6.0
generic-webhook-trigger:2.2.5
git:5.2.2
git-client:4.7.0
git-parameter:0.9.19
git-server:126.v0d945d8d2b_39
github:1.40.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1793.v1831e9c68d77
github-pullrequest:0.7.0
gitlab-plugin:1.8.2
google-compute-engine:4.579.vc9f7ca_9f728a_
google-oauth-plugin:1.330.vf5e86021cb_ec
gradle:2.13.1
groovy:457.v99900cb_85593
gson-api:2.11.0-41.v019fcf6125dc
h2-api:11.1.4.199-30.v1c64e772f3a_c
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.36
http_request:1.19
icon-shim:3.0.0
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
ivy:2.6
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.16
jersey2-api:2.44-151.v6df377fff741
jira:3.13
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-2
job-dsl:1.87
job-import-plugin:3.6
jobConfigHistory:1283.veb_dfb_00b_5ec0
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1311.v39e1716e4eb_e
kubernetes:4302.va_756e4b_67715
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:190.v03c305394deb_
ldap:725.v3cb_b_711b_1a_ef
localization-support:1.2
localization-zh-cn:371.v23851f835d6b_
lockable-resources:1245.1247.vea_384d871c85
mailer:470.vc91f60c5d8e2
mapdb-api:1.0.9-40.v58107308b_7a_7
matrix-auth:3.2.3
matrix-project:822.824.v14451b_c0fd42
maven-plugin:3.23
mercurial:1260.vdfb_723cdcc81
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.14.0-133.vcc091215a_358
mina-sshd-api-core:2.14.0-133.vcc091215a_358
momentjs:1.1.1
nodejs:1.6.1
oauth-credentials:0.653.v14cf2088e950
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.11
parameterized-trigger:806.vf6fff3e28c3e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:61.v629f2cc41d83
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:744.v5b_556ee7c253
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2218.v56d0cda_37c72
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2218.v56d0cda_37c72
pipeline-model-extensions:2.2218.v56d0cda_37c72
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2218.v56d0cda_37c72
pipeline-stage-view:2.34
pipeline-utility-steps:2.17.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:4.1.0
popper-api:1.16.1-3
popper2-api:2.11.6-5
prism-api:1.29.0-18
publish-over:0.22
publish-over-ssh:383.v4eb_4c44da_2dd
pubsub-light:1.18
rebuild:332.va_1ee476d8f6d
resource-disposer:0.24
role-strategy:717.v6a_69a_fe98974
run-condition:1.7
scm-api:696.v778d637b_a_762
script-security:1369.v9b_98a_4e95b_2d
simple-queue:1.4.6
slack:722.vd07f1ea_7ff40
snakeyaml-api:2.3-123.v13484c65210a_
sonar:2.17.2
sse-gateway:1.27
ssh:158.ve2a_e90fb_7319
ssh-agent:367.vf9076cd4ee21
ssh-credentials:343.v884f71d78167
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
subversion:1268.v0581f58a_b_3cd
thinBackup:2.1.1
throttle-concurrents:2.14
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
uno-choice:2.8.3
variant:60.v7290fc0eb_b_cd
windows-slaves:1.8.1
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1322.v857eeeea_9902
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:4000.v5198556e9cea_
workflow-cps-global-lib:612.v55f2f80781ef
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1385.vb_58b_86ea_fff1
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:920.v59f71ce16f04
ws-cleanup:0.46

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

1. Trigger Jenkins pipeline running on GCE cloud.

Expected Results

Pipeline creates a node (dynamic VM in GCP) and runs the stages.

Actual Results

No node is provisioned and the pipeline is stuck, waiting for node to get provisioned.

[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘Jenkins’ doesn’t have label ‘gx-dcc-pool’
‘gcp-vm-1’ doesn’t have label ‘gx-dcc-pool’
‘gcp-vm-2’ doesn’t have label ‘gx-dcc-pool’
‘gcp-vm-3’ doesn’t have label ‘gx-dcc-pool’
‘gcp-vm-4’ doesn’t have label ‘gx-dcc-pool’
‘gcp-vm-5’ doesn’t have label ‘gx-dcc-pool’
‘gx-dcc-pool-s66v6g’ is offline
‘gcp-vm-6’ doesn’t have label ‘gx-dcc-pool’
‘gcp-vm-7’ doesn’t have label ‘gx-dcc-pool’
‘gcp-vm-8’ doesn’t have label ‘gx-dcc-pool’

Anything else?

I see the below error in the Jenkins logs.

Error retrieving operation.
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
GET https://compute.googleapis.com/compute/v1/projects/gcp-systems-prj-prod-moto-6c55/zones/us-west1-a/operations/operation-1737353928494-62c1d3bc1cc70-473a6a62-4e85a582
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "The resource 'projects/gcp-systems-prj-prod-moto-6c55/zones/us-west1-a/operations/operation-1737353928494-62c1d3bc1cc70-473a6a62-4e85a582' was not found",
"reason" : "notFound"
} ],
"message" : "The resource 'projects/gcp-systems-prj-prod-moto-6c55/zones/us-west1-a/operations/operation-1737353928494-62c1d3bc1cc70-473a6a62-4e85a582' was not found"
}

---

Are you interested in contributing a fix?

No response

[gbhat618]

Just to make sure the IAM permissions are properly granted for the Jenkins Service Account to create the VM,
could you check, if these exist

instanceAdmin, networkAdmin and serviceAccountUser roles to the service account.

notes from https://github.com/jenkinsci/google-compute-engine-plugin/blob/develop/docs/Home.md#iam-credentials