Fix permadiff on dataflow_flex_template_job

[FilipKubawskiOkta]

fixes hashicorp/terraform-provider-google#21322

The sdkPipelineOptions object isn't the same depending on the template causing these values to default to 0 or nil

Release Note Template for Downstream PRs (will be copied)

See Write release notes for guidance.

dataflow: Fields `network`, `subnetwork`, `numWorkers`, `maxNumWorkers` and `machineType` will no longer cause permadiff on `dataflow_flex_template_job`

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[github-actions]

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@slevenick, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google-beta provider: Diff ( 1 file changed, 13 insertions(+), 5 deletions(-))

[modular-magician]

Tests analytics

Total tests: 25
Passed tests: 0
Skipped tests: 25
Affected tests: 0

Click here to see the affected service packages

• dataflow

🟢 All tests passed!

View the build log

[slevenick]

Can you add a test step that specifies these fields? I'd like to make sure this works correctly going forward, having a test for this will ensure that

[FilipKubawskiOkta]

Can you add a test step that specifies these fields? I'd like to make sure this works correctly going forward, having a test for this will ensure that

Yeah sure. Do you want a case that uses the google-made templates or a custom one. If a custom one, is there a way to upload the Docker image of the template to the projects that's running the tests? And a way to upload the json template to a bucket on the project

Resource used to reproduce the issue

resource "google_dataflow_flex_template_job" "flex_job" {
  provider = google-beta
  project = var.project_id
  service_account_email = var.terraform_service_account
  region = var.region
  name = "test-job-123"
  container_spec_gcs_path = "gs://test-bucketsss-123456789/samples/dataflow/templates/wordcount-go.json"
  on_delete = "cancel"
  network = google_compute_network.test-network.name
  subnetwork = google_compute_subnetwork.test-subnetwork.self_link
  parameters = {
    output="gs://test-bucketsss-123456789/samples/dataflow/templates/counts.txt"
  }
  labels = {
   "my_labels" = "value"
  }
  max_workers = 2
  machine_type = "n1-standard-1"
}

Template

{
    "defaultEnvironment": {},
    "image": "europe-central2-docker.pkg.dev/prj-cm-fk-sandbox/flex-templates/dataflow/wordcount-go:latest",
    "metadata": {
        "description": "An Apache Beam batch pipeline that reads text from GCS and counts occurrences of words.",
        "name": "Beam Go Wordcount",
        "parameters": [
            {
                "helpText": "GCS text files to read from.",
                "isOptional": true,
                "label": "File(s) to read.",
                "name": "input"
            },
            {
                "helpText": "GCS location to write output to.",
                "isOptional": false,
                "label": "Output file.",
                "name": "output"
            }
        ]
    },
    "sdkInfo": {
        "language": "GO"
    }
}

Thanks!

[slevenick]

Ideally a google made template, just so that setting up the test will be easier.

You can check out our existing tests and copy or modify one of them: https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/third_party/terraform/services/dataflow/resource_dataflow_flex_template_job_test.go.tmpl

[FilipKubawskiOkta]

done

[github-actions]

@slevenick This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

[github-actions]

@GoogleCloudPlatform/terraform-team @slevenick This PR has been waiting for review for 1 week. Please take a look! Use the label disable-review-reminders to disable these notifications.

[github-actions]

@GoogleCloudPlatform/terraform-team @slevenick This PR has been waiting for review for 2 weeks. Please take a look! Use the label disable-review-reminders to disable these notifications.

[FilipKubawskiOkta]

@slevenick is anything needed here?

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google-beta provider: Diff ( 2 files changed, 126 insertions(+), 5 deletions(-))

[modular-magician]

Non-exercised tests

🔴 Tests were added that are skipped in VCR:

• TestAccDataflowFlexTemplateJob_workerPoolsFallback

Tests analytics

Total tests: 27
Passed tests: 0
Skipped tests: 27
Affected tests: 0

Click here to see the affected service packages

• dataflow

🟢 All tests passed!

View the build log

[github-actions]

@GoogleCloudPlatform/terraform-team @slevenick This PR has been waiting for review for 3 weeks. Please take a look! Use the label disable-review-reminders to disable these notifications.

[slevenick]

Sorry, was out for a couple weeks there! I'm kicking off the test in CI as it's difficult to run locally. If that passes I'll approve