[Fix] Make spark_version field optional to work with defaults in policies
#4643
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This fails with: I suspect the field needs to be marked as optional in the OpenAPI spec for clusters to make this work. |
pietern
changed the title
spark_version field optional to work with defaults in policiesspark_version field optional to work with defaults in policies
pietern
commented
Apr 17, 2025
| @@ -399,7 +399,7 @@ type Cluster struct { | |||
| ClusterID string `json:"cluster_id,omitempty"` | |||
| ClusterName string `json:"cluster_name,omitempty"` | |||
|
|
|||
| SparkVersion string `json:"spark_version"` | |||
| SparkVersion string `json:"spark_version,omitempty"` | |||
There was a problem hiding this comment.
Without this change, the jobs unit tests failed with:
panic: new_cluster: inconsistency: spark_version is optional, default is empty, but has no omitempty
Reflection on the 2.0 structs flagged the incompatibility between an optional field in the schema without a default and a mandatory struct field. This struct is no longer used directly by a resource.
mgyucht
approved these changes
Apr 23, 2025
tanmay-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
|
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
tanmay-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
deco-sdk-tagging bot
added a commit
that referenced
this pull request
Apr 24, 2025
## Release v1.75.0 ### New Features and Improvements * Add support for `power_bi_task` in jobs ([#4647](#4647)) * Add support for `dashboard_task` in jobs ([#4646](#4646)) * Add `compute_mode` to `databricks_mws_workspaces` to support creating serverless workspaces ([#4670](#4670)). * Make `spark_version` optional in the context of jobs such that a cluster policy can provide a default value ([#4643](#4643)) ### Documentation * Document `performance_target` in `databricks_job` ([#4651](#4651)) * Add more examples for `databricks_model_serving` ([#4658](#4658)) * Document `on_streaming_backlog_exceeded` in email/webhook notifications in `databricks_job` ([#4660](#4660)) * Refresh `spark_python_task` option in `databricks_job` ([#4666](#4666)) ### Exporter * Emit files installed with `%pip install` in Python notebooks ([#4664](#4664)) * Correctly handle account-level identities when generating the code ([#4650](#4650)) * Add export of dashboard tasks in `datarbicks_job` ([#4665](#4665)) * Add export of PowerBI tasks in `datarbicks_job` ([#4668](#4668)) * Add `Ignore` implementation for `databricks_grants` to fix issue with wrongly generated dependencies ([#4650](#4650)) * Improve handling of `owner` for UC resources ([#4669](#4669))
github-merge-queue bot
pushed a commit
to databricks/cli
that referenced
this pull request
Apr 25, 2025
## Changes Upgrade TF provider to 1.75.0 Includes databricks/terraform-provider-databricks#4643 which fixes #2755
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
A cluster policy can enforce a specific
spark_versionfield and set it as the default. This mechanism allows for a centralized choice of the Databricks Runtime version across all jobs. To allow a job to inherit this field from a policy, it must be configured as optional in the schema.The job resource referred to
JobSettingsandJobSettingsResourcewith a mix ofjsandjsrvariable names. This PR updates references toJobSettingsResourceto be calledjsr.Tests
make testrun locallydocs/folder