Skip to content

Support serverless workspaces in databricks_mws_workspaces #4670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 24, 2025

Conversation

mgyucht
Copy link
Contributor

@mgyucht mgyucht commented Apr 23, 2025

Changes

A new field compute_mode has been added to databricks_mws_workspaces. This can be set to SERVERLESS to indicate that a workspace should be serverless. Serverless workspaces are documented at https://docs.databricks.com/aws/en/admin/workspace/serverless-workspaces. These workspaces need neither credentials_id or storage_configuration_id, so the setup should be much easier.

Tests

Added an integration test for creating serverless workspaces in AWS.

Todos

  • Need to verify if serverless workspaces work in GCP. EDIT: not yet supported.
  • Need to verify which regions users can specify for serverless workspaces. EDIT: this will be documented shortly. I'll include a link.
  • The test does not work right now, I think our E2 account still needs to be onboarded. EDIT: our account was onboarded and we verified that this works.

@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 11:43 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 11:44 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 11:48 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 11:48 — with GitHub Actions Inactive
@@ -577,7 +583,7 @@ func ResourceMwsWorkspaces() common.Resource {
var workspace Workspace
workspacesAPI := NewWorkspacesAPI(ctx, c)
common.DataToStructPointer(d, workspaceSchema, &workspace)
if err := requireFields(c.IsAws(), d, "aws_region", "credentials_id", "storage_configuration_id"); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have two cases - when compute_mode is serverless, then use this new condition, and when isn't, then use the old condition?

@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 12:29 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 12:30 — with GitHub Actions Inactive
@@ -5,7 +5,11 @@
### New Features and Improvements

* Add support for `power_bi_task` in jobs ([#4647](https://github.com/databricks/terraform-provider-databricks/pull/4647))
<<<<<<< HEAD
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to review merge conflict

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah new laptop/editor, need to setup autosave.

@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 12:57 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 12:58 — with GitHub Actions Inactive
@mgyucht mgyucht marked this pull request as ready for review April 23, 2025 13:06
@mgyucht mgyucht requested review from a team as code owners April 23, 2025 13:06
@mgyucht mgyucht requested review from parthban-db and removed request for a team April 23, 2025 13:06
@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 20:14 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 23, 2025 20:15 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 24, 2025 07:13 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 24, 2025 07:15 — with GitHub Actions Inactive
@mgyucht mgyucht temporarily deployed to test-trigger-is April 24, 2025 07:17 — with GitHub Actions Inactive
Copy link

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/terraform

Inputs:

  • PR number: 4670
  • Commit SHA: fb23cd23f90b18c3543caab43025d71f713a6fda

Checks will be approved automatically on success.

@mgyucht mgyucht temporarily deployed to test-trigger-is April 24, 2025 07:18 — with GitHub Actions Inactive
@mgyucht mgyucht added this pull request to the merge queue Apr 24, 2025
Merged via the queue into main with commit 557785b Apr 24, 2025
12 checks passed
@mgyucht mgyucht deleted the feat/add-compute-mode-databricks-mws-workspaces branch April 24, 2025 08:41
deco-sdk-tagging bot added a commit that referenced this pull request Apr 24, 2025
## Release v1.75.0

### New Features and Improvements

 * Add support for `power_bi_task` in jobs ([#4647](#4647))
 * Add support for `dashboard_task` in jobs ([#4646](#4646))
 * Add `compute_mode` to `databricks_mws_workspaces` to support creating serverless workspaces ([#4670](#4670)).
 * Make `spark_version` optional in the context of jobs such that a cluster policy can provide a default value ([#4643](#4643))
 

### Documentation

 * Document `performance_target` in `databricks_job` ([#4651](#4651))
 * Add more examples for `databricks_model_serving` ([#4658](#4658))
 * Document `on_streaming_backlog_exceeded` in email/webhook notifications in `databricks_job` ([#4660](#4660))
 * Refresh `spark_python_task` option in `databricks_job` ([#4666](#4666))

### Exporter

 * Emit files installed with `%pip install` in Python notebooks ([#4664](#4664))
 * Correctly handle account-level identities when generating the code ([#4650](#4650))
 * Add export of dashboard tasks in `datarbicks_job` ([#4665](#4665))
 * Add export of PowerBI tasks in `datarbicks_job` ([#4668](#4668))
 * Add `Ignore` implementation for `databricks_grants` to fix issue with wrongly generated dependencies ([#4650](#4650))
 * Improve handling of `owner` for UC resources ([#4669](#4669))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants