FAST project templates example (GoogleCloudPlatform#2897)

* wip

* project factory providers

* working example

* copyright, tfdoc

* rewording

* rewording

* tfdoc

* tfdoc

* tfdoc again

* fix tests

* tests

Author: ludoo

fast/project-templates/os-apt-registries/README.md

@@ -0,0 +1,92 @@
+# Artifact Registry APT Remote Registries
+
+This simple setup allows creating and configuring remote APT repositories, that can be used for instance package updates without the need for an Internet connection.
+
+## Prerequisites
+
+The [`project.yaml`](./project.yaml) file describes the project-level configuration needed in terms of API activation and IAM bindings.
+
+If you are deploying this inside a FAST-enabled organization, the file can be lightly edited to match your configuration, and then used directly in the [project factory](../../stages/2-project-factory/).
+
+This Terraform can of course be deployed using any pre-existing project. In that case use the YAML file to determine the configuration you need to set on the project:
+
+- enable the APIs listed under `services`
+- grant the permissions listed under `iam` to the principal running Terraform, either machine (service account) or human
+
+## VPC-SC Integration
+
+Access to upstream sources from inside a VPC-SC service perimeter [requires specific activation](https://cloud.google.com/artifact-registry/docs/repositories/remote-repo#vpc), which depends on a high-level IAM role on the VPC-SC policy.
+
+Granting such a role to the identity running this setup (either machine or human) is not realistic, so the choice made here is to output the relevant command, so that a VPC-SC administrator can run it using the appropriate credentials. The [relevant Terraform resource](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/artifact_registry_vpcsc_config) can of course be used to automate this task when needed.
+
+## Instance-level Access to the Repository
+
+Instances that need access to the created registries require the `roles/artifactregistry.writer` role assigned to the instance service accounts. This can be automated via the `apt_remote_registries` variable described below, to create IAm bindings for each registry.
+
+It's also possible (and maybe desirable) to grant the role at the project level, if access to multiple repositories is needed from the same set of principals. This needs of course to happen where the project is managed, for example in the project factory YAML file.
+
+Once proper access has been configured, the `apt_configs` output can be used as a basis to configure the APT sources lists on each instance.
+
+Instance need to have the `apt-transport-artifact-registry` package installed, which is served by the default internal repositories configured on GCE base images.
+
+```bash
+sudo apt install apt-transport-artifact-registry
+```
+
+## Variable Configuration
+
+This is an example of running this stage. Note that the `apt_remote_registries` has a default value that can be used when no IAM is needed at the registry level, and the default set of remotes is fine.
+
+```hcl
+project_id = "my-project"
+location   = "europe-west3"
+apt_remote_registries = [
+  { path = "DEBIAN debian/dists/bookworm" },
+  {
+    path = "DEBIAN debian-security/dists/bookworm-security"
+    # grant specific access permissions to this registry
+    writer_principals = [
+      "serviceAccount:[email protected]"
+    ]
+  }
+]
+# tftest skip
+```
+<!-- BEGIN TFDOC -->
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---:|:---:|:---:|
+| [project_id](variables.tf#L56) | Project id where the registries will be created. | <code>string</code> | ✓ |  |
+| [apt_remote_registries](variables.tf#L17) | Remote artifact registry configurations. | <code title="list&#40;object&#40;&#123;&#10;  path              &#61; string&#10;  writer_principals &#61; optional&#40;list&#40;string&#41;, &#91;&#93;&#41;&#10;&#125;&#41;&#41;">list&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> |  | <code title="&#91;&#10;  &#123; path &#61; &#34;DEBIAN debian&#47;dists&#47;bookworm&#34; &#125;,&#10;  &#123; path &#61; &#34;DEBIAN debian-security&#47;dists&#47;bookworm-security&#34; &#125;&#10;&#93;">&#91;&#8230;&#93;</code> |
+| [location](variables.tf#L43) | Region where the registries will be created. | <code>string</code> |  | <code>&#34;europe-west8&#34;</code> |
+| [name](variables.tf#L49) | Prefix used for all resource names. | <code>string</code> |  | <code>&#34;apt-remote&#34;</code> |
+
+## Outputs
+
+| name | description | sensitive |
+|---|---|:---:|
+| [apt_configs](outputs.tf#L23) | APT configurations for remote registries. |  |
+| [vpcsc_command](outputs.tf#L33) | Command to allow egress to remotes from inside a perimeter. |  |
+<!-- END TFDOC -->
+
+## Test
+
+```hcl
+module "test" {
+  source     = "./fabric/fast/project-templates/os-apt-registries"
+  project_id = "my-project"
+  location   = "europe-west3"
+  apt_remote_registries = [
+    { path = "DEBIAN debian/dists/bookworm" },
+    {
+      path = "DEBIAN debian-security/dists/bookworm-security"
+      # grant specific access permissions to this registry
+      writer_principals = [
+        "serviceAccount:[email protected]"
+      ]
+    }
+  ]
+}
+# tftest modules=3 resources=4
+```

fast/project-templates/os-apt-registries/main.tf

@@ -0,0 +1,41 @@
+/**
+ * Copyright 2025 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  apt_remote_registries = {
+    for v in var.apt_remote_registries : (v.path) => merge(v, {
+      name = element(split("/", split(" ", v.path)[1]), -1)
+    })
+  }
+}
+
+module "registries" {
+  source     = "../../../modules/artifact-registry"
+  for_each   = local.apt_remote_registries
+  project_id = var.project_id
+  location   = var.location
+  name       = "${var.name}-${each.value.name}"
+  format = {
+    apt = {
+      remote = {
+        public_repository = each.value.path
+      }
+    }
+  }
+  iam = {
+    "roles/artifactregistry.writer" = each.value.writer_principals
+  }
+}

fast/project-templates/os-apt-registries/outputs.tf

@@ -0,0 +1,38 @@
+/**
+ * Copyright 2025 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  format = "deb ar+https://%s-apt.pkg.dev/remote/%s/%s %s main"
+}
+
+# europe-west8-apt.pkg.dev/ldj-prod-os-apt-0/apt-remote-bookworm
+
+output "apt_configs" {
+  description = "APT configurations for remote registries."
+  value = {
+    for k, v in module.registries : v.name => format(
+      local.format, var.location, var.project_id,
+      v.name, local.apt_remote_registries[k].name
+    )
+  }
+}
+
+output "vpcsc_command" {
+  description = "Command to allow egress to remotes from inside a perimeter."
+  value = (
+    "gcloud artifacts vpcsc-config allow --project=${var.project_id} --location=${var.location}"
+  )
+}

fast/project-templates/os-apt-registries/project.yaml

@@ -0,0 +1,46 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# yaml-language-server: $schema=../../stages/2-project-factory/schemas/project.schema.json
+
+# TODO: edit and uncomment the following line to create the project in a folder
+# parent: shared
+
+name: os-apt-0
+services:
+  - accesscontextmanager.googleapis.com
+  - artifactregistry.googleapis.com
+
+automation:
+  # TODO: edit the automation project and optionally edit resource names
+  project: pf-automation-0
+  service_accounts:
+    rw:
+      description: Read/write automation service account for apt registries.
+  buckets:
+    tf-state:
+      description: Terraform state bucket for apt registries.
+      iam:
+        roles/storage.objectCreator:
+          - rw
+        roles/storage.objectViewer:
+          - rw
+iam:
+  roles/viewer:
+    - rw
+  roles/artifactregistry.admin:
+    - rw
+  # TODO: add instance service accounts that need access to the registries
+  # roles/artifactregistry.writer:
+  #   - serviceAccount:foo@bar

fast/project-templates/os-apt-registries/variables.tf

@@ -0,0 +1,59 @@
+/**
+ * Copyright 2025 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "apt_remote_registries" {
+  description = "Remote artifact registry configurations."
+  type = list(object({
+    path              = string
+    writer_principals = optional(list(string), [])
+  }))
+  nullable = false
+  default = [
+    { path = "DEBIAN debian/dists/bookworm" },
+    { path = "DEBIAN debian-security/dists/bookworm-security" }
+  ]
+  validation {
+    condition = alltrue([
+      for v in var.apt_remote_registries : length(split(" ", v.path)) == 2
+    ])
+    error_message = "Invalid registry path: format is [BASE] [path]."
+  }
+  validation {
+    condition = alltrue([
+      for v in var.apt_remote_registries :
+      contains(["DEBIAN", "UBUNTU"], element(split(" ", v.path), 0))
+    ])
+    error_message = "Invalid registry base: only 'DEBIAN' and 'UBUNTU' are supported."
+  }
+}
+
+variable "location" {
+  description = "Region where the registries will be created."
+  type        = string
+  default     = "europe-west8"
+}
+
+variable "name" {
+  description = "Prefix used for all resource names."
+  type        = string
+  nullable    = true
+  default     = "apt-remote"
+}
+
+variable "project_id" {
+  description = "Project id where the registries will be created."
+  type        = string
+}

fast/stages/2-project-factory/README.md

@@ -342,31 +342,34 @@ The approach is not shown here but reasonably easy to implement. The main projec
 <!-- BEGIN TFDOC -->
 ## Files
 
-| name | description | modules |
-|---|---|---|
-| [main.tf](./main.tf) | Project factory. | <code>project-factory</code> |
-| [outputs.tf](./outputs.tf) | Module outputs. |  |
-| [variables-fast.tf](./variables-fast.tf) | None |  |
-| [variables.tf](./variables.tf) | Module variables. |  |
+| name | description | modules | resources |
+|---|---|---|---|
+| [main.tf](./main.tf) | Project factory. | <code>project-factory</code> |  |
+| [outputs.tf](./outputs.tf) | Module outputs. |  | <code>google_storage_bucket_object</code> · <code>local_file</code> |
+| [variables-fast.tf](./variables-fast.tf) | None |  |  |
+| [variables.tf](./variables.tf) | Module variables. |  |  |
 
 ## Variables
 
 | name | description | type | required | default | producer |
 |---|---|:---:|:---:|:---:|:---:|
-| [billing_account](variables-fast.tf#L17) | Billing account id. If billing account is not part of the same org set `is_org_level` to false. | <code title="object&#40;&#123;&#10;  id           &#61; string&#10;  is_org_level &#61; optional&#40;bool, true&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ |  | <code>0-bootstrap</code> |
-| [prefix](variables-fast.tf#L65) | Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants. | <code>string</code> | ✓ |  | <code>0-bootstrap</code> |
+| [automation](variables-fast.tf#L17) | Automation resources created by the bootstrap stage. | <code title="object&#40;&#123;&#10;  outputs_bucket &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ |  | <code>0-bootstrap</code> |
+| [billing_account](variables-fast.tf#L26) | Billing account id. If billing account is not part of the same org set `is_org_level` to false. | <code title="object&#40;&#123;&#10;  id           &#61; string&#10;  is_org_level &#61; optional&#40;bool, true&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | ✓ |  | <code>0-bootstrap</code> |
+| [prefix](variables-fast.tf#L74) | Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants. | <code>string</code> | ✓ |  | <code>0-bootstrap</code> |
 | [factories_config](variables.tf#L17) | Configuration for YAML-based factories. | <code title="object&#40;&#123;&#10;  folders_data_path  &#61; optional&#40;string, &#34;data&#47;hierarchy&#34;&#41;&#10;  projects_data_path &#61; optional&#40;string, &#34;data&#47;projects&#34;&#41;&#10;  budgets &#61; optional&#40;object&#40;&#123;&#10;    billing_account       &#61; string&#10;    budgets_data_path     &#61; optional&#40;string, &#34;data&#47;budgets&#34;&#41;&#10;    notification_channels &#61; optional&#40;map&#40;any&#41;, &#123;&#125;&#41;&#10;  &#125;&#41;&#41;&#10;  context &#61; optional&#40;object&#40;&#123;&#10;    folder_ids        &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10;    iam_principals    &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10;    tag_values        &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10;    vpc_host_projects &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10;  &#125;&#41;, &#123;&#125;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>&#123;&#125;</code> |  |
-| [folder_ids](variables-fast.tf#L30) | Folders created in the resource management stage. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>1-resman</code> |
-| [groups](variables-fast.tf#L38) | Group names or IAM-format principals to grant organization-level permissions. If just the name is provided, the 'group:' principal and organization domain are interpolated. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>0-bootstrap</code> |
-| [host_project_ids](variables-fast.tf#L47) | Host project for the shared VPC. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>2-networking</code> |
-| [locations](variables-fast.tf#L55) | Optional locations for GCS, BigQuery, and logging buckets created here. | <code title="object&#40;&#123;&#10;  gcs &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>&#123;&#125;</code> | <code>0-bootstrap</code> |
-| [service_accounts](variables-fast.tf#L75) | Automation service accounts in name => email format. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>1-resman</code> |
-| [tag_values](variables-fast.tf#L83) | FAST-managed resource manager tag values. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>1-resman</code> |
+| [folder_ids](variables-fast.tf#L39) | Folders created in the resource management stage. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>1-resman</code> |
+| [groups](variables-fast.tf#L47) | Group names or IAM-format principals to grant organization-level permissions. If just the name is provided, the 'group:' principal and organization domain are interpolated. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>0-bootstrap</code> |
+| [host_project_ids](variables-fast.tf#L56) | Host project for the shared VPC. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>2-networking</code> |
+| [locations](variables-fast.tf#L64) | Optional locations for GCS, BigQuery, and logging buckets created here. | <code title="object&#40;&#123;&#10;  gcs &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>&#123;&#125;</code> | <code>0-bootstrap</code> |
+| [outputs_location](variables.tf#L39) | Enable writing provider, tfvars and CI/CD workflow files to local filesystem. Leave null to disable. | <code>string</code> |  | <code>null</code> |  |
+| [service_accounts](variables-fast.tf#L84) | Automation service accounts in name => email format. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>1-resman</code> |
+| [stage_name](variables.tf#L45) | FAST stage name. Used to separate output files across different factories. | <code>string</code> |  | <code>&#34;2-project-factory&#34;</code> |  |
+| [tag_values](variables-fast.tf#L92) | FAST-managed resource manager tag values. | <code>map&#40;string&#41;</code> |  | <code>&#123;&#125;</code> | <code>1-resman</code> |
 
 ## Outputs
 
 | name | description | sensitive | consumers |
 |---|---|:---:|---|
-| [projects](outputs.tf#L17) | Created projects. |  |  |
-| [service_accounts](outputs.tf#L22) | Created service accounts. |  |  |
+| [projects](outputs.tf#L32) | Created projects. |  |  |
+| [service_accounts](outputs.tf#L46) | Created service accounts. |  |  |
 <!-- END TFDOC -->

fast/stages/2-project-factory/outputs.tf

@@ -14,12 +14,63 @@
  * limitations under the License.
  */
 
+locals {
+  project_outputs = {
+    for k, v in module.projects.projects : k => {
+      bucket = try(
+        v.automation_buckets["state"],
+        v.automation_buckets["tf-state"],
+        null
+      )
+      project_id = v.project_id
+      sa         = try(v.automation_service_accounts["rw"], null)
+      sa_ro      = try(v.automation_service_accounts["ro"], null)
+    } if v.automation_enabled
+  }
+}
+
 output "projects" {
   description = "Created projects."
-  value       = module.projects.projects
+  value = {
+    for k, v in module.projects.projects : k => {
+      id     = v.project_id
+      number = v.number
+      automation = {
+        buckets          = v.automation_buckets
+        service_accounts = v.automation_service_accounts
+      }
+    }
+  }
 }
 
 output "service_accounts" {
   description = "Created service accounts."
-  value       = module.projects.service_accounts
+  value = {
+    for k, v in module.projects.service_accounts : k => {
+      email      = v.email
+      iam_emanil = v.iam_email
+    }
+  }
+}
+
+# generate tfvars file for subsequent stages
+
+resource "local_file" "providers" {
+  for_each = var.outputs_location == null ? {} : {
+    for k, v in local.project_outputs : k => v
+    if v.bucket != null && v.sa != null
+  }
+  file_permission = "0644"
+  filename        = "${try(pathexpand(var.outputs_location), "")}/providers/${var.stage_name}/${each.key}-providers.tf"
+  content         = templatefile("templates/providers.tf.tpl", each.value)
+}
+
+resource "google_storage_bucket_object" "tfvars" {
+  for_each = var.outputs_location == null ? {} : {
+    for k, v in local.project_outputs : k => v
+    if v.bucket != null && v.sa != null
+  }
+  bucket  = var.automation.outputs_bucket
+  name    = "providers/${var.stage_name}/${each.key}-providers.tf"
+  content = templatefile("templates/providers.tf.tpl", each.value)
 }