Merge branch 'master' into cloud_functions_trigger_v2_optionals

Author: wiktorn

blueprints/cloud-operations/network-dashboard/README.md

@@ -16,20 +16,25 @@ Three metric descriptors are created for each monitored resource: usage, limit a
 Clone this repository, then go through the following steps to create resources:
 - Create a terraform.tfvars file with the following content:
   ```tfvars
-  organization_id         = "<YOUR-ORG-ID>"
-  billing_account         = "<YOUR-BILLING-ACCOUNT>"
-  monitoring_project_id   = "project-0"               # Monitoring project where the dashboard will be created and the solution deployed
-  monitored_projects_list = ["project-1", "project2"] # Projects to be monitored by the solution
-  monitored_folders_list  = ["folder_id"]             # Folders to be monitored by the solution
-  v2                      = false                     # Set to true to use V2 Cloud Functions environment
-  ```
+  organization_id = "<YOUR-ORG-ID>"
+  billing_account = "<YOUR-BILLING-ACCOUNT>"
+  monitoring_project_id = "<YOUR-MONITORING-PROJECT>" 
+  # Monitoring project where the dashboard will be created and the solution deployed, a project named "mon-network-dahshboard" will be created if left blank
+  monitored_projects_list = ["project-1", "project2"] 
+  # Projects to be monitored by the solution
+  monitored_folders_list = ["folder_id"] 
+  # Folders to be monitored by the solution
+  prefix = "<YOUR-PREFIX>" 
+  # Monitoring project name prefix, monitoring project name is <YOUR-PREFIX>-network-dashboard, ignored if monitoring_project_id variable is provided
+  cf_version = V1|V2
+  # Set to V2 to use V2 Cloud Functions environment
+    ```
 - `terraform init`
 - `terraform apply`
 
 Note: Org level viewing permission is required for some metrics such as firewall policies.
 
-Once the resources are deployed, go to the following page to see the dashboard: https://console.cloud.google.com/monitoring/dashboards?project=<YOUR-MONITORING-PROJECT>.
-A dashboard called "quotas-utilization" should be created.
+Once the resources are deployed, go to the following page to see the dashboard: https://console.cloud.google.com/monitoring/dashboards?project=<YOUR-MONITORING-PROJECT> a dashboard called "quotas-utilization" should be created.
 
 The Cloud Function runs every 10 minutes by default so you should start getting some data points after a few minutes.
 You can use the metric explorer to view the data points for the different custom metrics created: https://console.cloud.google.com/monitoring/metrics-explorer?project=<YOUR-MONITORING-PROJECT>.
@@ -69,7 +74,6 @@ Note that metrics are created in the cloud-function/metrics.yaml file. You can a
 - The CF assumes custom routes importing/exporting is ON, this impacts static and dynamic routes usage calculation
 - The CF assumes all networks in peering groups have the same global routing and custom routes sharing configuration
 
-
 ## Next steps and ideas
 In a future release, we could support:
 - Google managed VPCs that are peered with PSA (such as Cloud SQL or Memorystore)
@@ -87,13 +91,13 @@ If you are interested in this and/or would like to contribute, please contact le
 |---|---|:---:|:---:|:---:|
 | [billing_account](variables.tf#L17) | The ID of the billing account to associate this project with | <code></code> | ✓ |  |
 | [monitored_projects_list](variables.tf#L36) | ID of the projects to be monitored (where limits and quotas data will be pulled) | <code>list&#40;string&#41;</code> | ✓ |  |
-| [organization_id](variables.tf#L47) | The organization id for the associated services | <code></code> | ✓ |  |
-| [prefix](variables.tf#L51) | Customer name to use as prefix for monitoring project | <code></code> | ✓ |  |
+| [organization_id](variables.tf#L46) | The organization id for the associated services | <code></code> | ✓ |  |
+| [prefix](variables.tf#L50) | Customer name to use as prefix for monitoring project | <code></code> | ✓ |  |
 | [cf_version](variables.tf#L21) | Cloud Function version 2nd Gen or 1st Gen. Possible options: 'V1' or 'V2'.Use CFv2 if your Cloud Function timeouts after 9 minutes. By default it is using CFv1. | <code></code> |  | <code>V1</code> |
 | [monitored_folders_list](variables.tf#L30) | ID of the projects to be monitored (where limits and quotas data will be pulled) | <code>list&#40;string&#41;</code> |  | <code>&#91;&#93;</code> |
 | [monitoring_project_id](variables.tf#L41) | Monitoring project where the dashboard will be created and the solution deployed; a project will be created if set to empty string | <code></code> |  |  |
-| [project_monitoring_services](variables.tf#L55) | Service APIs enabled in the monitoring project if it will be created. | <code></code> |  | <code title="&#91;&#10;  &#34;artifactregistry.googleapis.com&#34;,&#10;  &#34;cloudasset.googleapis.com&#34;,&#10;  &#34;cloudbilling.googleapis.com&#34;,&#10;  &#34;cloudbuild.googleapis.com&#34;,&#10;  &#34;cloudresourcemanager.googleapis.com&#34;,&#10;  &#34;cloudscheduler.googleapis.com&#34;,&#10;  &#34;compute.googleapis.com&#34;,&#10;  &#34;cloudfunctions.googleapis.com&#34;,&#10;  &#34;iam.googleapis.com&#34;,&#10;  &#34;iamcredentials.googleapis.com&#34;,&#10;  &#34;logging.googleapis.com&#34;,&#10;  &#34;monitoring.googleapis.com&#34;,&#10;  &#34;run.googleapis.com&#34;,&#10;  &#34;serviceusage.googleapis.com&#34;&#10;&#93;">&#91;&#8230;&#93;</code> |
-| [region](variables.tf#L75) | Region used to deploy the cloud functions and scheduler | <code></code> |  | <code>europe-west1</code> |
-| [schedule_cron](variables.tf#L80) | Cron format schedule to run the Cloud Function. Default is every 10 minutes. | <code></code> |  | <code>&#42;&#47;10 &#42; &#42; &#42; &#42;</code> |
+| [project_monitoring_services](variables.tf#L54) | Service APIs enabled in the monitoring project if it will be created. | <code></code> |  | <code title="&#91;&#10;  &#34;artifactregistry.googleapis.com&#34;,&#10;  &#34;cloudasset.googleapis.com&#34;,&#10;  &#34;cloudbilling.googleapis.com&#34;,&#10;  &#34;cloudbuild.googleapis.com&#34;,&#10;  &#34;cloudfunctions.googleapis.com&#34;,&#10;  &#34;cloudresourcemanager.googleapis.com&#34;,&#10;  &#34;cloudscheduler.googleapis.com&#34;,&#10;  &#34;compute.googleapis.com&#34;,&#10;  &#34;iam.googleapis.com&#34;,&#10;  &#34;iamcredentials.googleapis.com&#34;,&#10;  &#34;logging.googleapis.com&#34;,&#10;  &#34;monitoring.googleapis.com&#34;,&#10;  &#34;pubsub.googleapis.com&#34;,&#10;  &#34;run.googleapis.com&#34;,&#10;  &#34;servicenetworking.googleapis.com&#34;,&#10;  &#34;serviceusage.googleapis.com&#34;,&#10;  &#34;storage-component.googleapis.com&#34;&#10;&#93;">&#91;&#8230;&#93;</code> |
+| [region](variables.tf#L76) | Region used to deploy the cloud functions and scheduler | <code></code> |  | <code>europe-west1</code> |
+| [schedule_cron](variables.tf#L81) | Cron format schedule to run the Cloud Function. Default is every 10 minutes. | <code></code> |  | <code>&#42;&#47;10 &#42; &#42; &#42; &#42;</code> |
 
 <!-- END TFDOC -->

blueprints/cloud-operations/network-dashboard/main.tf

@@ -30,7 +30,7 @@ locals {
 module "project-monitoring" {
   count           = var.monitoring_project_id == "" ? 1 : 0
   source          = "../../../modules/project"
-  name            = "monitoring"
+  name            = "network-dashboards"
   parent          = "organizations/${var.organization_id}"
   prefix          = var.prefix
   billing_account = var.billing_account

blueprints/cloud-operations/network-dashboard/tests/variables.tf

@@ -36,7 +36,6 @@ variable "project_vm_services" {
     "servicenetworking.googleapis.com",
   ]
 }
-
 variable "region" {
   description = "Region used to deploy subnets"
   default     = "europe-west1"
@@ -45,4 +44,4 @@ variable "region" {
 variable "zone" {
   description = "Zone used to deploy vms"
   default     = "europe-west1-b"
-}
+}

blueprints/cloud-operations/network-dashboard/variables.tf

@@ -43,7 +43,6 @@ variable "monitoring_project_id" {
   default     = ""
 }
 
-
 variable "organization_id" {
   description = "The organization id for the associated services"
 }
@@ -59,19 +58,21 @@ variable "project_monitoring_services" {
     "cloudasset.googleapis.com",
     "cloudbilling.googleapis.com",
     "cloudbuild.googleapis.com",
+    "cloudfunctions.googleapis.com",
     "cloudresourcemanager.googleapis.com",
     "cloudscheduler.googleapis.com",
     "compute.googleapis.com",
-    "cloudfunctions.googleapis.com",
     "iam.googleapis.com",
     "iamcredentials.googleapis.com",
     "logging.googleapis.com",
     "monitoring.googleapis.com",
+    "pubsub.googleapis.com",
     "run.googleapis.com",
-    "serviceusage.googleapis.com"
+    "servicenetworking.googleapis.com",
+    "serviceusage.googleapis.com",
+    "storage-component.googleapis.com"
   ]
 }
-
 variable "region" {
   description = "Region used to deploy the cloud functions and scheduler"
   default     = "europe-west1"

blueprints/networking/filtering-proxy-psc/main.tf

@@ -27,7 +27,8 @@ module "project" {
   services = [
     "dns.googleapis.com",
     "compute.googleapis.com",
-    "logging.googleapis.com"
+    "logging.googleapis.com",
+    "monitoring.googleapis.com"
   ]
 }
 

blueprints/networking/filtering-proxy-psc/squid.conf

@@ -4,6 +4,14 @@ http_port 0.0.0.0:3128 require-proxy-header
 # only proxy, don't cache
 cache deny all
 
+# redirect all logs to /dev/stdout
+logfile_rotate 0
+cache_log stdio:/dev/stdout
+access_log stdio:/dev/stdout
+cache_store_log stdio:/dev/stdout
+
+pid_filename /var/run/squid/squid.pid
+
 acl ssl_ports port 443
 acl safe_ports port 80
 acl safe_ports port 443

modules/cloud-config-container/squid/cloud-config.yaml

@@ -68,10 +68,10 @@ write_files:
       [Service]
       Environment="HOME=/home/squid"
       ExecStartPre=/usr/bin/docker-credential-gcr configure-docker
-      ExecStart=/usr/bin/docker run --rm --name=squid  \
+      ExecStart=/usr/bin/docker run --rm --name=squid \
         --network host \
         -v /etc/squid:/etc/squid \
-        gcr.io/pso-cft-fabric/squid:0.10
+        gcr.io/pso-cft-fabric/squid:20221116
       ExecStop=/usr/bin/docker stop squid
       ExecStopPost=/usr/bin/docker rm squid
 

modules/cloud-config-container/squid/docker/Dockerfile

@@ -17,6 +17,7 @@ FROM debian:buster-slim
 ENV SQUID_VERSION=4.6 \
   SQUID_CACHE_DIR=/var/spool/squid \
   SQUID_LOG_DIR=/var/log/squid \
+  SQUID_PID_DIR=/var/run/squid \
   SQUID_USER=proxy
 
 RUN apt-get update \
@@ -26,5 +27,12 @@ RUN apt-get update \
 COPY entrypoint.sh /sbin/entrypoint.sh
 RUN chmod 755 /sbin/entrypoint.sh
 
+# Create the PID file directory as root, as the non-privileged user squid is not
+# allowed to write in /var/run.
+RUN mkdir -p ${SQUID_PID_DIR} \
+  && chown ${SQUID_USER}:${SQUID_USER} ${SQUID_PID_DIR}
+
+USER ${SQUID_USER}
+
 EXPOSE 3128/tcp
 ENTRYPOINT ["/sbin/entrypoint.sh"]

modules/cloud-config-container/squid/docker/cloudbuild.yaml

@@ -24,7 +24,7 @@ steps:
       - .
 
 substitutions:
-  _IMAGE_VERSION: "20210215"
+  _IMAGE_VERSION: "20221116"
 images:
   - "gcr.io/$PROJECT_ID/squid:${_IMAGE_VERSION}"
   - "gcr.io/$PROJECT_ID/squid:latest"

modules/cloud-config-container/squid/squid.conf

@@ -4,6 +4,14 @@ http_port 0.0.0.0:3128
 # only proxy, don't cache
 cache deny all
 
+# redirect all logs to /dev/stdout
+logfile_rotate 0
+cache_log stdio:/dev/stdout
+access_log stdio:/dev/stdout
+cache_store_log stdio:/dev/stdout
+
+pid_filename /var/run/squid/squid.pid
+
 acl ssl_ports port 443
 acl safe_ports port 80
 acl safe_ports port 443