The auth.default() does not pick up the correct user project when running on Vertex Pipelines or CustomJobs

[Ark-kun]

Usually the Vertex SDK gets the project ID automatically (by calling google.auth.default()). This works when running on GKE, GCE, Kubeflow Pipelines etc.

However when running on Google Cloud Vertex Pipelines or Vertex Training CustomJobs, the detected project is not the user project and is not usable.

This leads to failure when trying to create any resource in the project:

google.api_core.exceptions.PermissionDenied: 403 Permission 'aiplatform.models.upload' denied on resource '//aiplatform.googleapis.com/projects/gbd40bc90c7804989-tp/locations/us-central1' (or it may not exist).

Here gbd40bc90c7804989-tp is NOT the correct user project.

Fortunately there is a way to get project number from the Vertex environment. There is also a way to get project ID from the project number.

Inferring project number

project_number = os.environ.get("CLOUD_ML_PROJECT_ID")

Getting project ID:

    if not project:
        project_number = os.environ.get("CLOUD_ML_PROJECT_ID")
        if project_number:
            print(f"Inferred project number: {project_number}")
            project = project_number
            # To improve the naming we try to convert the project number into the user project ID.
            try:
                from googleapiclient import discovery

                cloud_resource_manager_service = discovery.build(
                    "cloudresourcemanager", "v3"
                )
                project_id = (
                    cloud_resource_manager_service.projects()
                    .get(name=f"projects/{project_number}")
                    .execute()["projectId"]
                )
                if project_id:
                    print(f"Inferred project ID: {project_id}")
                    project = project_id
            except Exception as e:
                print(e)

Environment details

• OS:
• Python version: 3.9
• pip version: 21.1.1
• google-auth version: 2.3.3

Steps to reproduce

1. google.auth.default()

See: googleapis/python-aiplatform#852