Auth problems App Engine J8/Std --> Stackdriver Montoring API

[DazWilkin]

Problem

Cloud Client Libraries (including Cloud Monitoring API v3) support Application Default Credentials and should use the App Engine service account to authenticate against other services. This does not appear to work as intended|documented:

com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED

A colleague provided a hack that obtains an access token using the App Engine service account and injects this into the credentials provided to the Cloud Monitoring API. This works but is unwieldy (see below) and should be unnecessary.

Repro

-- Maven generate an App Engine Standard J8 app
-- Mash-up w/ Google provided Custom Metric sample
-- Observe UNAUTHENTICATED problems auth'ing with MetricServiceClient.create()

Solution

Caveat: I do not understand why this code works... it does

-- Revise MetricServiceClient.create() with:

List<String> scopes = Arrays.asList("https://www.googleapis.com/auth/monitoring");

AppIdentityService appIdentityService = AppIdentityServiceFactory
    .getAppIdentityService();

String access_token = appIdentityService
    .getAccessToken(scopes)
    .getAccessToken();

AppEngineCredentials credentials = AppEngineCredentials
    .newBuilder()
    .setAppIdentityService(appIdentityService)
    .setScopes(scopes)
    .build();

CredentialsProvider credentialsProvider = FixedCredentialsProvider.create(credentials);

MetricServiceSettings metricServiceSettings =
    MetricServiceSettings.newBuilder()
        .setCredentialsProvider(credentialsProvider)
        .build();
MetricServiceClient metricServiceClient = MetricServiceClient
    .create(metricServiceSettings);        

Here:
https://gist.github.com/DazWilkin/05b1a2ed702e78019e20e862df274129#file-testservlet-java-L52-L74

Thanks @salrashid123 for providing the solution.

[neozwu]

Hi @DazWilkin on App Engine J8 std, the underlying credential is actually ComputeEngine Credential, which is obtained through http call to metadata (code). I suspect something is wrong with obtaining ComputeEngine credential. Can you provide a sample app, so we can investigate ?

[DazWilkin]

@neozwu -- I included a Gist reference hoping that would help. If not, please let me know what you need and I'll provide. Heading out on vacation tomorrow so will do this today if I can.

[anthmgoogle]

Treating as question for now. We should convert to bug if we determine that something here is not working as intended.

[brugz]

Hi folks,

For what it's worth, I'm tracking a similar issue, and I think the code neozwu provides is making a call to to tryGetComputeCredentials() 1 - which may return Null 2 on GAE Java8

I'm following up with one of our teams to investigate.

[pongad]

@DazWilkin @brugz I am unable to reproduce this. Code and mvn dependency:list pasted below.

Since this issue is quite old, I suspect that the fix might have landed between then and now. So, I'll close the issue for the time being. Please reopen if you're still seeing the problem and we can help investigate more.

org.threeten:threetenbp:jar:1.3.3:compile
io.grpc:grpc-core:jar:1.9.0:compile
com.google.api:gax:jar:1.17.0:compile
com.google.guava:guava:jar:23.4-android:compile
com.google.http-client:google-http-client-jackson:jar:1.23.0:compile
com.google.cloud:google-cloud-core:jar:1.16.0:compile
org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
io.grpc:grpc-auth:jar:1.9.0:compile
io.netty:netty-tcnative-boringssl-static:jar:2.0.7.Final:compile
com.google.auto.value:auto-value:jar:1.2:compile
io.grpc:grpc-protobuf:jar:1.9.0:compile
com.fasterxml.jackson.core:jackson-core:jar:2.1.3:compile
com.google.api:gax-grpc:jar:1.17.0:compile
net.bytebuddy:byte-buddy:jar:1.7.9:test
io.opencensus:opencensus-contrib-grpc-metrics:jar:0.10.0:compile
com.google.api-client:google-api-client:jar:1.23.0:compile
com.google.appengine:appengine-api-stubs:jar:1.9.60:test
org.objenesis:objenesis:jar:2.6:test
com.google.protobuf:protobuf-java-util:jar:3.5.1:compile
io.netty:netty-transport:jar:4.1.17.Final:compile
com.google.http-client:google-http-client-appengine:jar:1.23.0:compile
joda-time:joda-time:jar:2.9.2:compile
com.google.api.grpc:proto-google-common-protos:jar:1.0.5:compile
com.google.cloud:google-cloud-core-http:jar:1.16.0:compile
com.google.truth:truth:jar:0.39:test
io.netty:netty-common:jar:4.1.17.Final:compile
commons-logging:commons-logging:jar:1.1.1:compile
io.grpc:grpc-netty:jar:1.9.0:compile
org.hamcrest:hamcrest-core:jar:1.3:test
com.google.code.findbugs:jsr305:jar:3.0.1:compile
com.google.guava:guava-jdk5:jar:17.0:compile
io.netty:netty-resolver:jar:4.1.17.Final:compile
io.netty:netty-handler-proxy:jar:4.1.17.Final:compile
com.google.http-client:google-http-client:jar:1.23.0:compile
io.grpc:grpc-stub:jar:1.9.0:compile
io.netty:netty-codec-socks:jar:4.1.17.Final:compile
org.apache.httpcomponents:httpclient:jar:4.0.1:compile
com.google.apis:google-api-services-bigquery:jar:v2-rev372-1.23.0:compile
com.google.http-client:google-http-client-jackson2:jar:1.23.0:compile
com.google.auth:google-auth-library-credentials:jar:0.9.0:compile
com.google.oauth-client:google-oauth-client:jar:1.23.0:compile
com.google.api:api-common:jar:1.2.0:compile
com.google.api:gax-httpjson:jar:0.34.0:compile
io.opencensus:opencensus-api:jar:0.10.0:compile
com.google.protobuf:protobuf-java:jar:3.5.1:compile
com.google.cloud:google-cloud-core-grpc:jar:1.16.0:compile
io.netty:netty-codec:jar:4.1.17.Final:compile
com.google.api.grpc:proto-google-iam-v1:jar:0.1.29:compile
commons-codec:commons-codec:jar:1.3:compile
com.google.j2objc:j2objc-annotations:jar:1.1:compile
com.google.instrumentation:instrumentation-api:jar:0.4.3:compile
com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
io.netty:netty-buffer:jar:4.1.17.Final:compile
com.google.cloud:google-cloud-bigquery:jar:0.34.0-beta:compile
com.google.code.gson:gson:jar:2.7:compile
io.netty:netty-codec-http:jar:4.1.17.Final:compile
javax.servlet:javax.servlet-api:jar:3.1.0:provided
com.google.cloud:google-cloud-monitoring:jar:0.34.0-beta:compile
com.google.appengine:appengine-tools-sdk:jar:1.9.60:test
junit:junit:jar:4.12:test
io.grpc:grpc-context:jar:1.9.0:compile
io.netty:netty-handler:jar:4.1.17.Final:compile
org.apache.httpcomponents:httpcore:jar:4.0.1:compile
com.google.appengine:appengine-testing:jar:1.9.60:test
org.mockito:mockito-core:jar:2.15.0:test
net.bytebuddy:byte-buddy-agent:jar:1.7.9:test
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.60:compile
org.codehaus.jackson:jackson-core-asl:jar:1.9.11:compile
com.google.api.grpc:proto-google-cloud-monitoring-v3:jar:0.1.29:compile
io.netty:netty-codec-http2:jar:4.1.17.Final:compile
io.grpc:grpc-protobuf-lite:jar:1.9.0:compile
com.google.auth:google-auth-library-oauth2-http:jar:0.9.0:compile

public void doGet(HttpServletRequest req, HttpServletResponse response) throws IOException {
    try {
      try (MetricServiceClient client = MetricServiceClient.create()) {
        // Prepares an individual data point
        TimeInterval interval =
            TimeInterval.newBuilder()
                .setEndTime(Timestamps.fromMillis(System.currentTimeMillis()))
                .build();
        TypedValue value = TypedValue.newBuilder().setDoubleValue(123.45).build();
        Point point = Point.newBuilder().setInterval(interval).setValue(value).build();

        List<Point> pointList = new ArrayList<>();
        pointList.add(point);

        ProjectName name = ProjectName.create(PROJECT_ID);

        // Prepares the metric descriptor
        Map<String, String> metricLabels = new HashMap<String, String>();
        metricLabels.put("store_id", "Pittsburg");
        Metric metric =
            Metric.newBuilder()
                .setType("custom.googleapis.com/stores/daily_sales")
                .putAllLabels(metricLabels)
                .build();

        // Prepares the monitored resource descriptor
        Map<String, String> resourceLabels = new HashMap<String, String>();
        resourceLabels.put("project_id", PROJECT_ID);
        MonitoredResource resource =
            MonitoredResource.newBuilder().setType("global").putAllLabels(resourceLabels).build();

        // Prepares the time series request
        TimeSeries timeSeries =
            TimeSeries.newBuilder()
                .setMetric(metric)
                .setResource(resource)
                .addAllPoints(pointList)
                .build();
        List<TimeSeries> timeSeriesList = new ArrayList<>();
        timeSeriesList.add(timeSeries);

        CreateTimeSeriesRequest request =
            CreateTimeSeriesRequest.newBuilder()
                .setNameWithProjectName(name)
                .addAllTimeSeries(timeSeriesList)
                .build();

        // Writes time series data
        client.createTimeSeries(request);

        response.setContentType("text/plain");
        response.getWriter().println("Done writing time series data");
      }
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }