Authentication error after upgrading to 0.23.1

[lbergelson]

We've started seeing an authentication error in our project after we upgraded to 0.23.1, the issue also seems to be present in 0.24.0. Reverting to 0.22.0 solves the issue.

We start seeing the following 404 error when running a spark application that uses NIO to access gcs files:

code:      0
message:   Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
reason:    null
location:  null
retryable: false
com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:339)
	at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
	at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
	at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:91)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
	at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.checkAccess(CloudStorageFileSystemProvider.java:614)
	at java.nio.file.Files.exists(Files.java:2385)
	at htsjdk.samtools.util.IOUtil.assertFileIsReadable(IOUtil.java:346)
	at org.broadinstitute.hellbender.engine.ReadsDataSource.<init>(ReadsDataSource.java:206)
	at org.broadinstitute.hellbender.engine.ReadsDataSource.<init>(ReadsDataSource.java:162)
	at org.broadinstitute.hellbender.engine.ReadsDataSource.<init>(ReadsDataSource.java:118)
	at org.broadinstitute.hellbender.engine.ReadsDataSource.<init>(ReadsDataSource.java:87)
	at org.broadinstitute.hellbender.engine.spark.datasources.ReadsSparkSource.getHeader(ReadsSparkSource.java:182)
	at org.broadinstitute.hellbender.engine.spark.GATKSparkTool.initializeReads(GATKSparkTool.java:390)
	at org.broadinstitute.hellbender.engine.spark.GATKSparkTool.initializeToolInputs(GATKSparkTool.java:370)
	at org.broadinstitute.hellbender.engine.spark.GATKSparkTool.runPipeline(GATKSparkTool.java:360)
	at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:38)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:119)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:176)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:195)
	at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:131)
	at org.broadinstitute.hellbender.Main.mainEntry(Main.java:152)
	at org.broadinstitute.hellbender.Main.main(Main.java:233)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
	at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:137)
	at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:160)
	at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:146)
	at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
	at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
	at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
	at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
	at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
	at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
	at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:337)
	... 32 more
ERROR: (gcloud.dataproc.jobs.submit.spark) Job [cb87810a-0133-42b3-a954-363b62adce39] entered state [ERROR] while waiting for [DONE].

Looking at the dependency updates in this project, it seems like one of the auth libraries updated to version 0.8.0. Could that be the causing the issue?

Is there some new configuration setting we should be using in our gcloud project? Any help would be appreciated.

[neozwu]

@lbergelson from a quick scan of google-auth-client-java, it seems there's some code change regarding compute engine credential in version 0.7.1. Can you help check whether you could repro this issue with auth 0.7.1 ? (I saw you use gradle, so I assume simply declaring auth 0.7.1 as an explicit compile dependency should override auth 0.7.0)

[lbergelson]

@neozwu I tried running with 0.22.0 but forcing the version of auth to 0.7.1, and was unable to reproduce the error that way. I also tried forcing 0.8.0 and wasn't able to reproduce the problem that way either. So either I'm building it wrong or it's not the auth library. From running gradle dependencies it looks like the force is working though.

force 'com.google.auth:google-auth-library-oauth2-http:0.7.1'
force 'com.google.auth:google-auth-library-credentials:0.7.1'

[droazen]

@neozwu I was able to resolve the 404 error that @lbergelson reported by building a custom version of the latest master of google-cloud-java with the following patch applied:

diff --git a/pom.xml b/pom.xml
index 0a77a625b0..e0884bbf2d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -131,10 +131,10 @@
     <api-client.version>1.22.0</api-client.version>
 
     <api-common.version>1.1.0</api-common.version>
-    <gax.version>1.8.1</gax.version>
-    <gax-grpc.version>0.25.1</gax-grpc.version>
+    <gax.version>1.8.0</gax.version>
+    <gax-grpc.version>0.25.0</gax-grpc.version>
     <generatedProto.version>0.1.19</generatedProto.version>
-    <google.auth.version>0.8.0</google.auth.version>
+    <google.auth.version>0.7.0</google.auth.version>
     <grpc.version>1.6.1</grpc.version>
     <guava.version>20.0</guava.version>
     <http-client.version>1.22.0</http-client.version>

So, it's very likely that the culprit is the google-auth-library-java change in 0.7.1 that @neozwu pointed out (https://github.com/google/google-auth-library-java/releases/tag/v0.7.1)

It looks like they moved from using a domain name for the METADATA_SERVER_URL to a hardcoded IP address (see https://github.com/google/google-auth-library-java/pull/110/files) -- I wonder if that is causing this somehow.

[lbergelson]

So it seems like the reason that I wasn't able to reproduce the issue by forcing 0.7.1 was that we're using the shaded jar, so the force statement wasn't actually replacing the dependencies.

[droazen]

Any suggestions on how we could go about dealing with this issue? Does anyone think that the hardcoded IP address added in https://github.com/google/google-auth-library-java/pull/110/files could be the cause?

[lbergelson]

Does anyone have any input on this one? It's important to us that we be able to upgrade.

[neozwu]

@lukecwik Could you comment on whether there could be any potential issues with hardcoded IP address for metadata server ? I saw you suggested an explicit IP vs domain name here.

[lukecwik]

GCE metadata server team specifically recommend using the IP address as a best practice to avoid DNS issues when contacting the metadata server.

Note that all of the Apiary client libraries have been using the fixed IP address for quite some time and to my knowledge have not experienced this issue.

The downside is that if the GCE metadata server team were to ever change the IP address of the metadata server, it would break everyone who hardcoded the address.

[lbergelson]

@neozwu @lukecwik Does anyone have any suggestions about how we can proceed? Is there some information we can provide that would be helpful for debugging this?

[lukecwik]

Since you have a complicated setup, it would be best if you tried executing a simple program which only invokes the google-auth-library-java 0.7.1 from within the environment without adding additional dependencies which would exclude your network/host setup from the failure scenario or shell out and use curl to contact the metadata server from your application and log what you get back using the IP address directly. If a much simpler application/shelling out works means that you have a dependency/configuration of java issue going on.

If you exclude the host/network setup from the failure condition, then comparing the detailed maven dependency tree of the working and broken versions could help. For every library that changed, try doing a strict override on each changed dependency (and only that changed dependency keeping all other transitive dependencies the same) to narrow it down to which one is causing the failure.

[jean-philippe-martin]

I was able to reproduce the same exact error running gatk PrintReadsSpark on a fresh Dataproc cluster with no special configuration applied. This suggests cluster/firewall misconfiguration may not be the problem.

My repro's very short:

package repro_package;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import com.google.cloud.storage.contrib.nio.CloudStorageFileSystem;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;

public class Main {

  final String cluster = "jps-test-cluster-2";

  public static void exists() {
    CloudStorageFileSystem fs = CloudStorageFileSystem.forBucket("jpmartin-testing-project");
    java.nio.file.Path p = fs.getPath("/hellbender-test-inputs/CEUTrio.HiSeq.WGS.b37.ch20.1m-2m.NA12878.bam");
    boolean x = Files.exists(p);
    System.out.println("" + p + " exists: " + x);
  }

  public static Integer runRemotely() {
    exists();
    System.out.println("Hello.");
    return 1;
  }


  public static void main(final String[] args) {
    exists();
    ArrayList<Integer> l = new ArrayList<Integer>();
    l.add(1);
    l.add(2);
    final SparkConf sparkConf = new SparkConf().setAppName("repro_package").setMaster("yarn-client");
    final JavaSparkContext ctx = new JavaSparkContext(sparkConf);
    ctx.parallelize(l, l.size()).map(i -> runRemotely());
    System.out.println("Done.");
  }
}