quota exceeded retries may not be working as expected

[mtagle]

I recently saw this error while testing something locally:

[2016-11-10 13:29:42,733] ERROR Commit of WorkerSinkTask{id=bootstrap-test-03-3} offsets threw an unexpected exception:  (org.apache.kafka.connect.runtime.WorkerSinkTask:180)
com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException: Exception occurred while executing write threads
	at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.flush(BigQuerySinkTask.java:234)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:275)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:155)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException: Failed to write to BigQuery table {datasetId=u_moirat, tableId=pocdb_debezium_apilogger_log_api_log$20161110}
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.flush(BigQuerySinkTask.java:229)
	... 10 more
Caused by: com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException: Failed to write to BigQuery table {datasetId=u_moirat, tableId=pocdb_debezium_apilogger_log_api_log$20161110}
	at com.wepay.kafka.connect.bigquery.write.batch.DynamicBatchWriter.seekingWriteAll(DynamicBatchWriter.java:165)
	at com.wepay.kafka.connect.bigquery.write.batch.DynamicBatchWriter.writeAll(DynamicBatchWriter.java:101)
	at com.wepay.kafka.connect.bigquery.BigQuerySinkTask$TableWriter.call(BigQuerySinkTask.java:155)
	at com.wepay.kafka.connect.bigquery.BigQuerySinkTask$TableWriter.call(BigQuerySinkTask.java:132)
	... 4 more
Caused by: com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: Your table exceeded quota for table.insert or table.update per table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors
	at com.google.cloud.bigquery.spi.DefaultBigQueryRpc.translate(DefaultBigQueryRpc.java:93)
	at com.google.cloud.bigquery.spi.DefaultBigQueryRpc.patch(DefaultBigQueryRpc.java:218)
	at com.google.cloud.bigquery.BigQueryImpl$10.call(BigQueryImpl.java:329)
	at com.google.cloud.bigquery.BigQueryImpl$10.call(BigQueryImpl.java:326)
	at com.google.cloud.RetryHelper.doRetry(RetryHelper.java:179)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:244)
	at com.google.cloud.bigquery.BigQueryImpl.update(BigQueryImpl.java:325)
	at com.wepay.kafka.connect.bigquery.SchemaManager.updateSchema(SchemaManager.java:57)
	at com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.performWriteRequest(AdaptiveBigQueryWriter.java:86)
	at com.wepay.kafka.connect.bigquery.write.row.BigQueryWriter.writeRows(BigQueryWriter.java:158)
	at com.wepay.kafka.connect.bigquery.write.batch.DynamicBatchWriter.seekingWriteAll(DynamicBatchWriter.java:131)
	... 7 more
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "location" : "table.write",
    "locationType" : "other",
    "message" : "Exceeded rate limits: Your table exceeded quota for table.insert or table.update per table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors",
    "reason" : "rateLimitExceeded"
  } ],
  "message" : "Exceeded rate limits: Your table exceeded quota for table.insert or table.update per table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors"
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
	at com.google.cloud.bigquery.spi.DefaultBigQueryRpc.patch(DefaultBigQueryRpc.java:213)
	... 16 more

This stack trace seems to suggest that retries for 403 errors are not working as I would expect.

[mtagle]

This seems to be because the 403 error code doesn't actually propagate to the BigQueryException (at least in this case). This seems like an oversight, so I'll file a bug with the google cloud api folks.

[mtagle]

ps: since it looks like this only affects errors when trying to update a table, I don't consider it so urgent that we should change our code to accommodate.

[criccomini]

Agreed. Nice catch. :)

[mtagle]

bug report to google cloud: googleapis/google-cloud-java#1391

[mtagle]

fixed in #70