Datastore: Intermittent Backend Errors since wednesday (22:00 GMT) us-central

[simon-temple]

We suddenly started receiving these errors in the us-central region:

Caused by: com.google.api.services.datastore.client.DatastoreException: Backend Error
at com.google.api.services.datastore.client.RemoteRpc.makeException(RemoteRpc.java:115)
at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:81)
at com.google.api.services.datastore.client.BaseDatastoreFactory$RemoteRpc.call(BaseDatastoreFactory.java:41)
at com.google.api.services.datastore.client.Datastore.runQuery(Datastore.java:109)
at com.google.gcloud.spi.DefaultDatastoreRpc.runQuery(DefaultDatastoreRpc.java:163)
... 58 more
Caused by: com.google.api.client.http.HttpResponseException: 503 Service Unavailable
Backend Error
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1061)
at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:78)
... 61 more"

Some calls succeed others fail. I can't see a gcloud API usage pattern that causes it... it just randomly happens!

I asked Google to check the logs and they stated the failures were due to the use of an invalid project-id.
For example our project id is "production-1103" and they claim to see in the logs "s~production-1103"

We're using gcloud-java 0.1.4 and don't experience these problems in the EU region. Could some external service change influence the project-id corruption? Has anyone else seen this sort of thing in the past?

[mziccard]

This is a strange error indeed. How do you pass the project id to the datastore options? It might be worth trying to pass it using the projectId(...) setter to see if the errors occur again.
I'll try to set up and run a test application to see if I manage to reproduce the issue.

Not related to this issue but might be worth giving a try: starting with version 0.1.7 (release notes here) we use Cloud Datastore v1beta3.

[simon-temple]

Usually we don't pass the project id to the datastore options. It finds them using the metatdata request to http://metadata/computeMetadata/v1/project/project-id. I did try setting it using the GCLOUD_PROJECT environment value but this did not have any effect - the 503s continued.

Like I said, it works without error maybe two in every ten calls so I believe the value is correct after initialisation. I don't think gcloud-java re-evaluates the project-id on each call so it has to be something deeper.

I'm currently testing with version 0.2.0 and will deploy that shortly. Maybe the new API will avoid my current issue...

[mziccard]

I did try setting it using the GCLOUD_PROJECT environment value but this did not have any effect - the 503s continued.

Nice that you tried this. I suspected an error in getting/parsing the instance metadata but we ruled that out.

Like I said, it works without error maybe two in every ten calls so I believe the value is correct after initialization.

You mean that 8 out of 10 requests fail with 503? This is huge...

I don't think gcloud-java re-evaluates the project-id on each call so it has to be something deeper.

Yes, once the project id is set we never change it.

I'm currently testing with version 0.2.0 and will deploy that shortly. Maybe the new API will avoid my current issue...

Please keep us posted. Also, have you experienced this issue in several zones all in the us-central region? Could you please tell me one zone where you certainly had this issue so that I have better chances of reproducing?

[simon-temple]

This is only happening in the us-central region

I have now switched to using 0.2.0 and I no longer get 503's but I do get a number of these:

exception_message: "The project properties does not exist."
stacktrace:
"com.google.cloud.datastore.DatastoreException: The project properties does not exist.
at com.google.cloud.datastore.spi.DefaultDatastoreRpc.translate(DefaultDatastoreRpc.java:102)
at com.google.cloud.datastore.spi.DefaultDatastoreRpc.commit(DefaultDatastoreRpc.java:130)
at com.google.cloud.datastore.DatastoreImpl$4.call(DatastoreImpl.java:364)
at com.google.cloud.datastore.DatastoreImpl$4.call(DatastoreImpl.java:361)
at com.google.cloud.RetryHelper.doRetry(RetryHelper.java:181)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:247)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:237)
at com.google.cloud.datastore.DatastoreImpl.commit(DatastoreImpl.java:360)
at com.google.cloud.datastore.TransactionImpl.commit(TransactionImpl.java:99)
at com.amalto.b2auth.core.persist.gce.entities.core.EntityMapper.save(EntityMapper.java:128)
Caused by: com.google.datastore.v1beta3.client.DatastoreException: The project properties does not exist., code=INVALID_ARGUMENT
at com.google.datastore.v1beta3.client.RemoteRpc.makeException(RemoteRpc.java:126)
at com.google.datastore.v1beta3.client.RemoteRpc.makeException(RemoteRpc.java:169)
at com.google.datastore.v1beta3.client.RemoteRpc.call(RemoteRpc.java:89)
at com.google.datastore.v1beta3.client.Datastore.commit(Datastore.java:84)
at com.google.cloud.datastore.spi.DefaultDatastoreRpc.commit(DefaultDatastoreRpc.java:128)
... 55 more"

Any ideas?

[simon-temple]

The new 0.2.0 issue "The project properties does not exist." happens in US and EU regions...so it's not the same issue. This one is new!

My save code looks like this:

    Entity e = eb.build();
    Transaction tx = datastore.newTransaction();
    try
    {
        tx.delete( k );
        tx.add( e );
        tx.commit();
    }
    finally
    {
        if ( tx.active() ) {
            tx.rollback();
        }
    }

tx.commit() fails.

[mziccard]

Could you share the code that you are using to create entity and key? Or some toy code that produces the error?

I tried to run a similar transaction in a GCE instance, zone us-central1-a, and it works just fine.

[simon-temple]

I'll work on some toy code now using the latest local datastore emulator. I suspect it may be something to do with an entity kind of User with an embedded entity kind of Properties.

[simon-temple]

Has IncompleteKey changed? I have been passing it the name of the entity attribute in User: "properties" and next the kind of "UserProperties"

Now I see the first argument is called projectid?

            IncompleteKey propsKey = IncompleteKey.builder( key, ek.kind() ).build();

            FullEntity.Builder< IncompleteKey > b = FullEntity.builder( propsKey );
            for ( Map.Entry< Object, Object > ent : ( ( Properties ) o ).entrySet() ) {
                b.set( ent.getKey().toString(), ent.getValue().toString() );
            }

            eb.set( key, b.build() );

propsKey.toString() is:

partition_id {
project_id: "properties"
}
path {
kind: "UserProperties"
}

This is the basis of the error message I receive.

[simon-temple]

This may be my solution, but why did it work in previous versions?

    IncompleteKey propsKey = IncompleteKey.builder( datastore.options().projectId(), ek.kind() ).build();

[mziccard]

I am confused. IncompleteKey.builder(String, String) did not change from version 0.14.0 (code here) and I believe the first argument is called projectId since version 0.9.0.

Assuming that you mean key to be a the "properties" string, what you are attempting to do with:

IncompleteKey propsKey = IncompleteKey.builder( key, ek.kind() ).build();

is creating an entity with kind ek.kind() in the project properties, hence the complaint. In fact "properties" is the name you want to give the property of kind ek.kind() in what I assume being an entity of kind User, as you prove with the line:

eb.set( key, b.build() );

Your solution:

IncompleteKey propsKey = IncompleteKey.builder( datastore.options().projectId(), ek.kind() ).build();

is indeed correct. I doubt this code worked with previous versions of gcloud-java or datastore, unless the "Properties" entity was created in the properties dataset/project.

[simon-temple]

Sadly it did used to work! Or at least I was getting away with it...

Here is a properties embedded attribute from our development-1104 datastore, as seen in the developer console:

{ "key": { "partitionId": { "projectId": "properties" }, "path": [ { "kind": "UserProperties" } ] }, "properties": { "webSite": { "stringValue": "www.amalto.com" }, "city": { "stringValue": "Houston" }, "title": { "stringValue": "Dev" }, "street": { "stringValue": "4545 Post Oak Place Dr. - suite 350" }, "salutation": { "stringValue": "mr" }, "state": { "stringValue": "TX" }, "phone": { "stringValue": "" }, "companyName": { "stringValue": "Amalto Technologies" }, "zip": { "stringValue": "77027" }, "country": { "stringValue": "United Kingdom" }, "mobile": { "stringValue": "123" } } }

Note the projectId.

Maybe this is now detected in V1beta3 and not in earlier versions... so not a gcloud-java issue?

[mziccard]

Oh yes this in fact falls in the case

unless the "Properties" entity was created in the properties dataset/project.

In v1beta2 entities where created inside datasets, now the notion of dataset has been replaced with the notion of project which has a wider scope and is subject to more constraints (you must own the project).
I wouldn't be surprised if this was the cause of your recent Backend Error exceptions. BTW, do you still experience them with gcloud-java 0.2.0?

[simon-temple]

Okay, I think I understand my misuse of the API and how it worked in v1beta2
This is a bug I uncovered moving to 0.2.0. Thanks for helping me identify this.

I can confirm that using 0.2.0 in both EU and US regions no longer produces 503s

The final thing I don't quite get is why using 0.1.4 in our production project (us-central) caused "Backend Errors" yet when in our development project (eu) it did not. And how a stable production system stopped working when we continued to use the v1beta2 API via 0.1.4

This has highlighted the fact that Google Datastore API is still Beta and can not yet be relied upon for production environments.

[aozarov]

/cc @eddavisson

[eddavisson]

We believe we have identified the root cause of the intermittent errors you were seeing in the v1beta2 API and expect to have those resolved by later today.