RetryMiddleware uses CredentialsWrapper scope when invoked

[pkruithof]

The RetryMiddleware triggers a fatal error in some conditions: Call to undefined method Google\ApiCore\CredentialsWrapper::getRetryFunction()

The issue is caused when the middleware is invoked, seemingly with the scope of the CredentialsWrapper, causing the getRetryFunction() method not to be found in the class. I've tried to extract this portion of the client trait into a reproducible script, but that does not trigger the error unfortunately.

Environment details

• OS: Linux (on Docker)
• PHP version: 8.3.7
• Package name and version: google/gax

Steps to reproduce

I cannot reproduce it with a short script, however we occasionally have issues popping up in Sentry about this. I hope this information provides more insights into why this is happening.

[aa-dragosmocrii]

I am running into the same issue. Were you able to fix this?

[pkruithof]

Unfortunately not, hopefully someone at Google notices this 🤞

[ottaz]

Also having this issue. Does anyone have a workaround?

EDIT: 2024-12-10-0946
I seem to only have this problem when,

• using GRPC
• using Topic::publishBatch
• passing retrySettings in the options parameter to publishBatch

If I use Topic::publish with no additional parameters, it does not happen.

[bshaffer]

Could someone provide us with a repro case so we can look into fixing the issue?

I've tried to reproduce it with the following to match @ottaz's instructions, with no luck:

    $pubsub = new PubSubClient([
        'projectId' => $projectId,
    ]);
    $topic = $pubsub->topic($topicName);
    $messages = [['data' => 'test message 1'], ['data' => 'test message 2']];
    $retrySettings = \Google\ApiCore\RetrySettings::constructDefault();
    $topic->publishBatch($messages, [
        'retrySettings' => $retrySettings,
    ]);

edit: it would also help if you could provide the version of the PubSub and GAX libraries you're using (@pkruithof @aa-dragosmocrii @ottaz)

[aa-dragosmocrii]

@bshaffer I haven't noticed this issue occurring in the last 5 months. We've performed multiple major upgrades for our packages, so it's possible that this was caused by some older version we were using. I will keep an eye on our logs though, since I remember this issue was infrequent

[pkruithof]

It happens very infrequently, the last occurrence I found was 2 months ago. The package versions at that time (I'll just list all of the Google related packages, for good measure):

google/auth v1.46.0
google/cloud-core v1.62.0
google/cloud-pubsub v2.10.1
google/cloud-secret-manager v2.0.2
google/cloud-storage v1.46.0
google/common-protos 4.10.0
google/gax v1.36.0
google/grpc-gcp v0.4.0
google/longrunning 0.4.7
google/protobuf v4.29.3
grpc/grpc 1.57.0

The issue is very hard to reproduce, as it happens so infrequently, and Kubernetes cluster just restarts the workload, at which point the message will be retried and acknowledged. So I suspect it's more of a race condition, or other flaky type of issue.