chore(migrate): migrate otelgrpc pkg interceptor to stats handler(#18258)

[Jack-R-lantern]

This PR will migrate the otelgrpc pkg.
It changes server/clientinterceptor in otelgrpc to NewServer/ClientHandler
I think that otelgrpc should be updated to 0.52.0 or higher in the future, in which case grpc should also be updated to 1.64.

Closes #18258

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[codecov]

Codecov Report

Attention: Patch coverage is 77.77778% with 2 lines in your changes missing coverage. Please review.

Project coverage is 55.76%. Comparing base (683e4e0) to head (b6f81d5).

Files with missing lines	Patch %	Lines
cmpserver/server.go	0.00%	1 Missing ⚠️
pkg/apiclient/apiclient.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #18366      +/-   ##
==========================================
- Coverage   55.79%   55.76%   -0.04%     
==========================================
  Files         342      341       -1     
  Lines       57213    57196      -17     
==========================================
- Hits        31920    31893      -27     
- Misses      22655    22656       +1     
- Partials     2638     2647       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Jack-R-lantern]

[previous CI integratiion tests e2e log]

time="2024-05-28T09:10:36Z" level=error msg="../../dist/argocd app diff test-app-with-secrets --local testdata --server-side-generate --plaintext --server 127.0.0.1:8088 --auth-token *** --insecure failed exit status 20: time="2024-05-28T09:10:36Z" level=fatal msg="rpc error: code = Unimplemented desc = grpc_retry: cannot retry on ClientStreams, set grpc_retry.Disable()"" execID=c6fd7
time="2024-05-28T09:10:36Z" level=fatal msg="../../dist/argocd app diff test-app-with-secrets --local testdata --server-side-generate --plaintext --server 127.0.0.1:8088 --auth-token *** --insecure failed exit status 20: time="2024-05-28T09:10:36Z" level=fatal msg="rpc error: code = Unimplemented desc = grpc_retry: cannot retry on ClientStreams, set grpc_retry.Disable()""

Modified the code based on the logs from the previous E2E test.
I think this was the problem.
It seems that using WithStreamInterceptor without using a chain overwrote an existing Interceptor.

[Jack-R-lantern]

@blakepettersson Can you check on that PR?

[blakepettersson]

This LGTM @Jack-R-lantern

[blakepettersson]

For reference: #17790, #17197 and #17103 were previous attempts

[Jack-R-lantern]

@blakepettersson
Resolved all conflicts.
Please review this PR

[Jack-R-lantern]

@blakepettersson
The corresponding pr has been approved, but when will it be merged?

[crenshaw-dev]

@Jack-R-lantern looks like there's some duplicate work here: #22098

Would y'all mind cross-reviewing and making sure everything looks right?

[Jack-R-lantern]

@crenshaw-dev

Of course.
In the server/application, the gRPC client is created using client streaming, and the important part is that the grpc_retry.Disable() option has been added.
It doesn’t seem to cause any major issues in testing, but could you review this part?
Everything else looks fine.

[crenshaw-dev]

@Jack-R-lantern could you rebase?

[Jack-R-lantern]

@crenshaw-dev
sure

[crenshaw-dev]

Hm. Those test failures look very unrelated to these changes.

[Jack-R-lantern]

I'll check about that part.

[andrii-korotkov-verkada]

What some failed tests have in common is

		And(func() {
			go startCMPServer(t, "./testdata/cmp-gitsshcreds-disable-provide")
			time.Sleep(100 * time.Millisecond)
			t.Setenv("ARGOCD_BINARY_NAME", "argocd")
		}).

I wonder if the cmp server takes more to initialize with the new changes. Can you try bumping the sleep time to 1 second or more and see if the issue still persists, please?