[EPIC] Improve the supportability of CAPG (LFX Spring 2023 Mentorship)

[richardcase]

/kind feature
/help

Describe the solution you'd like

We would like to improve the supportability of CAPG. We would like to make it easier for operations/support users to investigate issues with CAPG.

There will be a number of areas of improvement:

• Add tracing using something like OpenTelemetry. This will enable us to see how many and how long reconcile loops, GCP API calls etc take, which will be very helpful for issue investigation.
• Add a pprof endpoint that can be enabled

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Items to do (not exhaustive yet)

• Implement OTEL in CAPG
• Docs
• e2e test to be added
• otel backend for local dev
• otel backend for e2e

[k8s-ci-robot]

@richardcase:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/kind feature
/help

Describe the solution you'd like

We would like to improve the supportability of CAPG. We would like to make it easier for operations/support users to investigate issues with CAPG.

There will be a number of areas of improvement:

• Add tracing using something like OpenTelemetry. This will enable us to see how many and how long reconcile loops, AWS API calls etc take which will be very helpful for issue investigation.
• Add a pprof endpoint that can be enabled

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[richardcase]

This will be used for the LFX mentorship Spring 2023 program. Until that starts:

/assign

[MdSahil-oss]

@richardcase This feature looks interesting to me, I would like to work on this feature as an lfx mentee, can you give me brief idea about the prerequisites to learn ?

[hackeramitkumar]

@richardcase it's an interesting feature, I would like to contribute to this feature in the spring mentorship, can you give me an idea about the prerequisites?

[richardcase]

Thanks @MdSahil-oss and @hackeramitkumar 👍

[richardcase]

Some information for anyone interested in this.

The LFX Spring Mentorship program will open for applications between 1st - 14th February. The applications are then reviewed & decided upon between 15 - 28th February. And then the mentorship program begins 1st March and ends May 26.

You will not need access to your own GCP account, as you will be added to one you can use for this issue during the mentorship period.

For prerequisites:

• Golang
• Kubernetes

Everything else (like Cluster API, tracing, profiling) will be learnt as part of the mentorship.

[richardcase]

When it comes to the application, the more information you can supply as to why this interests you and what you expect to gain from mentorship will help the reviewers.

If you have any questions, feel free to ask here or ping me or @cpanato in Slack.

[hackeramitkumar]

@richardcase thank you very much.

[dharmicksai]

Hello all 👋
I am Dharmick a final-year undergrad student from India,

This proposal looks really exciting 🔥💥
Looking forward to contributing to LFX this spring

[Chinwendu20]

Ohh interesting currently contributing to open telemetry now. Looking forward to contributing to this essential tool that I use as a DevOps engineer

[lucky29-git]

Hey @richardcase and @cpanato , I'm Chandrashekhar from India . I started my journey of opensource with Hacktoberfest 2022 and it excites me a lot . I'm really interested in this project and would like to contribute in it .
My Techstack's are - Java , Go , Docker , Kubernetes
and currently learning other devops chapters .