Skip to content

Dynamic selection of log_group_name and log_stream_name in aws cloudwatch logs exporter #31382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sorobon opened this issue Feb 22, 2024 · 20 comments · Fixed by #37297
Closed
Labels
enhancement New feature or request exporter/awscloudwatchlogs awscloudwatchlogs exporter waiting-for-code-owners

Comments

@sorobon
Copy link

sorobon commented Feb 22, 2024

Component(s)

exporter/awscloudwatchlogs

Is your feature request related to a problem? Please describe.

Duplicate with #14554
We are running the opentelemetry collector to send logs from our edge systems to a central opentelemetry gateway service. From the gateway we'd like to write logs to cloudwatch logs in different log groups and streams based on log resources.

Describe the solution you'd like

We'd like to be able to set the log_group_name and log_stream_name dynamically based on fields from the log event:

awscloudwatchlogs:
    log_group_name: "{log_file_name}"
    log_stream_name: "{resource.hostname}"

Describe alternatives you've considered

No response

Additional context

No response

@sorobon sorobon added enhancement New feature or request needs triage New item requiring triage labels Feb 22, 2024
@github-actions github-actions bot added the exporter/awscloudwatchlogs awscloudwatchlogs exporter label Feb 22, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Apr 23, 2024
@sorobon
Copy link
Author

sorobon commented Apr 25, 2024

Any update?

@github-actions github-actions bot removed the Stale label Apr 26, 2024
@abhishekmahajan0709222
Copy link

Any update, we are also looking for same feature

@bpalermo
Copy link

Duplicate #30773?

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@rgoltz
Copy link

rgoltz commented Mar 24, 2025

I love the idea to define log_group_name and log_stream_name dynamically. This update will also reset the stale flag here in this issue.

@github-actions github-actions bot removed the Stale label Mar 24, 2025
@piotrkolanowski
Copy link

piotrkolanowski commented Mar 24, 2025

I also like the idea of controlling the name of the group and stream dynamically

@itsmesuniljacob
Copy link

Any update on this, I think this is a needed functionality

andrzej-stencel added a commit that referenced this issue Apr 17, 2025
…roup and logstream (#37297)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Add dynamic log_group_name and log_group_stream naming, based on
[awsemfexporter](https://github.com/LucianoGiannotti/opentelemetry-collector-contrib/tree/main/exporter/awsemfexporter)
<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
[#31382](#31382)

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Similar testing as on awsemfexporter, testing that names are replaced
properly before pushing logs to CW.


<!--Please delete paragraphs that you did not use before submitting.-->

---------

Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Matthieu MOREL <[email protected]>
Signed-off-by: Jared Tan <[email protected]>
Signed-off-by: Michael Shen <[email protected]>
Signed-off-by: Dani Louca <[email protected]>
Signed-off-by: Célian Garcia <[email protected]>
Signed-off-by: ChrsMark <[email protected]>
Signed-off-by: Marc Lopez Rubio <[email protected]>
Signed-off-by: Murphy Chen <[email protected]>
Co-authored-by: Antoine Toulme <[email protected]>
Co-authored-by: Sam DeHaan <[email protected]>
Co-authored-by: Andrzej Stencel <[email protected]>
Co-authored-by: Irina <[email protected]>
Co-authored-by: Curtis Robert <[email protected]>
Co-authored-by: Constança Manteigas <[email protected]>
Co-authored-by: Paschalis Tsilias <[email protected]>
Co-authored-by: Evan Bradley <[email protected]>
Co-authored-by: Evan Bradley <[email protected]>
Co-authored-by: Matthieu MOREL <[email protected]>
Co-authored-by: Adriel Perkins <[email protected]>
Co-authored-by: Sean Marciniak <[email protected]>
Co-authored-by: Nikos Angelopoulos <[email protected]>
Co-authored-by: Antoine Toulme <[email protected]>
Co-authored-by: Alex Boten <[email protected]>
Co-authored-by: mcube8 <[email protected]>
Co-authored-by: Will Li <[email protected]>
Co-authored-by: Luke Gao <[email protected]>
Co-authored-by: Andrew Wilkins <[email protected]>
Co-authored-by: Jared Tan <[email protected]>
Co-authored-by: Alex Van Boxel <[email protected]>
Co-authored-by: Michael Shen <[email protected]>
Co-authored-by: Roger Coll <[email protected]>
Co-authored-by: Braydon Kains <[email protected]>
Co-authored-by: Dani Louca <[email protected]>
Co-authored-by: wojtekzyla <[email protected]>
Co-authored-by: Célian GARCIA <[email protected]>
Co-authored-by: Murphy Chen <[email protected]>
Co-authored-by: Anthony Yeh <[email protected]>
Co-authored-by: Christos Markou <[email protected]>
Co-authored-by: Yang Song <[email protected]>
Co-authored-by: Vihas Makwana <[email protected]>
Co-authored-by: Daniel Jaglowski <[email protected]>
Co-authored-by: Khurshidali Shiekh <[email protected]>
Co-authored-by: Cyril Cressent <[email protected]>
Co-authored-by: Marc Lopez Rubio <[email protected]>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <[email protected]>
Co-authored-by: Stuart Buckingham <[email protected]>
Co-authored-by: Anthony Mirabella <[email protected]>
Co-authored-by: Jorge Creixell <[email protected]>
Co-authored-by: Robert Lankford <[email protected]>
Co-authored-by: Paulo Janotti <[email protected]>
Co-authored-by: Ludovic Fernandez <[email protected]>
Co-authored-by: Vishal Raj <[email protected]>
Co-authored-by: Gregor Jasny <[email protected]>
Co-authored-by: Pablo Baeyens <[email protected]>
andrzej-stencel added a commit to andrzej-stencel/opentelemetry-collector-contrib that referenced this issue Apr 17, 2025
…to emeritus codeowners

@boostchicken and @rapphil haven't been responding to issues and pull requests for a couple months now.
See for example open-telemetry#31382, open-telemetry#37297.
@boostchicken, @rapphil please respond here or in CNCF Slack #otel-collector-dev if you want to keep being codeowners.

Also marks the exporter as seeking new codeowners.
akshays-19 pushed a commit to akshays-19/opentelemetry-collector-contrib that referenced this issue Apr 23, 2025
…roup and logstream (open-telemetry#37297)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Add dynamic log_group_name and log_group_stream naming, based on
[awsemfexporter](https://github.com/LucianoGiannotti/opentelemetry-collector-contrib/tree/main/exporter/awsemfexporter)
<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
[open-telemetry#31382](open-telemetry#31382)

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Similar testing as on awsemfexporter, testing that names are replaced
properly before pushing logs to CW.


<!--Please delete paragraphs that you did not use before submitting.-->

---------

Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Matthieu MOREL <[email protected]>
Signed-off-by: Jared Tan <[email protected]>
Signed-off-by: Michael Shen <[email protected]>
Signed-off-by: Dani Louca <[email protected]>
Signed-off-by: Célian Garcia <[email protected]>
Signed-off-by: ChrsMark <[email protected]>
Signed-off-by: Marc Lopez Rubio <[email protected]>
Signed-off-by: Murphy Chen <[email protected]>
Co-authored-by: Antoine Toulme <[email protected]>
Co-authored-by: Sam DeHaan <[email protected]>
Co-authored-by: Andrzej Stencel <[email protected]>
Co-authored-by: Irina <[email protected]>
Co-authored-by: Curtis Robert <[email protected]>
Co-authored-by: Constança Manteigas <[email protected]>
Co-authored-by: Paschalis Tsilias <[email protected]>
Co-authored-by: Evan Bradley <[email protected]>
Co-authored-by: Evan Bradley <[email protected]>
Co-authored-by: Matthieu MOREL <[email protected]>
Co-authored-by: Adriel Perkins <[email protected]>
Co-authored-by: Sean Marciniak <[email protected]>
Co-authored-by: Nikos Angelopoulos <[email protected]>
Co-authored-by: Antoine Toulme <[email protected]>
Co-authored-by: Alex Boten <[email protected]>
Co-authored-by: mcube8 <[email protected]>
Co-authored-by: Will Li <[email protected]>
Co-authored-by: Luke Gao <[email protected]>
Co-authored-by: Andrew Wilkins <[email protected]>
Co-authored-by: Jared Tan <[email protected]>
Co-authored-by: Alex Van Boxel <[email protected]>
Co-authored-by: Michael Shen <[email protected]>
Co-authored-by: Roger Coll <[email protected]>
Co-authored-by: Braydon Kains <[email protected]>
Co-authored-by: Dani Louca <[email protected]>
Co-authored-by: wojtekzyla <[email protected]>
Co-authored-by: Célian GARCIA <[email protected]>
Co-authored-by: Murphy Chen <[email protected]>
Co-authored-by: Anthony Yeh <[email protected]>
Co-authored-by: Christos Markou <[email protected]>
Co-authored-by: Yang Song <[email protected]>
Co-authored-by: Vihas Makwana <[email protected]>
Co-authored-by: Daniel Jaglowski <[email protected]>
Co-authored-by: Khurshidali Shiekh <[email protected]>
Co-authored-by: Cyril Cressent <[email protected]>
Co-authored-by: Marc Lopez Rubio <[email protected]>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <[email protected]>
Co-authored-by: Stuart Buckingham <[email protected]>
Co-authored-by: Anthony Mirabella <[email protected]>
Co-authored-by: Jorge Creixell <[email protected]>
Co-authored-by: Robert Lankford <[email protected]>
Co-authored-by: Paulo Janotti <[email protected]>
Co-authored-by: Ludovic Fernandez <[email protected]>
Co-authored-by: Vishal Raj <[email protected]>
Co-authored-by: Gregor Jasny <[email protected]>
Co-authored-by: Pablo Baeyens <[email protected]>
Fiery-Fenix pushed a commit to Fiery-Fenix/opentelemetry-collector-contrib that referenced this issue Apr 24, 2025
…roup and logstream (open-telemetry#37297)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Add dynamic log_group_name and log_group_stream naming, based on
[awsemfexporter](https://github.com/LucianoGiannotti/opentelemetry-collector-contrib/tree/main/exporter/awsemfexporter)
<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
[open-telemetry#31382](open-telemetry#31382)

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Similar testing as on awsemfexporter, testing that names are replaced
properly before pushing logs to CW.


<!--Please delete paragraphs that you did not use before submitting.-->

---------

Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Matthieu MOREL <[email protected]>
Signed-off-by: Jared Tan <[email protected]>
Signed-off-by: Michael Shen <[email protected]>
Signed-off-by: Dani Louca <[email protected]>
Signed-off-by: Célian Garcia <[email protected]>
Signed-off-by: ChrsMark <[email protected]>
Signed-off-by: Marc Lopez Rubio <[email protected]>
Signed-off-by: Murphy Chen <[email protected]>
Co-authored-by: Antoine Toulme <[email protected]>
Co-authored-by: Sam DeHaan <[email protected]>
Co-authored-by: Andrzej Stencel <[email protected]>
Co-authored-by: Irina <[email protected]>
Co-authored-by: Curtis Robert <[email protected]>
Co-authored-by: Constança Manteigas <[email protected]>
Co-authored-by: Paschalis Tsilias <[email protected]>
Co-authored-by: Evan Bradley <[email protected]>
Co-authored-by: Evan Bradley <[email protected]>
Co-authored-by: Matthieu MOREL <[email protected]>
Co-authored-by: Adriel Perkins <[email protected]>
Co-authored-by: Sean Marciniak <[email protected]>
Co-authored-by: Nikos Angelopoulos <[email protected]>
Co-authored-by: Antoine Toulme <[email protected]>
Co-authored-by: Alex Boten <[email protected]>
Co-authored-by: mcube8 <[email protected]>
Co-authored-by: Will Li <[email protected]>
Co-authored-by: Luke Gao <[email protected]>
Co-authored-by: Andrew Wilkins <[email protected]>
Co-authored-by: Jared Tan <[email protected]>
Co-authored-by: Alex Van Boxel <[email protected]>
Co-authored-by: Michael Shen <[email protected]>
Co-authored-by: Roger Coll <[email protected]>
Co-authored-by: Braydon Kains <[email protected]>
Co-authored-by: Dani Louca <[email protected]>
Co-authored-by: wojtekzyla <[email protected]>
Co-authored-by: Célian GARCIA <[email protected]>
Co-authored-by: Murphy Chen <[email protected]>
Co-authored-by: Anthony Yeh <[email protected]>
Co-authored-by: Christos Markou <[email protected]>
Co-authored-by: Yang Song <[email protected]>
Co-authored-by: Vihas Makwana <[email protected]>
Co-authored-by: Daniel Jaglowski <[email protected]>
Co-authored-by: Khurshidali Shiekh <[email protected]>
Co-authored-by: Cyril Cressent <[email protected]>
Co-authored-by: Marc Lopez Rubio <[email protected]>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <[email protected]>
Co-authored-by: Stuart Buckingham <[email protected]>
Co-authored-by: Anthony Mirabella <[email protected]>
Co-authored-by: Jorge Creixell <[email protected]>
Co-authored-by: Robert Lankford <[email protected]>
Co-authored-by: Paulo Janotti <[email protected]>
Co-authored-by: Ludovic Fernandez <[email protected]>
Co-authored-by: Vishal Raj <[email protected]>
Co-authored-by: Gregor Jasny <[email protected]>
Co-authored-by: Pablo Baeyens <[email protected]>
@sorobon
Copy link
Author

sorobon commented Apr 29, 2025

Thank you!

@lasith-kg
Copy link

Thank you OpenTelemetry team!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exporter/awscloudwatchlogs awscloudwatchlogs exporter waiting-for-code-owners
Projects
None yet