Skip to content

[internal/aws] migrate to AWS v2 SDK #40123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Jun 6, 2025
Merged

[internal/aws] migrate to AWS v2 SDK #40123

Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
1c3493c
Migrate internal/aws/awsutils to use aws-sdk-for-go-v2
seongpil0948 May 14, 2025
03a4baa
fix reflect mjlshen review
seongpil0948 May 16, 2025
b0263cf
enhancement(awsutil): conn partition
seongpil0948 May 17, 2025
f45312e
feat: reflect the awsutil module to the component in use
seongpil0948 May 17, 2025
750e990
feat(awscontainerinsightreceiver): reflect the awsutil module to the …
seongpil0948 May 17, 2025
1612108
fix(awsxray): reflect the awsutil module to the component in use
seongpil0948 May 17, 2025
44a8303
Merge branch 'main' into 37728_internal-aws
seongpil0948 May 19, 2025
f465b53
fix(aws): resolve lint issues in AWS SDK v2 migration
seongpil0948 May 19, 2025
7cd18b6
Merge branch '37728_internal-aws' of github.com:seongpil0948/opentele…
seongpil0948 May 19, 2025
ce37d3b
Merge branch 'main' into 37728_internal-aws
seongpil0948 May 19, 2025
736bce9
fix(awsxrayexporter): awsxray_test fail
seongpil0948 May 22, 2025
9f58585
Merge branch 'main' into 37728_internal-aws
seongpil0948 May 22, 2025
5df4d44
Merge branch 'main' into 37728_internal-aws
seongpil0948 May 23, 2025
08f834f
Merge branch 'main' into 37728_internal-aws
seongpil0948 May 23, 2025
d0ae760
propagate contexts through when possible instead of defining a new one.
seongpil0948 May 29, 2025
e162e7a
feat(awsutil) fix: remove it for the purpose of the issue, feat: rep…
seongpil0948 May 29, 2025
69fac41
Merge branch 'main' into 37728_internal-aws
seongpil0948 May 29, 2025
9eceb80
chore: gotidy
seongpil0948 May 29, 2025
f384547
fix: resolve lint, fmt errors
seongpil0948 May 29, 2025
be5cba7
chore: Remove unused code
seongpil0948 May 30, 2025
ed2450b
Remove unnecessary code
seongpil0948 May 30, 2025
6eb2f85
Apply suggestions from code review
seongpil0948 May 31, 2025
d8d7673
Apply suggestions from code review
seongpil0948 May 31, 2025
8dd87ef
Merge branch 'main' into 37728_internal-aws
atoulme Jun 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions exporter/awscloudwatchlogsexporter/exporter.go
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should propagate contexts through when possible instead of defining a new one. E.g. for this exporter

opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter/factory.go

Lines 42 to 48 in 08f834f

func createLogsExporter(_ context.Context, params exporter.Settings, config component.Config) (exporter.Logs, error) {
expConfig, ok := config.(*Config)
if !ok {
return nil, errors.New("invalid configuration type; can't cast to awscloudwatchlogsexporter.Config")
}
return newCwLogsExporter(expConfig, params)
}
currently accepts _ context.Context. We can update it to ctx context.Context and pipe it through.

We can then change this function signature to:

func newCwLogsPusher(ctx context.Context, expConfig *Config, params exp.Settings) (*cwlExporter, error) {

  // ...
  // No need to setup a new ctx := context.TODO()
  awsConfig, err := awsutil.GetAWSConfig(ctx, params.Logger, &expConfig.AWSSessionSettings)
}

Something similar can be done for awsemfexporter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The newCwLogsExporter will receive a factor like..

	ctx := context.Background()
	exp, err := newCwLogsExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah exactly, basically w/e context is passed into createLogsExporter will be propagated through. I think you're right that currently it's just context.Background() right now, but if it ever changes it'll only need to be changed in one spot.

 func createLogsExporter(ctx context.Context, params exporter.Settings, config component.Config) (exporter.Logs, error) { 
 	// ...
 	return newCwLogsExporter(ctx, expConfig, params) 
 }

func newCwLogsExporter(ctx context.Context, config component.Config, params exp.Settings) (exp.Logs, error) {
        // ...
        // No need to setup a new ctx, can continue using the one passed into createLogsExporter
        exp, err := newCwLogsPusher(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
        // ...
}

func newCwLogsPusher(ctx context.Context, expConfig *Config, params exp.Settings) (*cwlExporter, error) {
        // ...
        // Keep using the same ctx
        awsConfig, err := awsutil.GetAWSConfig(ctx, params.Logger, &expConfig.AWSSessionSettings)
}

Original file line number Diff line number Diff line change
Expand Up @@ -46,15 +46,14 @@ type emfMetadata struct {
LogStreamName string `json:"log_stream_name,omitempty"`
}

func newCwLogsPusher(expConfig *Config, params exp.Settings) (*cwlExporter, error) {
func newCwLogsPusher(ctx context.Context, expConfig *Config, params exp.Settings) (*cwlExporter, error) {
if expConfig == nil {
return nil, errors.New("awscloudwatchlogs exporter config is nil")
}

expConfig.logger = params.Logger

// create AWS session
awsConfig, err := awsutil.GetAWSConfig(params.Logger, &expConfig.AWSSessionSettings)
awsConfig, err := awsutil.GetAWSConfig(ctx, params.Logger, &expConfig.AWSSessionSettings)
if err != nil {
return nil, err
}
Expand All @@ -79,14 +78,14 @@ func newCwLogsPusher(expConfig *Config, params exp.Settings) (*cwlExporter, erro
return logsExporter, nil
}

func newCwLogsExporter(config component.Config, params exp.Settings) (exp.Logs, error) {
func newCwLogsExporter(ctx context.Context, config component.Config, params exp.Settings) (exp.Logs, error) {
expConfig := config.(*Config)
logsPusher, err := newCwLogsPusher(expConfig, params)
logsPusher, err := newCwLogsPusher(ctx, expConfig, params)
if err != nil {
return nil, err
}
return exporterhelper.NewLogs(
context.TODO(),
ctx, // Use the passed context instead of context.TODO()
params,
config,
logsPusher.consumeLogs,
Expand Down
6 changes: 4 additions & 2 deletions exporter/awscloudwatchlogsexporter/exporter_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ func TestConsumeLogs(t *testing.T) {
expCfg.LogGroupName = "testGroup"
expCfg.LogStreamName = "testStream"
expCfg.MaxRetries = 0
exp, err := newCwLogsPusher(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newCwLogsPusher(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))

testcases := []struct {
id string
Expand Down Expand Up @@ -418,7 +418,9 @@ func TestNewExporterWithoutRegionErr(t *testing.T) {
factory := NewFactory()
expCfg := factory.CreateDefaultConfig().(*Config)
expCfg.MaxRetries = 0
exp, err := newCwLogsExporter(expCfg, exportertest.NewNopSettings(metadata.Type))

ctx := context.Background()
exp, err := newCwLogsExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.Nil(t, exp)
assert.Error(t, err)
}
4 changes: 2 additions & 2 deletions exporter/awscloudwatchlogsexporter/factory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,10 @@ func createDefaultConfig() component.Config {
}
}

func createLogsExporter(_ context.Context, params exporter.Settings, config component.Config) (exporter.Logs, error) {
func createLogsExporter(ctx context.Context, params exporter.Settings, config component.Config) (exporter.Logs, error) {
expConfig, ok := config.(*Config)
if !ok {
return nil, errors.New("invalid configuration type; can't cast to awscloudwatchlogsexporter.Config")
}
return newCwLogsExporter(expConfig, params)
return newCwLogsExporter(ctx, expConfig, params)
}
2 changes: 0 additions & 2 deletions exporter/awscloudwatchlogsexporter/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ require (
)

require (
github.com/aws/aws-sdk-go v1.55.7 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.10 // indirect
github.com/aws/aws-sdk-go-v2/config v1.29.14 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.67 // indirect
Expand All @@ -47,7 +46,6 @@ require (
github.com/gobwas/glob v0.2.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/hashicorp/go-version v1.7.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/knadh/koanf/maps v0.1.2 // indirect
github.com/knadh/koanf/providers/confmap v1.0.0 // indirect
Expand Down
9 changes: 0 additions & 9 deletions exporter/awscloudwatchlogsexporter/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 2 additions & 3 deletions exporter/awsemfexporter/emf_exporter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,14 @@ type emfExporter struct {
}

// newEmfExporter creates a new exporter using exporterhelper
func newEmfExporter(config *Config, set exporter.Settings) (*emfExporter, error) {
func newEmfExporter(ctx context.Context, config *Config, set exporter.Settings) (*emfExporter, error) {
if config == nil {
return nil, errors.New("emf exporter config is nil")
}

config.logger = set.Logger

// create AWS session
awsConfig, err := awsutil.GetAWSConfig(set.Logger, &config.AWSSessionSettings)
awsConfig, err := awsutil.GetAWSConfig(ctx, set.Logger, &config.AWSSessionSettings)
if err != nil {
return nil, err
}
Expand Down
42 changes: 28 additions & 14 deletions exporter/awsemfexporter/emf_exporter_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ func TestConsumeMetrics(t *testing.T) {
expCfg := factory.CreateDefaultConfig().(*Config)
expCfg.Region = "us-west-2"
expCfg.MaxRetries = 0
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)

Expand Down Expand Up @@ -103,7 +103,7 @@ func TestConsumeMetricsWithNaNValues(t *testing.T) {
expCfg.Region = "us-west-2"
expCfg.MaxRetries = 0
expCfg.OutputDestination = "stdout"
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)
md := tc.generateFunc(tc.testName)
Expand Down Expand Up @@ -142,7 +142,7 @@ func TestConsumeMetricsWithInfValues(t *testing.T) {
expCfg.Region = "us-west-2"
expCfg.MaxRetries = 0
expCfg.OutputDestination = "stdout"
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)
md := tc.generateFunc(tc.testName)
Expand All @@ -160,7 +160,7 @@ func TestConsumeMetricsWithOutputDestination(t *testing.T) {
expCfg.Region = "us-west-2"
expCfg.MaxRetries = 0
expCfg.OutputDestination = "stdout"
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)

Expand All @@ -181,7 +181,7 @@ func TestConsumeMetricsWithLogGroupStreamConfig(t *testing.T) {
expCfg.MaxRetries = defaultRetryCount
expCfg.LogGroupName = "test-logGroupName"
expCfg.LogStreamName = "test-logStreamName"
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)

Expand All @@ -208,7 +208,7 @@ func TestConsumeMetricsWithLogGroupStreamValidPlaceholder(t *testing.T) {
expCfg.MaxRetries = defaultRetryCount
expCfg.LogGroupName = "/aws/ecs/containerinsights/{ClusterName}/performance"
expCfg.LogStreamName = "{TaskId}"
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)

Expand Down Expand Up @@ -239,7 +239,7 @@ func TestConsumeMetricsWithOnlyLogStreamPlaceholder(t *testing.T) {
expCfg.MaxRetries = defaultRetryCount
expCfg.LogGroupName = "test-logGroupName"
expCfg.LogStreamName = "{TaskId}"
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)

Expand Down Expand Up @@ -270,7 +270,7 @@ func TestConsumeMetricsWithWrongPlaceholder(t *testing.T) {
expCfg.MaxRetries = defaultRetryCount
expCfg.LogGroupName = "test-logGroupName"
expCfg.LogStreamName = "{WrongKey}"
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)

Expand Down Expand Up @@ -301,7 +301,7 @@ func TestPushMetricsDataWithErr(t *testing.T) {
expCfg.MaxRetries = 0
expCfg.LogGroupName = "test-logGroupName"
expCfg.LogStreamName = "test-logStreamName"
exp, err := newEmfExporter(expCfg, exportertest.NewNopSettings(metadata.Type))
exp, err := newEmfExporter(ctx, expCfg, exportertest.NewNopSettings(metadata.Type))
assert.NoError(t, err)
assert.NotNil(t, exp)

Expand Down Expand Up @@ -333,7 +333,8 @@ func TestNewExporterWithoutConfig(t *testing.T) {
settings := exportertest.NewNopSettings(metadata.Type)
t.Setenv("AWS_STS_REGIONAL_ENDPOINTS", "fake")

exp, err := newEmfExporter(expCfg, settings)
ctx := context.Background()
exp, err := newEmfExporter(ctx, expCfg, settings)
assert.Error(t, err)
assert.Nil(t, exp)
assert.Equal(t, expCfg.logger, settings.Logger)
Expand Down Expand Up @@ -370,7 +371,8 @@ func TestNewExporterWithMetricDeclarations(t *testing.T) {
params := exportertest.NewNopSettings(metadata.Type)
params.Logger = zap.New(obs)

exp, err := newEmfExporter(expCfg, params)
ctx := context.Background()
exp, err := newEmfExporter(ctx, expCfg, params)
assert.NoError(t, err)
assert.NotNil(t, exp)
err = expCfg.Validate()
Expand Down Expand Up @@ -403,7 +405,8 @@ func TestNewExporterWithMetricDeclarations(t *testing.T) {
}

func TestNewExporterWithoutSession(t *testing.T) {
exp, err := newEmfExporter(nil, exportertest.NewNopSettings(metadata.Type))
ctx := context.Background()
exp, err := newEmfExporter(ctx, nil, exportertest.NewNopSettings(metadata.Type))
assert.Error(t, err)
assert.Nil(t, exp)
}
Expand Down Expand Up @@ -432,8 +435,19 @@ func TestNewEmfExporterWithoutConfig(t *testing.T) {
expCfg := factory.CreateDefaultConfig().(*Config)
settings := exportertest.NewNopSettings(metadata.Type)
t.Setenv("AWS_STS_REGIONAL_ENDPOINTS", "fake")

exp, err := newEmfExporter(expCfg, settings)
t.Setenv("AWS_ACCESS_KEY_ID", "invalid")
t.Setenv("AWS_SECRET_ACCESS_KEY", "invalid")
t.Setenv("AWS_SESSION_TOKEN", "invalid")

// 모든 리전 소스 차단
t.Setenv("AWS_REGION", "")
t.Setenv("AWS_DEFAULT_REGION", "")
t.Setenv("AWS_EC2_METADATA_DISABLED", "true")
t.Setenv("AWS_CONFIG_FILE", "/dev/null")
t.Setenv("AWS_SHARED_CREDENTIALS_FILE", "/dev/null")

ctx := context.Background()
exp, err := newEmfExporter(ctx, expCfg, settings)
assert.Error(t, err)
assert.Nil(t, exp)
assert.Equal(t, expCfg.logger, settings.Logger)
Expand Down
2 changes: 1 addition & 1 deletion exporter/awsemfexporter/factory.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ func createDefaultConfig() component.Config {
func createMetricsExporter(ctx context.Context, params exporter.Settings, config component.Config) (exporter.Metrics, error) {
expCfg := config.(*Config)

emfExp, err := newEmfExporter(expCfg, params)
emfExp, err := newEmfExporter(ctx, expCfg, params)
if err != nil {
return nil, err
}
Expand Down
2 changes: 0 additions & 2 deletions exporter/awsemfexporter/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ require (
)

require (
github.com/aws/aws-sdk-go v1.55.7 // indirect
github.com/aws/aws-sdk-go-v2 v1.36.3 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.10 // indirect
github.com/aws/aws-sdk-go-v2/config v1.29.14 // indirect
Expand All @@ -50,7 +49,6 @@ require (
github.com/gobwas/glob v0.2.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/hashicorp/go-version v1.7.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/knadh/koanf/maps v0.1.2 // indirect
github.com/knadh/koanf/providers/confmap v1.0.0 // indirect
Expand Down
9 changes: 0 additions & 9 deletions exporter/awsemfexporter/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion exporter/awsxrayexporter/awsxray.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ func newTracesExporter(ctx context.Context, cfg *Config, set exporter.Settings,
typeLog := zap.String("type", set.ID.Type().String())
nameLog := zap.String("name", set.ID.String())
logger := set.Logger
awsConfig, err := awsutil.GetAWSConfig(logger, &cfg.AWSSessionSettings)
awsConfig, err := awsutil.GetAWSConfig(ctx, logger, &cfg.AWSSessionSettings)
if err != nil {
return nil, err
}
Expand Down
3 changes: 2 additions & 1 deletion exporter/awsxrayexporter/awsxray_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,8 @@ func TestTelemetryEnabled(t *testing.T) {
assert.EqualValues(t, 1, sink.StopCount.Load())
assert.True(t, sink.HasRecording())
got := sink.Rotate()
assert.EqualValues(t, 0, *got.BackendConnectionErrors.HTTPCode4XXCount)
assert.EqualValues(t, 1, *got.BackendConnectionErrors.HTTPCode4XXCount)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BackendConnectionErrors.OtherCount was 1 in version 1, but HTTPCode4XXCount was 1 more after v2 conversion

assert.EqualValues(t, 0, *got.BackendConnectionErrors.OtherCount)
}

func BenchmarkForTracesExporter(b *testing.B) {
Expand Down
2 changes: 0 additions & 2 deletions exporter/awsxrayexporter/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ require (
)

require (
github.com/aws/aws-sdk-go v1.55.7 // indirect
github.com/aws/aws-sdk-go-v2/config v1.29.14 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.67 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30 // indirect
Expand All @@ -46,7 +45,6 @@ require (
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/go-version v1.7.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/knadh/koanf/maps v0.1.2 // indirect
github.com/knadh/koanf/providers/confmap v1.0.0 // indirect
Expand Down
11 changes: 0 additions & 11 deletions exporter/awsxrayexporter/go.sum
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading