Skip to content

Collector: init memory allocation investigation, need to upgrade aws-sdk-go #1827

Open
@maxday

Description

@maxday

Running the collector binary with GODEBUG="inittrace=1" allows to see init information.

An interesting part in the aws-sdk-go dependency where the alloc number is huge due to the endpoint static definition.

GODEBUG="inittrace=1" ./build/extensions/collector 2>&1 | grep "aws-sdk-go/"

init github.com/aws/aws-sdk-go/internal/sdkrand @0.88 ms, 0.038 ms clock, 5448 bytes, 3 allocs
init github.com/aws/aws-sdk-go/internal/ini @4.0 ms, 0.002 ms clock, 96 bytes, 1 allocs
init github.com/aws/aws-sdk-go/aws/credentials @4.5 ms, 0.001 ms clock, 512 bytes, 7 allocs
init github.com/aws/aws-sdk-go/aws/awsutil @4.6 ms, 0.032 ms clock, 3512 bytes, 31 allocs
-----> init github.com/aws/aws-sdk-go/aws/endpoints @4.7 ms, 3.2 ms clock, 2233936 bytes, 13063 allocs <-----
init github.com/aws/aws-sdk-go/aws @11 ms, 0.003 ms clock, 128 bytes, 2 allocs
init github.com/aws/aws-sdk-go/aws/request @11 ms, 0.010 ms clock, 832 bytes, 7 allocs
init github.com/aws/aws-sdk-go/aws/corehandlers @11 ms, 0.004 ms clock, 3184 bytes, 50 allocs
init github.com/aws/aws-sdk-go/private/protocol @11 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aws/aws-sdk-go/private/protocol/json/jsonutil @11 ms, 0 ms clock, 48 bytes, 2 allocs
init github.com/aws/aws-sdk-go/private/protocol/rest @11 ms, 0.001 ms clock, 32 bytes, 2 allocs
init github.com/aws/aws-sdk-go/aws/signer/v4 @11 ms, 0.005 ms clock, 2136 bytes, 6 allocs
init github.com/aws/aws-sdk-go/service/sso @12 ms, 0 ms clock, 256 bytes, 2 allocs
init github.com/aws/aws-sdk-go/service/ssooidc @12 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aws/aws-sdk-go/aws/credentials/ssocreds @12 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aws/aws-sdk-go/service/sts @12 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aws/aws-sdk-go/aws/session @12 ms, 0.003 ms clock, 320 bytes, 5 allocs

In addition to perform huge amount of allocation, it creates overhead during init time.

We should migrate to the sdk v2 which does not have this huge startup allocation.

Investigation which package is responsible to import this running go mod why github.com/aws/aws-sdk-go/aws/endpoints gives us

github.com/open-telemetry/opentelemetry-lambda/collector/internal/lifecycle
github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter
github.com/prometheus/prometheus/config
github.com/prometheus/common/sigv4
github.com/aws/aws-sdk-go/aws
github.com/aws/aws-sdk-go/aws/endpoints

So we need to upgrade github.com/open-telemetry/opentelemetry-collector-contrib dependency. Fortunately there is ongoing work to fix this issue here, so I created this issue to keep track of that not to forget about upgrading. open-telemetry/opentelemetry-collector-contrib#40123

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions