[AWS S3 Exporter] Add support for predictably ordered S3 keys

[dacort]

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

Today, the S3 Exporter uses a random uniqueKey function to prevent collisions on file uploads.

Unfortunately, this can make it hard to know in what order files were uploaded. When using the S3 Exporter to stream raw log files (like stderr/stdout), this can make it difficult to maintain the ordering of the log lines once those files have been uploaded. For systems that don't provided structured logging (Spark), it results in a poor user experience.

Describe the solution you'd like

I'd like to be able to configure a different UniqueKeyField as mentioned in the todo that maintains ordering on upload.

Specifically, either an incrementing integer or UUIDv7. The latter seems better from an implementation perspective. The former is slightly more human-friendly, but could result in files getting overwritten if the exporter is restarted.

Describe alternatives you've considered

• Other solutions like fluent-bit that do support this (via the $INDEX formatter)
• Utilizing multi-part uploads and UploadPartCopy to emulate appends in S3 as demonstrated here: https://github.com/dacort/s3-diff-uploader

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/awss3: @atoulme @pdelewski @Erog38

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

Hello!

I agree that we should provide an option to configure the ordering.

Regarding the approaches you shared, I feel like we should stick with UUID7.

If we try to implement this with a sequential integer, we might lose the last known value between collector runs. For example, during the first run, we might save 100 files and the counter would be at 100. However, if the process is restarted, the counter could reset to 0 (unless we implement a mechanism to persist and restore the last known counter value) and it might point to same bucket.

[dacort]

nod Yep, my thoughts as well.

Happy to contribute the PR - I have an existing test of the UUIDv7 approach I can clean up.

[VihasMakwana]

@dacort Sounds good. I'll be happy to review it.
I'd like to hear what codeowners i.e. (@atoulme @pdelewski @Erog38) has to say about this

[atoulme]

Sure, sgtm

[Erog38]

Conceptually this makes sense to me. Being able to understand the order in which files are read would help troubleshooting issues as well.

The UUIDv7 approach also sounds like a decent solution since you can still eyeball the order of them to a degree. I couldn't see an incrementing int working in a distributed system where multiple collectors are possibly writing to the same S3 bucket though unless you still appended a randomness factor per-collector.