Auto detect the log type

[constanca-m]

Component(s)

extension/encoding/awslogsencoding

Is your feature request related to a problem? Please describe.

Currently, the user needs to specify the format of the log type, like this:

extensions:
  awslogs_encoding/cloudwatch:
    format: cloudwatch_logs_subscription_filter

It would be useful it this would not be necessary, and the extension could automate this part.

The format should still be support in case the extension needs to accept a more strict format.

Describe the solution you'd like

We would define the configuration like this:

extensions:
  awslogs_encoding/cloudwatch:
    format: auto

Then, once the extension gets a log to unmarshal, it should auto-detect the type.

Describe alternatives you've considered

N/A

Additional context

Currently we accept the following formats:

• Cloudwatch subscription filter format - JSON format
• S3 access log: plain text
• VPC flow log: compressed or parquet (parquet yet to be implemented)

[github-actions]

Pinging code owners:

• extension/encoding/awslogsencoding: @axw @constanca-m

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[axw]

In principle I like the idea, but only if:

• it can be done perfectly, with zero chance of misidentifying the format
• the overhead is negligible

Also consider that we'll need to handle WAF, CloudTrail, and CloudFront access logs.

So I guess I'd like to see a proof of concept of this, but maybe only after those formats are implemented to make sure we're not making it harder to support them all.

[axw]

/label -needs-triage

[constanca-m]

@axw I was thinking of something like this:

Log	Compression	Format	Starts with
VPC flow log	gzip / parquet	Plain	-
S3 access log	-	-	-
Cloudtrail log	gzip	JSON	{"Records": [
WAF log	gzip	JSON
Subscription filter log	gzip	JSON

I thought that for now the auto format will work like this:

• Check the first bytes of the header to see which compression was used
  • If not, then it is a S3 access log.
• Decompress the content. Now check the first byte.
  • Is it {? If yes, continue, otherwise it is VPC flow log.
• Check the first bytes.
  • Does it match the WAF Log?
• Otherwise, it can only be WAF log or subscription filter log. We can decide to do bytes.contains logGroup, for example, but this would mean going over the data.

---

So the auto format relies on which kind of logs we support at the moment, so the way we identify VPC flow log, S3 access log and Cloudtrail log could change in the future. I think for this 3 it could be quite fast, but then for WAF log and subscription filter I don't see a way other than looking for specific fields.

It is not the most logic algorithm I think... But I don't see any cheap way in identifying the type of logs 😅

[axw]

@constanca-m that sounds like an OK approach to me. I think we'll need to try it out to see if it's going to work.

Rather than using bytes.Contains, perhaps we can use https://pkg.go.dev/github.com/tidwall/gjson to avoid false positives.