Run integration tests with Go

[MrAlias]

Replace Integration Tests with Go testing implementation

This replaces the existing integration test run on a kind server with ones run directly from Go on the host machine.

Resolves #93

Add instrumentation_integration_test.go

This is the integration testing file. It runs integration tests for all existing coverage:

• autosdk
• databasesql
• gin
• grpc
• kafka-go
• nethttp
• nethttp_custom
• otelglobal

Each test is run as a sub-test of the main TestIntegration. This allows users to run all test:

go test -run=TestIntegration

Or just a single test for a packge. For example:

go test -run=TestIntegration/KafkaGo

The test is skipped if being run in short mode (i.e. go test -short), or if the user does not have permissions to set memory limits on the host (the same as our eBPF verifier tests).

The test can be run from within a virtualized environment if permissions are not available by default with something like the following:

docker run --rm -it --privileged --network=host --user root -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD":/usr/src/go.opentelemetry.io/auto -w /usr/src/go.opentelemetry.io/auto golang:1.24.1 sh -c "apt update; apt install docker.io bats jq -y; go test -v -run=TestIntegration"

Note, the Makefile has been updated (see below).

Processing reuse where possible

Instead of writing every part of the end-to-end tests in Go, existing functionality is reused where possible. This means that the test results are still evaluated with BATS.

This is an area of improvement to address after this PR (see below).

Event flow via signal

The end-to-end tests no longer rely on sleeps/timing to manage event flow. The integration tests signal the spawned process using Linux process signaling. Once signaled, the application knows it is being monitored and starts it processing.

Kafka Go testing

The Kafka testing is unique. It needs to run a Kafka server to communicate with. This consolidates that process into the end-to-end testing application.

The Docker Go client is used to managed and run a Kafka container.

Removed BATS verification of traces.json

This was talked about at the last SIG meeting and is accomplished in these changes. The BATS verification against a static JSON output is unstable. Instead of updating all traces.json to the lastest version of Go used in this change (and other spans added to the Kafka testing), the verification of traces.json is removed.

Makefile

All fixture targets are removed from the Makefile.

The docker development target (make docker-dev) is updated. It now runs with the correct permissions and mounts to run integration tests.

The docker test target (make docker-test) is updated. It now runs with the correct permissions and mounts to run the full test suite, including integration tests.

Kind Integration tests

The kind integration testing is removed along will all of the related artifacts.

Follow-up items

• Compatibility testing. Now that these end-to-end tests are reasonably fast, Github actions can be added to test a matrix of supported versions.
  • Investigate running a unified Kafka server
• Replace BATS. There is no need to shell-out to evaluate the captured telemetry. This can be replaced with an entirely Go solution.

[RonFed]

This is amazing.
Going to go over it more tomorrow, left a few comments regarding the CI.

[RonFed]

I really like this.

1. Regarding the kafka case I'm wondering how it'll scale once we expand the tests to have more cases in the matrix (client version, go version etc') - we don't want to create the kafka broker for each case - so I think we need to have a way to register a "setup" function to our tests - this can be relevant also to the database/sql one and more future libraries.
2. Also relating to the first point - I think we can make use of https://github.com/testcontainers/testcontainers-go instead of directly using the docker api, I think that lib can provide some nice abstractions for us.

[MrAlias]

1. Regarding the kafka case I'm wondering how it'll scale once we expand the tests to have more cases in the matrix (client version, go version etc') - we don't wan't to create the kafka broker for each case - so I think we need to have a way to register a "setup" function to our tests - this can be relevant also to the database/sql one and more future libraries.

I'm not seeing an issue running a broker per test given they will be run in parallel.

But more importantly, I do not see it being relevant to this PR. We can address that issue when we address running multiple of these tests in a matrix test. This is something out-of-scope for this PR and is already being tracked as a follow up task in the description. I'll be sure to include this feedback in the issue I create for that task when this PR merges.

[MrAlias]

2. Also relating to the first point - I think we can make use of https://github.com/testcontainers/testcontainers-go instead of directly using the docker api, I think that lib can provide some nice abstractions for us.

I've tried switching to this but am not able to get this working. It looks to be due to my lack of knowledge about how to get the testcontainers to configure the network appropriately.

Given we have a working solution for the single kafka test case, and given the abstractions are not going to make a significant reduction of code here (my non-working prototype had 10s of less lines) , I'm going to keep the use of the Docker API the way it is.

It is something we can address in the future if we find a need or have the motivation to.

[pellared]

I think we can make use of https://github.com/testcontainers/testcontainers-go instead of directly using the docker api, I think that lib can provide some nice abstractions for us.

Based on experience I had so far (amount of issues, stability of API), I would rather look into https://github.com/ory/dockertest. We use it in https://github.com/signalfx/splunk-otel-go. But using Docker API seems also fine and I do not think it should block this PR.