Optimizing the trace sdk

[jschaf]

Problem Statement

I'm exploring ways to optimize the trace functionality and wanted to gauge the appetite for optimization.

I'm willing to send PRs.

Proposed Solution

There are a few areas I've exprimented with in https://github.com/jschaf/observe. Ranked roughly in terms of feasibility, impact, and long term maintenance costs.

0. Use atomics for Span.End instead of a mutex. Provides about a 1.5x speed up on the Span.End benchmark. As a side benefit, reduces the Span size by 16 bytes since time.Time takes 24 bytes.

1. Inline hex encoding and decoding for TraceID and SpanID. Provides about a 2x speedup for parsing (27 ns to 12 ns) and reduces allocations to 0.

2. Zero allocation state parsing. This is a fair bit more complex than the current parsing, but has about the same line count. Provides about a 1.8x speed up and reduces allocations to zero.

3. [Still baking] Get syscall.Gettimeofday directly instead of time.Now. Saves about 15 ns compare to 30 ns for time.Now. I suspect there's a bunch of unhandled complexity here. It'd be swell if there was an alternative time library that handled this.

Alternatives

Doing nothing is perfectly reasonable.

Prior Art

https://www.pingcap.com/blog/how-we-trace-a-kv-database-with-less-than-5-percent-performance-impact/: They squeeze out performance using time-stamp counters and thread local state, both of which would be difficult to emulate in Go.

[dmathieu]

Any improvement is welcome. But each separate improvement should be its own PR with benchmark comparison (either using the existing benchmarks, or adding new ones).

[jschaf]

@dmathieu Sent the first PR: #6730

[pellared]

@jschaf, why has this issue been closed?

[jschaf]

I figured the cost-benefit wasn't worth it. @dmathieu said the absolute speedup of the atomics PR was small—the other optimizations are even smaller.

I'm happy to reopen if y'all are interested. I've already implemented them in my experimental code, so it's not a big lift.

[dmathieu]

I think looking at your other proposals would be nice.

[jschaf]

Sent #6791 for optimization 1: inline hex encoding and decoding.

Speeds up the string method by 1.8x.

TraceID_DotString-12                                   42.37n ±  0%    24.80n ±  0%  -41.45% (p=0.000 n=10)
SpanID_DotString-12                                    31.30n ±  0%    17.22n ±  0%  -44.98% (p=0.000 n=10)

[jschaf]

Added a benchmark for parsing the hex. It's about 4x faster and doesn't allocate.

goos: darwin
goarch: arm64
pkg: go.opentelemetry.io/otel/sdk/trace
cpu: Apple M2 Max
                  │ ../private/base_hex.txt │       ../private/new_hex.txt        │
                  │         sec/op          │   sec/op     vs base                │
TraceIDFromHex-12               56.47n ± 0%   15.96n ± 0%  -71.74% (p=0.000 n=10)
SpanIDFromHex-12               34.680n ± 0%   8.742n ± 1%  -74.79% (p=0.000 n=10)
geomean                         44.26n        11.81n       -73.31%

                  │ ../private/base_hex.txt │         ../private/new_hex.txt          │
                  │          B/op           │    B/op     vs base                     │
TraceIDFromHex-12                16.00 ± 0%    0.00 ± 0%  -100.00% (p=0.000 n=10)
SpanIDFromHex-12                 8.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
geomean                          11.31                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean

                  │ ../private/base_hex.txt │         ../private/new_hex.txt          │
                  │        allocs/op        │ allocs/op   vs base                     │
TraceIDFromHex-12                1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
SpanIDFromHex-12                 1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
geomean                          1.000                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean