Add tracing to the tenant federation mergeQueryable

[simonswine]

This is a work item that came out the review of #4117

The mergeQueryable aggregates data from underlying queryables concurrently. Understanding the call sequence is only possible providing spans.

/assign @simonswine

Some examples that creates spans:

cortex/pkg/querier/blocks_store_queryable.go

Line 318 in d382e1d

spanLog, spanCtx := spanlogger.New(q.ctx, "blocksStoreQuerier.LabelNames")

cortex/pkg/querier/worker/scheduler_processor.go

Line 134 in 5009276

tracer := opentracing.GlobalTracer()

[pstibrany]

Your first linked example is what I would suggest to use… getting spanlogger from existing context, closing it at the end of method, and passing new context to the called methods is a good start. Additionally logging supplied tenantIDs via new spanlogger (to make them visible when viewing the trace) would be useful.

We can go even further and create one child span per processed tenant – that can be useful for people troubleshooting slow queries to understand if there is specific tenant that was slow for some reason.

[vlad-diachenko]

I have implemented a logic to add tenant_id tag to each tracing span if tenant_id is defined in context.
Also, I have added a few additional spans to mergeQuerier:

• mergeQuerier.select
• mergeQuerier.labelValues
• mergeQuerier.labelNames