proposal: new net/http/pprof/v2 package

[mknyszek]

proposal: new net/http/pprof/v2 package

Background

The net/http/pprof package provides standard endpoints for operators to collect runtime diagnostic data from Go programs. It is critical to Go's production stack, with 31,000+ public package imports.

Unfortunately, it has one major problem. net/http/pprof registers all of its handlers with the default net/http server mux. This property poses a security risk by making it easy to accidentally install potentially insecure endpoints (leaking data through execution traces, profiles, etc.). Although this convenience is fine for servers that only ever run internally, it is a big problem for public-facing services. Big projects have run into the problem, requiring a hasty and urgent change to plug the security hole, and ultimately resolved the problem by only exposing net/http/pprof endpoints over an alternative.

This security risk is captured by the following two issues:

• net/http/pprof: assess and document security implications of the goroutines endpoint #46307
• proposal: net/http/pprof/v2: disallow package to register to the default mux #42834

A second, more minor problem, is that the package is undermaintained. For the most part, the fact that the package has not moved much is okay, since most of our diagnostics haven't changed in ways that would need it to keep up. However, the lack of prioritization has led to several minor proposals for additional functionality accumulating over the years, and it is starting to fall behind. In particular, two useful pieces of functionality, the injection of CPU profile sample events into execution traces and the new flight recorder API in Go 1.25, are missing from the package entirely. Below is the full set of public proposals that haven't received much attention, but come from diagnostics stakeholders like DataDog.

• proposal: net/http/pprof: function to register all profiles to a mux #71213: function to register all profiles to a mux
• net/http/pprof: traces collected from the /debug/pprof/trace endpoint don't include CPU profile samples by default #66679: traces collected from the /debug/pprof/trace endpoint don't include CPU profile samples by default
• proposal: net/http/pprof: support finer-grained delta computation #57765: support finer-grained delta computation
• proposal: net/http/pprof: add parameter for sampling frequency for CPU profiles #57488: add parameter for sampling frequency for CPU profiles

I believe part of the problem is that we're reluctant to extend the net/http/pprof package given its issues, and we've avoided making simple and obvious improvements because the package needs to be rethought a bit. This proposal is exactly that.

Proposal

I propose the creation of a new net/http/pprof/v2 package which would no longer install endpoints to the default net/http mux. Simultaneously, I propose we add some simple new features to encourage users to reexamine their use of net/http/pprof and move over to the new package.

This proposal is broken up into several sub-proposals to make it more digestible. I propose to:

1. Create a new package, net/http/pprof/v2, that does not install any handlers on package initialization.
   1. Re-export most of the existing functionality with tweaked names.
   2. Drop the Index endpoint.
   3. Add a cpu endpoint, which is the same as the profile endpoint for CPU profiling.
   4. Provide a convenient way to register all handlers (the new RegisterHandlers function).
2. Add handlers that control flight recording.
3. Add a query parameter to the cpu endpoint for controlling the CPU profiling rate.
4. Add a query parameter to the trace endpoint for enabling simultaneous CPU profiling.

Side-note: the security issue is potentially serious enough that a deprecation notice might be worthwhile in net/http/pprof, despite the annoyance to downstream users.

New package

I propose we create a new package, net/http/pprof/v2, that does not install any handlers on package initialization. Instead, it provides a convenient way to register all handlers (RegisterHandlers) at the default location for a given net/http.ServeMux, essentially implementing #71213.

I also propose removing the Index handler. Its functionality overlaps with the proposed RegisterHandlers function (including the human-readable HTML page for discovering available diagnostics), since it still assumes all the endpoints are installed at /debug/pprof and thus doesn't offer any new customizability.

The full proposed API, with documentation, is listed below.

package net/http/pprof/v2

// RegisterHandlers is a convenience function that installs all the handlers provided by
// this package into the provided ServeMux with the path prefix /debug/pprof.
//
// It also installs a handler for /debug/pprof which responds with an HTML page
// containing a list of all available profiles.
func RegisterHandlers(mux *http.ServeMux)

// HandleCommandLine is an HTTP handler func that responds with the running program's
// command line, with arguments separated by NUL bytes.
//
// RegisterHandlers registers this handler as /debug/pprof/cmdline.
func HandleCommandLine(w http.ResponseWriter, r *http.Request)

// NewProfileHandler creates an http.Handler that responds with the named profile.
// Available profiles can be found at [runtime/pprof.Profile]. If NewProfileHandler
// is called with an invalid profile name, the result will be nil.
//
// If no seconds GET parameter is specified, the profile produced by this endpoint
// is the total accumulated profile since program start. If a seconds GET parameter
// is specified, this endpoint produces a delta profile, taking the difference of
// two profiles captured immediately, and again at the end of the duration specified
// by seconds.
//
// RegisterHandlers registers each named profile's handler as /debug/pprof/name.
func NewProfileHandler(name string) http.Handler

// CPUProfileHandler is an HTTP handler func that responds with the
// pprof-formatted cpu profile.
//
// Profiling lasts for duration specified in seconds GET parameter, or for
// 30 seconds if not specified.
//
// RegisterHandlers registers this handler as /debug/pprof/cpu. It also registers it
// as /debug/pprof/profile, for compatibility with the v1 package.
func HandleCPUProfile(w http.ResponseWriter, r *http.Request)

// HandleTrace is an HTTP handler func that responds with the execution trace
// in binary form. See the runtime/trace package and the `go tool trace` command
// to see how to use this data.
//
// Tracing lasts for the duration specified in the seconds GET parameter, or for 1 second
// if not specified. Adding the GET parameter cpuprofiling with value some integer
// >0 will also enable CPU profiling for the same duration, writing CPU sample events
// to the trace.
//
// RegisterHandlers registers this handler as /debug/pprof/trace.
func HandleTrace(w http.ResponseWriter, r *http.Request)

// HandleSymbols is an HTTP handler func that looks up the program counters listed in
// the request and responds with a table mapping program counters to function names.
//
// This endpoint is designed to work with the pprof tool.
// 
// POST requests to this endpoint must populate the request body with the list of
// program counters as hex values with the prefix "0x", separated by "+" characters.
// GET requests to this endpoint must specify the same format in the URL's query
// parameters.
//
// The format of the result is a single line indicating the number of symbols
// returned ("num_symbols: N"), followed by one line per program counter passed as input.
// Each line consists of the program counter, followed by a space, followed by the
// function name that program counter maps to.
//
// RegisterHandlers registers this handler as /debug/pprof/symbol.
func HandleSymbols(w http.ResponseWriter, r *http.Request)

Sub-proposal: new handlers for flight recording

I propose adding three new handlers for flight recording. The goal behind these endpoints is to expose the controls afforded by the runtime/trace package directly through a stateful REST API. This is useful for an external program monitor or sidecar to capture a trace snapshot based on externally-available signals. An example is provided in the proposed API documentation.

// HandleFlightRecordingStart is an HTTP handler func that enables flight recording.
//
// The caller must make a POST request to the endpoint, which will enable flight
// recording in the program. It responds with a token that must be used to both capture
// trace data and stop flight recording.
//
// The minimum duration and maximum size of the trace snapshot may be controlled by
// setting the minageseconds and maxbytes query parameters respectively, otherwise the
// values are implementation defined, but can be assumed to capture a few seconds of
// execution.
//
// RegisterHandlers registers this handler as /debug/pprof/flightrecording/start.
func HandleFlightRecordingStart(w http.ResponseWriter, r *http.Request)

// HandleFlightRecordingCapture is an HTTP handler func that produces a trace
// representing the last few seconds of execution.
//
// The caller must make a GET request to the endpoint with the token parameter
// whose value must be a valid token produced by a HandleFlightRecordingStart endpoint.
//
// RegisterHandlers registers this handler as /debug/pprof/flightrecording/capture.
func HandleFlightRecordingCapture(w http.ResponseWriter, r *http.Request)

// HandleFlightRecordingStop is an HTTP handler func that enables flight recording.
//
// The caller must make a POST request to the endpoint with the token parameter
// whose value must be a valid token produced by a HandleFlightRecordingStart endpoint.
// Upon successful completion of the request, the token becomes invalid.
//
// RegisterHandlers registers this handler as /debug/pprof/flightrecording/stop.
func HandleFlightRecordingStop(w http.ResponseWriter, r *http.Request)

Sub-proposal: controlling the CPU profiling rate

I propose the following addition to the API (bold text) to support a customizable CPU profiling rate, as per #57488. Although the runtime.SetCPUProfileRate API is notoriously broken, that doesn't mean we can't orthogonally support this in net/http/pprof.

// CPUProfileHandler is an HTTP handler func that responds with the
// pprof-formatted cpu profile.
//
// Profiling lasts for duration specified in seconds GET parameter, or for
// 30 seconds if not specified. The CPU profiling rate is controlled by
// the rate query parameter, whose value is specified in samples per second.
//
// RegisterHandlers registers this handler as /debug/pprof/cpu. It also registers it
// as /debug/pprof/profile, for compatibility with the v1 package.
func HandleCPUProfile(w http.ResponseWriter, r *http.Request)

Sub-proposal: CPU profiling while tracing

I propose the following addition to the API (bold text) to support enabling CPU profiling simultaneously with a trace, to ensure CPU sample events end up in the trace, as per #66679.

// HandleTrace is an HTTP handler func that responds with the execution trace
// in binary form. See the runtime/trace package and the `go tool trace` command
// to see how to use this data.
//
// Tracing lasts for the duration specified in the seconds GET parameter, or for 1 second
// if not specified. Adding the query parameter cpuprofiling with value some integer
// >0 will also enable CPU profiling for the same duration, writing CPU sample events
// to the trace. The query parameter cpuprofilingrate can be used to control the sample
// rate.
//
// RegisterHandlers registers this handler as /debug/pprof/trace.
func HandleTrace(w http.ResponseWriter, r *http.Request)

Other sub-proposals

Unfortunately, I choose not to address #57765 here because there are still some open questions on whether it's worth it. There's a clear workaround (compute the delta profile yourself) but the argument is mainly about the cost of making an additional request, which may not be compelling enough.

Alternatives considered

Breaking compatibility

One alternative we considered was to simply have net/http/pprof stop installing handlers by default. While tempting, I suspect this untenable. Projects like Tile38 use the implicitly-installed handlers by exposing the default net/http server mux only on the local network. They have already worked around the security issue, and this would only break them. With 31,000+ public importers, while it's possible many programs would automatically become safer, many would also likely break.

The introduction of a new package instead has the downside that it's less likely to fix existing problematic uses of net/http/pprof, but this can likely be mitigated by additional tooling improvements. For example, a modernizer that warns against using net/http/pprof and instead suggests using the v2 package.

[gabyhelp]

Related Issues

• proposal: net/http/pprof: function to register all profiles to a mux #71213
• proposal: new `Recorder` types in runtime/pprof #74545
• net/http/pprof: Index doesn't include Symbol #69897 (closed)
• proposal: net/http/pprof/v2: disallow package to register to the default mux #42834
• proposal: runtime/pprof: extend profile configuration to support perf events #53286
• proposal: net/http/pprof: provide a mechanism to render on /debug/pprof profiles than those that invoke pprof.NewProfile #41324 (closed)
• proposal: net/http/pprof: endpoint for buildinfo #60883

Related Code Changes

• net/http/pprof: update the /debug/pprof endpoint
• net/http/pprof: support the "seconds" param for block, mutex profiles
• doc: make it clear which pprof package is used

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

// RegisterHandlers is a convenience function that installs all the handlers provided by
// this package into the provided ServeMux with the path prefix /debug/pprof.
//
// It also installs a handler for /debug/pprof which responds with an HTML page
// containing a list of all available profiles.
func RegisterHandlers(mux *http.ServeMux)

Small point, but it seems a bit prescriptive to force registration at /debug/pprof/. Maybe this should take the prefix to use?

My initial thought was to provide func NewServeMux() *http.ServeMux, which simply registers everything at root, and is intended to be used like:

http.Handle("/debug/pprof/", pprof.NewServeMux())

Unfortunately, this doesn't work because the mux receives the full path (rather than having /debug/pprof/ stripped off), so the registered paths don't match.

[mknyszek]

I think part of the point of RegisterHandlers for me is that it'll just install the handlers at a well-defined and standard location without you having to make any additional decisions. The goal is to make it as easy as possible to add the new routes wherever they need to go. If you really want to customize it, you can do the slightly more annoying thing of registering each handler individually.

That being said, I don't feel strongly about this. We could alternatively have a global constant prefix that's used in all the examples as a way to try to enforce that standardization, which doesn't seem that bad either.

[database64128]

Currently for Index I have to do something like:

prefix := strings.TrimSuffix(indexPath, "/debug/pprof/")
mux.Handle(indexPath, logPprofRequests(http.StripPrefix(prefix, http.HandlerFunc(pprof.Index))))

It'd be nice if we could have this instead:

mux.Handle(indexPath, logPprofRequests(pprof.NewIndexHandler(indexPath)))

[mknyszek]

@database64128 I considered that, but it didn't seem worth it. There's a strong argument for full convenience (akin to v1's anonymous imports, covered by RegisterHandlers) and a strong argument for exposing all the endpoints for full customization. But what you're doing here is somewhere in between. The Index handler is very barebones, and may not even work properly if you don't happen to get everything to line up correctly. Plus, as you point out, you currently have to jump through hoops to use it. At that point I figure it's better long-term to just generate your own index page, though it does require some boilerplate.

This brings to mind a valid concern about the full customization case, which is that you might miss some handlers, so maybe the proposal needs an iterator over all the handlers? Something like:

// AllHandlers returns an iterator over all available handlers. The iterator produces
// the handler's name and the handler itself. All the names are guaranteed to be unique.
//
// The purpose of this function is to make it easier for customized debug endpoints
// to dynamically discover all the available diagnostics exposed by this package.
func AllHandlers() iter.Seq2[string, http.Handler]

Then it becomes much easier to write a fully customized set of debug handler endpoints.

pathPrefix := "/my/debug/path/prefix"
var handlePaths []string
for name, h := range pprof.AllHandlers() {
    path := pathPrefix+"/"+name
    mux.Handle(path, h)
    handlePaths = append(handlePaths, path)
}
mux.HandleFunc(pathPrefix, func(w http.ResponseWriter, r *http.Request) {
    // Render index page here, generating links for each path in handlePaths.
})

[rittneje]

Unfortunately, this doesn't work because the mux receives the full path (rather than having /debug/pprof/ stripped off), so the registered paths don't match.

@prattmic I believe http.StripPrefix would resolve this.

One alternative we considered was to simply have net/http/pprof stop installing handlers by default. While tempting, I suspect this untenable. Projects like Tile38 use the implicitly-installed handlers by exposing the default net/http server mux only on the local network. They have already worked around the security issue, and this would only break them. With 31,000+ public importers, while it's possible many programs would automatically become safer, many would also likely break.

@mknyszek Would a GODEBUG be acceptable here?

[mknyszek]

@mknyszek Would a GODEBUG be acceptable here?

We also considered that as well. I think it still comes back to this having a really wide blast radius. I forget who, but someone on the team did a quick-and-dirty analysis of the actual uses of net/http/pprof, and >50% of them really are just the anonymous import. It's this, once again coupled with the fact that we don't know who we're breaking that didn't do anything wrong in the first place.

Put another way, unlike other compatibility GODEBUGs, we wouldn't really be fixing "obviously broken" code enough, I think. It sits in this awkward space between "widely used" and "pretty bad but not obviously severe." This proposal takes the approach of removing the footgun for new code, rather than breaking old code, which is a bit on the conservative side.

That being said, if in this discussion we discover strong reasons to prefer a GODEBUG, I'm certainly not opposed, and I can just turn the rest of the new functionality into their own proposals. A v2 package is itself admittedly a dramatic change.