runtime/pprof: compatibility guarantees for pgo file format

[zamazan4ik]

Hi!

We are evaluating using Profile-Guided Optimization (PGO) with Go. Right now the Go compiler stores the profile as a file, that later can be reused. However, I cannot find the information about the backward/forward compatibility guarantees of the pprof format (the format used by the Go compiler) and how the Go compiler deals with it. For us, it's important to know because right now we cannot understand whether will it be possible to use PGO profiles from the older compiler with a newer Go version or not (and vice versa).

In LLVM PGO infrastructure we already met some problems with that - the Clang/LLVM documentation also hasn't this information yet. We want to avoid such problems with Go in the future. A similar situation is with GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112717

The original issue was created here: google/pprof#821 but originally I chose the wrong place for that.

@aalexand friendly ping as you asked :)

[prattmic]

Thanks for filing this issue. I thought I had documented this on https://go.dev/doc/pgo, but it looks like I have not and thus need to do so.

With regards to your questions:

Forward compatibility: I'll define this as meaning a profile collected with Go 1.N is usable by the Go 1.N+1 toolchain.

Yes, it is absolutely our intention that profiles continue to be valid across upgrade of the Go toolchain. This is an important component of our "AutoFDO"/source stability approach to PGO. Just as changes to source code or dependency versions are OK with older profiles, modulo some degradation in optimizations where hot code has changed, upgrading the Go toolchain should be OK as well.

Of course, just as with source code, there may be some degradation in optimizations when using a profile across the upgrade. There are a few ways for this to manifest:

• Upgrading the toolchain also upgrades the standard library, bringing in many source code changes. This is the same as any other package dependency; we may not be able to match samples in the profile to parts of the changed source code.
• If we implement basic block level optimizations (on the roadmap) changes to optimization passes may cause identical functions to generate different basic blocks, which could make aligning with the profile difficult. We have not implemented any such optimizations yet, but in these cases it may be necessary to gracefully degrade and not perform such optimizations across versions.
• Similarly, unnamed functions (like closures) have symbol names generated by the compiler (e.g., pkg.Func.func1 for a closure in pkg.Func). If this naming changes across versions (some discussion in cmd/compile: closure func name changes when inlining #60324), then again we may not properly match functions to samples in the profile.

Backwards compatibility: I'll define this as meaning a profile collected with Go 1.N is usable by the Go 1.N-1 toolchain.

This is something we haven't considered much, nor am I sure how important it is, but I can imagine some use cases for wanting this (notably when upgrading a set of services but perhaps not all at once). This is something that I don't expect to break often, as newer versions are likely to simply add new information to the profile, which older versions would ignore. However, a larger shift in format could break things, and I'm not sure if we should have a guarantee here.

For both forward and backward compatibility, I don't think we've decided how long compatibility should be guaranteed. I'm not sure that using a 1.21 profile with 1.99 is particularly useful, as there have probably been massive code changes. I don't think we can outright reject an old profile (that could be considered breaking the Go 1 compatibility promise to keep old code building), but we could detect it as too old and simply not apply any PGO optimizations.

[prattmic]

cc @cherrymui

[prattmic]

Oh, since you asked about pprof itself: the pprof format itself has a very long compatibility horizon (I'm not sure it have ever made an incompatible change). So I don't think the format itself has much bearing on PGO forward or backward compatibility. That has more to do with the data we put in the file or if we were to change file formats altogether.

[aalexand]

For the PGO format compatibility, maybe https://go.dev/doc/devel/release policy could be considered?

Each major Go release is supported until there are two newer major releases. For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release. We fix critical problems, including critical security problems, in supported releases as needed by issuing minor revisions (for example, Go 1.6.1, Go 1.6.2, and so on).

In other words, if 1.21 is the current major version (like now), then profiles from 1.20 and 1.21 should mix and match.

[zamazan4ik]

@prattmic thank you a lot for the so brief answer!

My main concern about PGO profile re-usage across different Go versions is profile sensitivity to the performed optimizations by the compiler. Right now the Go compiler implements only a few PGO optimizations but this definitely will be changed in the future (at least I hope so!). And since PGO profiles are sensitive to the generated code, each change in the code generation could affect the PGO profiles matching with a newer compiler version. The same version was discussed for Clang here.

Do you plan to implement some kind of mitigation against this? Because if I am reading https://go.dev/doc/pgo#source-stability correctly, the current approach has no built-in mechanisms to mitigate it (I could be wrong since I have no insights about the PGO implementation in the Go compiler yet). Over time, when the PGO-related optimizations will be stabilized in the Go compiler, this probably won't be a huge issue (until you don't tweak a lot of internal optimization constants, reshuffle aggressively optimizations passes between the compiler versions, etc) but for now, when Go's PGO infra is fast-evolving, it could be a problem.

Also, there is a related question about comparing different profiles with each other. Is there something like llvm-profdata overlap (docs) functionality but for PGO profiles? It's useful to have since with this kind of tooling we could easily estimate the changes between profiles that could be introduced by changing the compiler optimization options.

[prattmic]

Do you plan to implement some kind of mitigation against this? Because if I am reading https://go.dev/doc/pgo#source-stability correctly, the current approach has no built-in mechanisms to mitigate it (I could be wrong since I have no insights about the PGO implementation in the Go compiler yet).

Today profile frames include both real and inlined frames, and when doing PGO analysis, the compiler treats frames identically whether or not they were inlined. This is a mitigation against sensitivity to whether the previous build inlined a function or not. So I would say we have some mitigation today.

More intricate optimizations such as basic block level optimizations have not yet been designed or implemented, so I can't say that there are or aren't mitigations. Trying to maintain fidelity across versions will certainly get thought in the design, but I don't think we will have a guarantee of perfect fidelity.

Also, there is a related question about comparing different profiles with each other. Is there something like llvm-profdata overlap (docs) functionality but for PGO profiles?

The pprof CLI provides some similar functionality, like pprof -diff_base, though not exactly this functionality. I do believe it would be fairly easy to build an equivalent tool using the github.com/google/pprof/profile package. Along these lines, we've also considered having the compiler report a "match" metric. i.e., a single metric that reports how much of the weights in the profile could be successfully matched to the source code being compiled.