Resolution of symbols by the server #1009

GabrielSiliceum · 2025-03-20T14:31:55Z

The main features in this pull request is to make the server be able to locally resolve the symbol and serialize them in trace file.
For this the client sends his image data to the server.

Why this can be a improvement

The client has less work to do.
You can resolve symbols in the profiler even if the client machine doesn't have the files containing debug information
Less bandwidth used for communication between client and server over time (except at the begin of the recording).
Working on unification of the symbols resolution code.
By disabling automatic symbol resolution entirely, this can speed up the capture time and size.

Implementation details

In the global settings you can specify symbol resolution behaviour (Attempt profiler resolution by profiler, Prevent resolution by application ).
Tracy_callstack is now used by Tracy server (in the common folder).
Tracy_callstack can use debug GUIDs sent by the client to locate and load symbol information.
Only working for Windows right now
- Designed to work for Linux/mac later on
- On linux, we already store the GNU build id, so this will be forward compatible
Client caches its image information and sends it to the server after the handshake. If new images are loaded later on, a symbol query will be issued (as before) leading to discovery of the module and its debug id will be sent to the server.

Areas of improvements/TODO list

Make the resolution of symbols on the server asynchronous (similarly to the symbol worker of the client).
Attempt to load debug information using a user-provided path.
Provide a way for the user to specify the symbols location from the UI (we can with this make the server totally independent of the client for image caching).
ELF/Macho-o resolution by modifying libbacktrace to support resolution of another process. (Apparently not that many modifications required)
Rework the content of Tracy_callstack.cpp to provide an API without global state (ie: some "context"/"state" that we could store in the Worker instead of relying on a global cache)
Offline symbol resolution (the update program) should use the same mechanism.
- For now it only uses the image path, never the GUID. This means that you could in theory end up using incorrect symbols (if program was recompiled, OS kernel modules updated, etc).
- It uses addr2line which is slow and should only serve as a fallback mechanism, after having confirmed the path to the executable matches the one from the capture

The main idea for this feature is to make Tracy more flexible, especially when you can't or don't want the application make the symbol resolution job.

…ue by default

wolfpld · 2025-03-20T17:27:44Z

How does the server-side symbol resolution work? Let's say I'm running the server on Linux and the client is a Windows application.

Lectem · 2025-03-20T18:08:43Z

For now it will simply fallback to querying the client for the symbols.
We rely on the debug guid (type is known and sent to the server) to ensure we are using the correct symbol file (and also possibly locate it, using symbol servers).

The idea for the first step is to only allow to resolve symbols on the server when the client has a similar executable format (ELF, Macho, PE with pdb). Which means mostly windows ciuld resolve windows, Linux can resolve Linux+Android+... And Macos could resolve macos/ios.

In the future however, I'd like to make it possible to resolve GNU binaries (first elf, macho, perhaps later PE files with mingw) using a windows server too. (The other way is harder since it would require something to replace DbgHelp)
This seems feasible by modifying libbacktrace (or using another library if you're up to it).

Lectem · 2025-03-20T18:13:53Z

I also just saw that the CI build is failing, seemingly due to the enum using an underlying type. (weirdly we didn't get this while building on our private CI, probably missed some executable that does not enable c++11)

I'll fix this tomorrow at work, but would like to check with you what the minimal supported version of C++ we should be using (ie: should we not use such enums or should we fix the projects to target c++11?)

Edit: actually after reading the error log again, it could just be that were missing the <stdint.h> include

wolfpld · 2025-04-06T12:46:21Z

profiler/src/profiler/TracySourceView.cpp

@@ -1157,7 +1157,8 @@ void SourceView::RenderSymbolView( Worker& worker, View& view )

    const auto shortenName = view.GetShortenName();
    auto sym = worker.GetSymbolData( m_symAddr );
-    assert( sym );
+    if( sym == nullptr ) return; // Might no have received symbol info yet


User should be informed why there is nothing displayed.

wolfpld · 2025-04-06T12:49:11Z

profiler/src/profiler/TracyView.cpp

@@ -36,7 +36,7 @@ namespace tracy
 double s_time = 0;

 View::View( void(*cbMainThread)(const std::function<void()>&, bool), const char* addr, uint16_t port, ImFont* fixedWidth, ImFont* smallFont, ImFont* bigFont, SetTitleCallback stcb, SetScaleCallback sscb, AttentionCallback acb, const Config& config, AchievementsMgr* amgr )
-    : m_worker( addr, port, config.memoryLimit == 0 ? -1 : ( config.memoryLimitPercent * tracy::GetPhysicalMemorySize() / 100 ) )
+    : m_worker( addr, port, config.memoryLimit == 0 ? -1 : ( config.memoryLimitPercent * tracy::GetPhysicalMemorySize() / 100 ), Worker::SymbolResolutionConfig{ config.symbolsAttemptResolutionByServer, config.symbolsPreventResolutionByClient } )


It feels that the data structure being created here should just accept the config and extract what it needs on its own.

wolfpld · 2025-04-06T12:51:12Z

capture/src/capture.cpp

-    tracy::Worker worker( address, port, memoryLimit );
+
+    tracy::Worker::SymbolResolutionConfig symConfig{};
+    symConfig.m_attemptResolutionByWorker = false;


What is the reason for setting this explicitly?

wolfpld · 2025-04-06T12:55:58Z

public/tracy/Tracy.hpp

@@ -13,7 +13,7 @@
 #endif

 #ifndef TracyLine
-#  define TracyLine __LINE__
+#  define TracyLine TracyConcat(__LINE__,U) // MSVC Edit and continue __LINE__ is non-constant. See https://developercommunity.visualstudio.com/t/-line-cannot-be-used-as-an-argument-for-constexpr/195665


What does this do?

wolfpld · 2025-04-06T12:57:48Z

public/TracyClient.cpp

 #ifdef _MSC_VER
 #  pragma comment(lib, "ws2_32.lib")
-#  pragma comment(lib, "dbghelp.lib")


How are all these removals handled now? They are still needed, aren't they?

Lectem and others added 24 commits March 20, 2025 13:32

Ignore example build files

3853354

Fix D3D11 issues when TRACY_CALLSTACK==0 (default for examples)

8a9e92a

[Add] server callstack solver

286ebd2

Move TracyCallstack (and dependencies) to common/ folder

7ef3fe3

Fix module info init and free. Clean InitCallstack

e8e60d4

Remove s_clientSendImageInfo for now and make s_serverLocalResolve tr…

ce22a0e

…ue by default

IsEnv now supports default value

1444ad3

Now fallback correctly when symbol is not found

2a74e0c

Unify image caches across platforms

901e826

Symbol resolution configuration + update versions

361618b

Clean formating and remove unneeded std usage

e51f755

Fix lifetime of callstack symbol resolver (Init/EndCallstack)

595188e

Always save image infos so that we may resolve symbols later on

8762675

Fix TryResolveCallstack when worker does not allow to query symbols

49880f9

Warn once about symsrv if symbol resolution fails

140a33d

Resolve Symbols button

43faca0

Self Profiling working

3f9e2a9

Update tracy's document for Server symbol resolution

24bb230

Fix grammar issue

b185d37

WriteDebugFieldToPacket is now a static function

c6f8487

modify some comment

676ed7a

Add get build_id while dl_iterate_phdr

62f0f59

Add Sending Image Data from backend when found new image

2992339

fix decodeCallStack flag for linux

Loading
Loading status checks…

b0fb501

Fix build (missing stdint.h include)

073df4e

wolfpld reviewed Apr 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolution of symbols by the server #1009

Resolution of symbols by the server #1009

GabrielSiliceum commented Mar 20, 2025

wolfpld commented Mar 20, 2025

Lectem commented Mar 20, 2025

Lectem commented Mar 20, 2025 •

edited

Loading

wolfpld Apr 6, 2025

wolfpld Apr 6, 2025

wolfpld Apr 6, 2025

wolfpld Apr 6, 2025

wolfpld Apr 6, 2025

Resolution of symbols by the server #1009

Are you sure you want to change the base?

Resolution of symbols by the server #1009

Conversation

GabrielSiliceum commented Mar 20, 2025

Why this can be a improvement

Implementation details

Areas of improvements/TODO list

wolfpld commented Mar 20, 2025

Lectem commented Mar 20, 2025

Lectem commented Mar 20, 2025 • edited Loading

wolfpld Apr 6, 2025

Choose a reason for hiding this comment

wolfpld Apr 6, 2025

Choose a reason for hiding this comment

wolfpld Apr 6, 2025

Choose a reason for hiding this comment

wolfpld Apr 6, 2025

Choose a reason for hiding this comment

wolfpld Apr 6, 2025

Choose a reason for hiding this comment

Lectem commented Mar 20, 2025 •

edited

Loading