Pyroscope garbles application logs

[samthebest]

Describe the bug

We have some log entries like this:

2025-03-22 20:23:30.174{"level":"INFO","ts":"1742675010174","deviceID":"1234","message":"Bla bla"}
 [ERROR] Error uploading snapshot: 422 {"code":"unknown","message":"pushing IngestInput-pprof failed unknown: at least 2 live replicas required, could only find 1"}

This should be two log entries like this:

{"level":"INFO","ts":"1742675010174","deviceID":"1234","message":"Bla bla"}
2025-03-22 20:23:30.174 [ERROR] Error uploading snapshot: 422 {"code":"unknown","message":"pushing IngestInput-pprof failed unknown: at least 2 live replicas required, could only find 1"}

As you can see they are being interleaved. The second log line comes from pyroscope, the former from our application.

Unfortunately I'm not fully familiar with our infrastructure setup, especially pyroscope & grafana, and it would take me a few weeks of archeology to figure out answers to basic questions such as "how do we deploy pyroscope", etc.

All I really know is that the application is a JVM process, and we aggregate logs into Loki.

So I was just hoping that someone might be able to point me quickly to some documentation or a known issue where pyroscope is interleaving it's log lines with the JVM application. Does this problem have a solution? Can pyroscope be told to write it's logs to a different output? If so how?

[simonswine]

The only thing you can control right now is the log level, so an environment variable won't solve you issue completely: PYROSCOPE_LOG_LEVEL=error. See also https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/java/#configuration-options.

Customizable logger were requested before, but nothing has been implemented: grafana/pyroscope-java#126

By the sounds of it you would be happy with logging to a file or disabling logs all together?

[samthebest]

Thanks @simonswine, yes being able to log to a file would work!