[Bug]: macOS agent eventually starts sending malformed data

[kode54]

Description

I was running the macOS agent on two machines, and one of them randomly started sending malformed data blocks. I connected an SSH client to it to see what it was sending, and it was still live, but suddenly, the 'container' object was missing a double quote character.

Expected Behavior

The agent should continue sending data.

Steps to Reproduce

1. Start an agent on macOS Apple Silicon
2. Leave it running for several hours

OS / Architecture

darwin/arm64 (agent) / linux/amd64 (hub)

Beszel version

0.9.1

Installation method

Binary

Configuration

Hub Logs

From the hub logs:


id 	gd4bp2903rtotc3
level 	ERROR (8)
created 	2025-02-06 11:31:59.654 UTC
message 	Failed to get system stats:
data.err 	invalid character '\u0000' looking for beginning of value


I substituted a \u0000 for the actual null character that was shown in the logs.

Agent Logs

The agent is sending this malformed data:


{"stats":{"cpu":5.75,"m":24,"mu":15.15,"mp":63.14,"mb":8.44,"d":926.35,"du":565.55,"dp":61.05,"dr":0.79,"dw":0.01,"ns":0.02,"nr":0.03,"t":{"NAND CH0 temp":39,"PMU tcal":51.82,"PMU tdev1":44.26,"PMU tdev3":45.07,"PMU tdev4":43.86,"PMU tdev5":46.01,"PMU tdev6":41.94,"PMU tdev7":45.07,"PMU tdev8":43.99,"PMU tdie1":43.89,"PMU tdie10":43.49,"PMU tdie11":44.77,"PMU tdie12":45.01,"PMU tdie13":44.45,"PMU tdie14":44.77,"PMU tdie2":43.73,"PMU tdie3":44.21,"PMU tdie4":44.05,"PMU tdie5":44.77,"PMU tdie6":45.4,"PMU tdie7":45.8,"PMU tdie8":45.09,"PMU tdie9":44.93}},"info":{"h":"ChristohersMini.local","k":"\u00004.3.0","c":14,"t":14,"m":"Apple M4 Pro","u":806937,"cpu":5.75,"mp":63.14,"dp":61.05,"b":0.05,"v":"0.9.1"},container":null}


I see the kernel version string also suddenly started with a null character instead of the `2` it should start with.

The correct info being returned now after restarting it:


{"stats":{"cpu":4.81,"m":24,"mu":15.2,"mp":63.33,"mb":8.13,"d":926.35,"du":565.64,"dp":61.06,"dr":0.02,"dw":0.56,"ns":0.01,"nr":0.01,"t":{"NAND CH0 temp":39,"PMU tcal":51.82,"PMU tdev1":43.72,"PMU tdev3":44.76,"PMU tdev4":43.15,"PMU tdev5":45.72,"PMU tdev6":41.59,"PMU tdev7":44.49,"PMU tdev8":43.33,"PMU tdie1":43.33,"PMU tdie10":42.61,"PMU tdie11":44.05,"PMU tdie12":44.13,"PMU tdie13":43.73,"PMU tdie14":44.05,"PMU tdie2":43.65,"PMU tdie3":43.65,"PMU tdie4":43.73,"PMU tdie5":44.13,"PMU tdie6":45.09,"PMU tdie7":44.93,"PMU tdie8":44.61,"PMU tdie9":43.25}},"info":{"h":"ChristohersMini.local","k":"24.3.0","c":14,"t":14,"m":"Apple M4 Pro","u":808093,"cpu":4.81,"mp":63.33,"dp":61.06,"b":0.02,"v":"0.9.1"},"container":null}

[henrygd]

I'll look into it. Thanks for verifying that it's a problem on the agent side.

If you're still running the agent and it happens again, please check if the agent logs Error encoding stats with the specific error.

[kode54]

Nearest I can tell, the only thing I did unique to this copy of the agent was, I stupidly thought I needed to run:

codesign -s - beszel-agent

on it, but it was failing to load for other reasons than the signature.

[kode54]

Hmm, now the other Mac is doing it:

{"stats":{"cpu":20.8,"m":16,"mu":9.71,"mp":60.71,"mb":5.31,"d":228.27,"du":180.78,"dp":79.19,"dr":0.02,"dw":0,"ns":0,"nr":0,"t":{"ANE MTR Temp Sensor1":30,"GPU MTR Temp Sensor1":30,"GPU MTR Temp Sensor4":30,"ISP MTR Temp Sensor5":30,"NAND CH0 temp":30,"PMGR SOC Die Temp Sensor0":31.73,"PMGR SOC Die Temp Sensor1":32.02,"PMGR SOC Die Temp Sensor2":32.64,"PMU TP3w":37.94,"PMU tcal":51.85,"PMU tdev1":31.98,"PMU tdev2":33.55,"PMU tdev3":31.8,"PMU tdev6":31.19,"PMU tdev7":31.63,"PMU tdev8":33.03,"PMU tdie1":38.69,"PMU tdie2":37.4,"PMU tdie4":38.47,"PMU tdie5":37.94,"PMU tdie6":37.29,"PMU tdie7":38.79,"PMU tdie8":36.65,"PMU2 tcal":51.85,"PMU2 tdev1":32.24,"PMU2 tdev2":30.31,"PMU2 tdev3":31.01,"PMU2 tdev4":30.84,"PMU2 tdev5":31.89,"PMU2 tdie1":38.69,"PMU2 tdie2":38.69,"PMU2 tdie3":36,"PMU2 tdie4":38.26,"PMU2 tdie5":37.83,"PMU2 tdie6":37.83,"PMU2 tdie7":37.51,"PMU2 tdie8":38.47,"SOC MTR Temp Sensor0":29.8,"SOC MTR Temp Sensor1":31.17,"SOC MTR Temp Sensor2":31.88,"eACC MTR Temp Sensor0":29.92,"eACC MTR Temp Sensor3":30.27,"pACC MTR Temp Sensor2":36.17,"pACC MTR Temp Sensor3":36.06,"pACC MTR Temp Sensor4":36.8,"pACC MTR Temp Sensor5":34.67,"pACC MTR Temp Sensor7":33.23,"pACC MTR Temp Sensor8":33.53,"pACC MTR Temp Sensor9":32.83}},"info":{"h":"Spencers-Mini.local","k":"24.3.0","c":8,"t":8,"m":"\u0000pple M1","u":1054549,"cpu":20.8,"mp":60.71,"dp":79.19,"b":0,"v":"0.9.1"},container":null}

"m": "\u0000pple M1"

,container"

Edit: No messages logged from stdout or stderr to indicate the problem.

[kode54]

Same issue has kept occurring, so I took a different approach:

1. Cloned the Beszel repo
2. Installed Go from Homebrew (it's 1.24.0)
3. Built the agent myself
4. Installed the ad hoc signed binaries in three machines

So far, it's been okay for over 12 hours, but time will tell if it starts clobbering its strings again.

[henrygd]

Thanks for the update. If it continues to work then it may be a problem with cross compilation / Cgo.

[DoidoYo]

@kode54 , did that solve the problem? I’m experiencing the same thing

[kode54]

It has not solved it. It just happens at different times. I don't know what's causing it.

[luckman212]

I think I might be hitting this problem as well. M4 Mac Mini running 0.10.2 (just updated to 0.11, not sure if anything changed there).

The Agent eventually goes "Offline" in my Hub, even though the brew service is still running. Restarting the agent service gets it back online immediately. Perhaps due to this malformed data? Or is that a different issue altogether?

I set my LOG_LEVEL=debug for now and will keep posted after 0.11 to see if anything's new.

Is there any way to disable docker stats collection completely? I see 2025/04/28 09:35:53 DEBUG Docker stats err="Get \"http://localhost/containers/json\": dial unix /var/run/docker.sock: connect: no such file or directory" in my logs, and although I use OrbStack from time to time, I don't really care to collect these stats on macOS and would rather not see these errors in the logs.

[henrygd]

@luckman212 Just added that in 0.11.1. Setting DOCKER_HOST="" will skip attempting to connect to the Docker daemon.

The disconnecting is likely due to this issue or the throttling issue mentioned in this discussion: #276

If it disconnects consistently, please try lowering the agent's process priority with renice:

renice -n 10 <pid>

I'll check if there's a way to change the default nice value via Homebrew.

[luckman212]

The problem occurred again last night at 12:03am local time. The macOS agent went "offline" (even though according to brew services it's still up and running). This is with v0.11.1.

I also see in the local beszel logs (I have LOG_LEVEL=debug) that the agent is running and logging data.

However, one odd thing I noticed, in the JSON that is shown in the logs, there is a \x00 character inserted before the key name for sensorKey, not sure that is supposed to be there? (which started at 7:24pm last night)

e.g.

[luckman212]

Yep, the agent is definitely outputting invalid JSON

Here's the raw JSON that is produced (scroll all the way right...)

{"stats":{"cpu":5.39,"m":48,"mu":19.09,"mp":39.77,"mb":17.65,"d":1858.19,"du":428.56,"dp":23.06,"dr":0.01,"dw":0.49,"ns":0.02,"nr":0.03,"t":{"NAND CH0 temp":42,"PMU tcal":51.82,"PMU tdev1":45.76,"PMU tdev3":51.16,"PMU tdev4":45.96,"PMU tdev5":51.94,"PMU tdev6":45.96,"PMU tdev7":51.94,"PMU tdev8":47.22,"PMU tdie1":49.16,"PMU tdie10":47.24,"PMU tdie11":48.12,"PMU tdie12":48.44,"PMU tdie13":47.88,"PMU tdie14":47.96,"PMU tdie2":48.28,"PMU tdie3":47.64,"PMU tdie4":48.2,"PMU tdie5":48.84,"PMU tdie6":50.2,"PMU tdie7":49.48,"PMU tdie8":49,"PMU tdie9":47.48}},"info":{"h":"mini4","k":"15.4.1","c":14,"t":14,"m":"Apple M4 Pro","u":219921,"cpu":5.39,"mp":39.77,"dp":23.06,"b":0.05,"v":"0.11.1","dt":51.93548583984375,"os":1},�container":null}