runtime: Crash in GC on Debian riscv64 (qemu) #54104

phpeterson-usf · 2022-07-28T01:05:03Z

What version of Go are you using (`go version`)?

$ go version
go version go1.18.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

I am cross-compiling the micro editor on my Mac targeting a Debian riscv64 guest under QEMU. The Debian guest is Linux debian 5.18.0-2-riscv64 https://github.com/zyedidia/micro/issues/1 SMP Debian 5.18.5-1 (2022-06-16) riscv64 GNU/Linux

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/phil/Library/Caches/go-build"
GOENV="/Users/phil/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/phil/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/phil/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/phil/riscv64-ubuntu/micro/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bl/ncrkxg1d6r130_5pb8mn3y3m0000gn/T/go-build3611809763=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

After cloning the repo, the build command is GOOS=linux GOARCH=riscv64 CGO_ENABLED=0 make build.
The resulting binary is called micro and I scp that to the Debian guest

What did you expect to see?

I expect to run the micro editor

What did you see instead?

The micro editor crashes on launch with the output below, apparently showing a crash in the Go GC. I opened an issue in the micro repo, but the author suggests that the root cause is inside Go.

Interestingly, the same micro binary runs on an Ubuntu guest Linux ubuntu 5.13.0-1026-generic #29~20.04.1-Ubuntu SMP Fri Jun 3 11:55:52 UTC 2022 riscv64 riscv64 riscv64 GNU/Linux but crashes on launch on the Debian guest.

stack trace stack trace

runtime: lfstack.push invalid packing: node=0xffffff8cc2b980 cnt=0x1 packed=0xffff8cc2b9800001 -> node=0xffff8cc2b980

fatal error: lfstack.push
runtime stack:

runtime.throw({0x5966ba, 0xc})

runtime/panic.go:992 +0x58

runtime.(*lfstack).push(0xaf62e8, 0xffffff8cc2b980)

runtime/lfstack.go:30 +0x144

runtime.(*spanSetBlockAlloc).free(...)

runtime/mspanset.go:292

runtime.(*spanSet).reset(0xaef898)

runtime/mspanset.go:265 +0xac

runtime.finishsweep_m()

runtime/mgcsweep.go:260 +0xc8

runtime.gcStart.func1()

runtime/mgc.go:664 +0x1c

runtime.systemstack()

runtime/asm_riscv64.s:133 +0x50
goroutine 1 [running]:

runtime.systemstack_switch()

runtime/asm_riscv64.s:96 +0x8 fp=0xc0004b3720 sp=0xc0004b3718 pc=0x74098

runtime.gcStart({0x0, 0x0, 0x0})

runtime/mgc.go:663 +0x500 fp=0xc0004b3798 sp=0xc0004b3720 pc=0x29188

runtime.mallocgc(0x30, 0x5576e0, 0x1)

runtime/malloc.go:1205 +0x71c fp=0xc0004b3810 sp=0xc0004b3798 pc=0x1bdb4

runtime.growslice(0x5576e0, {0x0, 0x0, 0x0}, 0x1)

runtime/slice.go:278 +0x4e0 fp=0xc0004b3860 sp=0xc0004b3810 pc=0x5e4b8

regexp/syntax.(*compiler).inst(...)

regexp/syntax/compile.go:164

regexp/syntax.(*compiler).init(...)

regexp/syntax/compile.go:83

regexp/syntax.Compile(0xc000131810)

regexp/syntax/compile.go:73 +0x9c fp=0xc0004b3920 sp=0xc0004b3860 pc=0x151214

regexp.compile({0x59c8fd, 0x14}, 0xd4, 0x0)

regexp/regexp.go:180 +0xa4 fp=0xc0004b39b8 sp=0xc0004b3920 pc=0x167c94

regexp.Compile(...)

regexp/regexp.go:135

regexp.MustCompile({0x59c8fd, 0x14})

regexp/regexp.go:315 +0x3c fp=0xc0004b3a38 sp=0xc0004b39b8 pc=0x1688ac

main.LoadInput({0xc00001c310, 0x0, 0x0})

github.com/zyedidia/micro/v2/cmd/micro/micro.go:166 +0x14c fp=0xc0004b3bb8 sp=0xc0004b3a38 pc=0x4af5d4

main.main()

github.com/zyedidia/micro/v2/cmd/micro/micro.go:319 +0x748 fp=0xc0004b3f80 sp=0xc0004b3bb8 pc=0x4b0988

runtime.main()

runtime/proc.go:250 +0x228 fp=0xc0004b3fd8 sp=0xc0004b3f80 pc=0x48480

runtime.goexit()

runtime/asm_riscv64.s:497 +0x4 fp=0xc0004b3fd8 sp=0xc0004b3fd8 pc=0x75ec4
goroutine 6 [sleep]:

time.Sleep(0x1dcd65000)

runtime/time.go:194 +0x150

github.com/zyedidia/micro/v2/internal/buffer.backupThread()

github.com/zyedidia/micro/v2/internal/buffer/backup.go:37 +0x28

created by github.com/zyedidia/micro/v2/internal/buffer.init.0

github.com/zyedidia/micro/v2/internal/buffer/backup.go:52 +0x68
goroutine 8 [syscall]:

os/signal.signal_recv()

runtime/sigqueue.go:151 +0x38

os/signal.loop()

os/signal/signal_unix.go:23 +0x1c

created by os/signal.Notify.func1.1

os/signal/signal.go:151 +0x2c
goroutine 9 [select]:

github.com/zyedidia/tcell/v2.(*tScreen).mainLoop(0xc000190000)

github.com/zyedidia/tcell/[email protected]/tscreen.go:1580 +0xe0

created by github.com/zyedidia/tcell/v2.(*tScreen).Init

github.com/zyedidia/tcell/[email protected]/tscreen.go:210 +0x6cc
goroutine 10 [IO wait]:

internal/poll.runtime_pollWait(0xffffff8cc35198, 0x72)

runtime/netpoll.go:302 +0xe0

internal/poll.(*pollDesc).wait(0xc000058e58, 0x72, 0x1)

internal/poll/fd_poll_runtime.go:83 +0x34

internal/poll.(*pollDesc).waitRead(...)

internal/poll/fd_poll_runtime.go:88

internal/poll.(*FD).Read(0xc000058e40, {0xc0003fb000, 0x1000, 0x1000})

internal/poll/fd_unix.go:167 +0x1f0

os.(*File).read(...)

os/file_posix.go:31

os.(*File).Read(0xc00000c780, {0xc0003fb000, 0x1000, 0x1000})

os/file.go:119 +0x70

github.com/zyedidia/tcell/v2.(*tScreen).inputLoop(0xc000190000)

github.com/zyedidia/tcell/[email protected]/tscreen.go:1633 +0x94

created by github.com/zyedidia/tcell/v2.(*tScreen).Init

github.com/zyedidia/tcell/[email protected]/tscreen.go:211 +0x724

The text was updated successfully, but these errors were encountered:

mengzhuo · 2022-07-28T01:22:11Z

It will be fixed after this CL been merged.
https://go-review.googlesource.com/c/go/+/409055

phpeterson-usf · 2022-07-28T01:40:42Z

Thanks. I would be thrilled if this could make the 1.19 release train to support my teaching objectives.

cherrymui · 2022-07-28T14:09:41Z

Can this reproduce on a real hardware?

For QEMU, I guess it would be possible to configure it to not enable SV57 mode?

phpeterson-usf · 2022-07-28T14:39:34Z

I believe SV57 is enabled in the 5.18 kernel which is distributed with Debian/riscv64. I have looked unsuccessfully for runtime configuration for that feature. If there's a way to change that in QEMU, I don't know what it is.

I don't have an Unmatched board, and SiFive discontinued them in favor of a pre-release collaboration with Intel.

If it's too late for 1.19, the two possible workarounds I see are:

Downgrade the kernel to pre-SV57
Build Go from source, patching in Ian's change myself

phpeterson-usf · 2022-07-28T16:34:13Z

FWIW, I built from go1.19rc2 + Ian's change (e9f157e.diff) and cross-compiled micro to avoid bootstrapping Go on riscv64. The resulting micro binary seems to work fine on my Debian/riscv64 guest.

mengzhuo · 2022-07-29T02:31:15Z

Can this reproduce on a real hardware?

For QEMU, I guess it would be possible to configure it to not enable SV57 mode?

Most of existed real hardwares (that I can get) using sv39

Sifive unmatched(Go build farm): sv39
Starfive v1: sv39
Allwinner D1: sv39

Qemu configured with -cpu=sifive-u54 to disable SV57.

yuzibo · 2022-07-29T07:07:06Z

Hi,
I just have Unmatched boards by hand and I tested it easily:

vimer@unmatched:~/go/proj/bin$ uname -a
Linux unmatched 5.18.0-2-riscv64 #1 SMP Debian 5.18.5-1 (2022-06-16) riscv64 GNU/Linux
vimer@unmatched:~/go/proj/bin$ go env
GO111MODULE=""
GOARCH="riscv64"
GOBIN=""
GOCACHE="/home/vimer/.cache/go-build"
GOENV="/home/vimer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="riscv64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/vimer/go/proj/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/vimer/go/proj"
GOPRIVATE=""
GOPROXY="https://proxy.golang.com.cn"
GOROOT="/home/vimer/go/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/vimer/go/go/pkg/tool/linux_riscv64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2096220389=/tmp/go-build -gno-record-gcc-switches"

vimer@unmatched:~/go/proj/bin$ ./micro --version
Version: 2.0.11-dev.100
Commit hash: 225927b9
Compiled on July 29, 2022

vimer@unmatched:~/go/proj/bin$ ip a | ./micro

The micro editor runs fine in my view also.
Please let me know if there are any issues.

gopherbot · 2022-07-29T11:13:19Z

Change https://go.dev/cl/409055 mentions this issue: runtime: support riscv64 SV57 mode

riscv64 has SV57 mode when user-space VA is 56 bits. Linux kernel recently got support for this mode and Go binaries started crashing as: runtime: lfstack.push invalid packing: node=0xffffff5908a940 cnt=0x1 packed=0xffff5908a9400001 -> node=0xffff5908a940 Adjust lfstack code to use only 8 top bits of pointers on riscv64. For context see: https://groups.google.com/g/syzkaller-bugs/c/lU0GQTZoNQQ/m/O_c3vmE3AAAJ Update #54104 Change-Id: Ib5d3d6a79c0c6eddf11618d73fcc8bc1832a9c25 Reviewed-on: https://go-review.googlesource.com/c/go/+/409055 Reviewed-by: Joel Sing <[email protected]> Reviewed-by: Meng Zhuo <[email protected]> Reviewed-by: Michael Knyszek <[email protected]> Reviewed-by: Cherry Mui <[email protected]>

riscv64 has SV57 mode when user-space VA is 56 bits. Linux kernel recently got support for this mode and Go binaries started crashing as: runtime: lfstack.push invalid packing: node=0xffffff5908a940 cnt=0x1 packed=0xffff5908a9400001 -> node=0xffff5908a940 Adjust lfstack code to use only 8 top bits of pointers on riscv64. For context see: https://groups.google.com/g/syzkaller-bugs/c/lU0GQTZoNQQ/m/O_c3vmE3AAAJ Update golang#54104 Change-Id: Ib5d3d6a79c0c6eddf11618d73fcc8bc1832a9c25 Reviewed-on: https://go-review.googlesource.com/c/go/+/409055 Reviewed-by: Joel Sing <[email protected]> Reviewed-by: Meng Zhuo <[email protected]> Reviewed-by: Michael Knyszek <[email protected]> Reviewed-by: Cherry Mui <[email protected]>

4a6f656c · 2022-09-17T07:04:59Z

This should be fixed via https://go.dev/cl/409055, closing.

phpeterson-usf mentioned this issue Jul 28, 2022

Cross compiled for riscv64 crashes on Debian zyedidia/micro#2514

Closed

mengzhuo changed the title ~~affected/package: Crash in GC on Debian riscv64~~ runtime: Crash in GC on Debian riscv64 (qemu) Jul 28, 2022

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 28, 2022

mengzhuo added arch-riscv Issues solely affecting the riscv64 architecture. compiler/runtime Issues related to the Go compiler and/or runtime. and removed compiler/runtime Issues related to the Go compiler and/or runtime. labels Jul 28, 2022

cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 28, 2022

cherrymui added this to the Backlog milestone Jul 28, 2022

mknyszek added this to Go Compiler / Runtime Aug 3, 2022

mknyszek self-assigned this Aug 3, 2022

mknyszek moved this to All-But-Submitted in Go Compiler / Runtime Aug 3, 2022

4a6f656c closed this as completed Sep 17, 2022

Repository owner moved this from All-But-Submitted to Done in Go Compiler / Runtime Sep 17, 2022

mknyszek removed this from Go Compiler / Runtime Apr 19, 2023

MaxTheMooshroom mentioned this issue Jul 6, 2023

mediamtx crashes after a variable amount of time running nucleus-labs/doorbellian-docker#4

Closed

golang locked and limited conversation to collaborators Sep 17, 2023

gopherbot added the FrozenDueToAge label Sep 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: Crash in GC on Debian riscv64 (qemu) #54104

runtime: Crash in GC on Debian riscv64 (qemu) #54104

phpeterson-usf commented Jul 28, 2022

mengzhuo commented Jul 28, 2022

phpeterson-usf commented Jul 28, 2022

cherrymui commented Jul 28, 2022

phpeterson-usf commented Jul 28, 2022 •

edited

Loading

phpeterson-usf commented Jul 28, 2022

mengzhuo commented Jul 29, 2022 •

edited

Loading

yuzibo commented Jul 29, 2022

gopherbot commented Jul 29, 2022

4a6f656c commented Sep 17, 2022

runtime: Crash in GC on Debian riscv64 (qemu) #54104

runtime: Crash in GC on Debian riscv64 (qemu) #54104

Comments

phpeterson-usf commented Jul 28, 2022

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

mengzhuo commented Jul 28, 2022

phpeterson-usf commented Jul 28, 2022

cherrymui commented Jul 28, 2022

phpeterson-usf commented Jul 28, 2022 • edited Loading

phpeterson-usf commented Jul 28, 2022

mengzhuo commented Jul 29, 2022 • edited Loading

yuzibo commented Jul 29, 2022

gopherbot commented Jul 29, 2022

4a6f656c commented Sep 17, 2022

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

phpeterson-usf commented Jul 28, 2022 •

edited

Loading

mengzhuo commented Jul 29, 2022 •

edited

Loading