cmd/compile: fatal error: runtime: cannot map pages in arena address space #23222
Labels
FrozenDueToAge
NeedsDecision
Feedback is required from experts, contributors, and/or the community before a change can be made.
release-blocker
Milestone
Comments
|
Does this also happen with git master, soon to be go1.10? |
titanous
added
the
WaitingForInfo
|
We've received a similar crash from a user running Linux (ethereum/go-ethereum#15743), so it seems to be unrelated to Windows itself and may be a general fault on 386. I'll have to check latest master. |
This works fine for me for recent version of Go: Alex |
|
Go 1.10beta1 blows up on AppVeyor the same way https://ci.appveyor.com/project/ethereum/go-ethereum/build/master.4840/job/y99c2g1ywdho2d34#L264 I'll have to try somehow a master build on AppVeyor to see if it's still present on latest code. |
|
Managed to reliably reproduce it locally with Go 1.10, both beta1 and latest master. The invocation is a bit different than my repro for Go1.9.2: Go 1.10beta1, prebuilt from the website: Go master, built locally: |
|
@karalabe, can you confirm that adding |
|
Confirmed, |
bradfitz
added
NeedsDecision
|
So, I guess go/types is just memory hungry and that hurts on 386's small address space. Presumably we'd have the same problem on arm and other 32-bit platforms. Optimizing go/types at this point seems too much too late. @griesemer, @alandonovan ... ? Or is the GC failing us here? @aclements. Maybe just detect this runtime error from vet and ignore it, as if vet didn't run? @ianlancetaylor, @rsc. |
|
The particular error code is odd. 487 is ERROR_INVALID_ADDRESS, which is not what I'd expect if we just ran out of memory. @alexbrainman, any idea why VirtualAlloc(MEM_COMMIT) would return that, especially given that we must have just successfully called VirutalAlloc(MEM_RESERVE) for the same address, and the addresses all look totally reasonable (e.g., 0x43f20000)? @karalabe, I wasn't able to make sense of the issue you linked to (ethereum/go-ethereum#15743), since the traceback appears incomplete. Why do think it's related? |
|
@bradfitz Assuming it's not some other go/types unrelated issue, it would be interesting to see which exact code leads to the problem and if it's always the same. The following patch prints the location and operands before the code goes into go/constant.BinaryOp: It probably needs to be refined otherwise it prints too much, but it may be a starting point. (I'm out through 1/1/18 but can try to help after that.) |
I don't have good explanation for the error we see. I would insert println near every VirtualAlloc call to understand allocation pattern here - that might give some clue about what is happening. I am not near computer to try that now. But I wouldn't spend much time trying to understand the reason for ERROR_INVALID_ADDRESS. This crash is happening on Linux too. So, like Brad said, I reckon we use too much memory for these environments. Alex |
|
Change https://golang.org/cl/85887 mentions this issue: |
rsc
added a commit
to rsc/go-bindata
that referenced
this issue
Jan 5, 2018
The current code prints long string concatenations like a + b + c + d + e + f The new code prints ((a + (b + c)) + (d + (e + f))) This has the effect of minimizing the overall height of the expression tree, which makes recursive traversals of the tree a little happier and makes incremental computation of string constants as in go/types and go/vet MUCH happier. The compilers and go/types should be fixed so that this isn't necessary, but we can't fix the old releases, and this is very easy, so it seems worth doing. For golang/go#23222.
rsc
added a commit
to rsc/go-ethereum
that referenced
this issue
Jan 5, 2018
Use current rsc/go-bindata instead of jteeuwen/go-bindata, to get balanced tree in very long string concatenations. This works around problems in current Go distributions. For golang/go#23222.
kevinburke
pushed a commit
to kevinburke/go-bindata
that referenced
this issue
Jan 5, 2018
The current code prints long string concatenations like a + b + c + d + e + f The new code prints ((a + (b + c)) + (d + (e + f))) This has the effect of minimizing the overall height of the expression tree, which makes recursive traversals of the tree a little happier and makes incremental computation of string constants as in go/types and go/vet MUCH happier. The compilers and go/types should be fixed so that this isn't necessary, but we can't fix the old releases, and this is very easy, so it seems worth doing. For golang/go#23222.
karalabe
pushed a commit
to ethereum/go-ethereum
that referenced
this issue
Jan 5, 2018
* go-ethereum/dashboard: update assets.go Use current rsc/go-bindata instead of jteeuwen/go-bindata, to get balanced tree in very long string concatenations. This works around problems in current Go distributions. For golang/go#23222. * dashboard: run last two go:generate steps for linter
|
TL;DR: For cmd/compile, this is a dup of #16394, which @mdempsky fixed in November (thanks!). For cmd/vet (really go/types), there is a new problem, for which I will file an issue and fix for Go 1.10. There is also a minor cmd/gofmt issue for Go 1.11, which I will file. For the older releases I also sent a PR for go-bindata to emit balanced trees and also a PR to ethereum to use that go-bindata. Those are linked above and will fix the problems with older toolchains and ethereum. (Thanks @kevinburke and @karalabe.)
The results on my 2015 Mac laptop, in both 64-bit and 32-bit mode, are below. means that a tool was run on five different programs each with a single expression containing 1000 addition operations. The The compiler does very well, because Matthew fixed this in the parser to address #16394, in https://golang.org/cl/76450, during the Go 1.10 cycle. The reports of stack overflow in the Go 1.9.2 compiler are essentially dups of #16394, now fixed. Even before that change, the compiler needed stack to recurse over the whole expression but otherwise handled it efficiently, with a single allocation to construct the whole concatenation (I did that in 2014, mainly to speed program execution, but it had the effect of speeding compilation too (https://golang.org/cl/83090046)). With Matthew's fix, there's basically nothing left to do in the compiler. The go/types package does not do as well, because it does not have the OADDSTR optimization in typecheck, nor in the parser. When go/constant builds up the actual value of the expression, it is accidentally quadratic on Finally gofmt. I ran gofmt originally as a control, thinking that comparing to gotype would let us see how much time was spent in typechecking (gotype minus gofmt). Unfortunately, gofmt itself seems to have some accidentally quadratic behavior in making its layout decisions. (I won't be surprised if the problem is in the code to decide whether to add spaces around operators, which would make it my fault, and if so, my apologies). I will file a Go 1.11 issue for gofmt. amd64go tool compilegotypego vetgofmt386go tool compilegotypego vetgofmt |
This was referenced Jan 5, 2018
|
Change https://golang.org/cl/86395 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Jan 5, 2018
It is natural for tools to take a large string concatenation like
"1" + "1" + "1" + ... + "1"
and translate that into a sequence of go/constant calls:
x := constant.MakeString("1")
x = constant.BinaryOp(x, token.ADD, constant.MakeString("1"))
x = constant.BinaryOp(x, token.ADD, constant.MakeString("1"))
x = constant.BinaryOp(x, token.ADD, constant.MakeString("1"))
x = constant.BinaryOp(x, token.ADD, constant.MakeString("1"))
...
If the underlying representation of a string constant is a Go string,
then this leads to O(N²) memory for the concatenation of N strings,
allocating memory for "1", "11", "111", "1111", and so on.
This makes go/types and in particular cmd/vet run out of memory
(or at least use far too much) on machine-generated string concatenations,
such as those generated by go-bindata.
This CL allows code like the above to operate efficiently, by delaying
the evaluation of the actual string constant value until it is needed.
Now the representation of a string constant is either a string or an
explicit addition expression. The addition expression is turned into
a string the first time it is requested and then cached for future use.
This slows down the use of single strings, but analyses are likely not
dominated by that use anyway. It speeds up string concatenations,
especially large ones, significantly.
On my Mac running 32-bit code:
name old time/op new time/op delta
StringAdd/1-8 160ns ± 2% 183ns ± 1% +13.98% (p=0.000 n=10+10)
StringAdd/4-8 650ns ± 1% 927ns ± 4% +42.73% (p=0.000 n=10+10)
StringAdd/16-8 3.93µs ± 1% 2.78µs ± 2% -29.12% (p=0.000 n=8+9)
StringAdd/64-8 37.3µs ± 9% 10.1µs ± 5% -73.06% (p=0.000 n=10+10)
StringAdd/256-8 513µs ± 5% 38µs ± 1% -92.63% (p=0.000 n=10+10)
StringAdd/1024-8 5.67ms ± 4% 0.14ms ± 2% -97.45% (p=0.000 n=8+10)
StringAdd/4096-8 77.1ms ± 9% 0.7ms ± 2% -99.10% (p=0.000 n=10+9)
StringAdd/16384-8 1.33s ± 7% 0.00s ±10% -99.64% (p=0.000 n=10+10)
StringAdd/65536-8 21.5s ± 4% 0.0s ± 8% -99.89% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
StringAdd/1-8 232B ± 0% 256B ± 0% +10.34% (p=0.000 n=10+10)
StringAdd/4-8 1.20kB ± 0% 1.24kB ± 0% +3.33% (p=0.000 n=10+10)
StringAdd/16-8 14.7kB ± 0% 4.6kB ± 0% -68.87% (p=0.000 n=10+10)
StringAdd/64-8 223kB ± 0% 16kB ± 0% -92.66% (p=0.000 n=10+10)
StringAdd/256-8 3.48MB ± 0% 0.07MB ± 0% -98.07% (p=0.000 n=10+10)
StringAdd/1024-8 55.7MB ± 0% 0.3MB ± 0% -99.53% (p=0.000 n=10+10)
StringAdd/4096-8 855MB ± 0% 1MB ± 0% -99.88% (p=0.000 n=10+10)
StringAdd/16384-8 13.5GB ± 0% 0.0GB ± 0% -99.97% (p=0.000 n=9+10)
StringAdd/65536-8 215GB ± 0% 0GB ± 0% -99.99% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
StringAdd/1-8 3.00 ± 0% 3.00 ± 0% ~ (all equal)
StringAdd/4-8 9.00 ± 0% 11.00 ± 0% +22.22% (p=0.000 n=10+10)
StringAdd/16-8 33.0 ± 0% 25.0 ± 0% -24.24% (p=0.000 n=10+10)
StringAdd/64-8 129 ± 0% 75 ± 0% -41.86% (p=0.000 n=10+10)
StringAdd/256-8 513 ± 0% 269 ± 0% -47.56% (p=0.000 n=10+10)
StringAdd/1024-8 2.05k ± 0% 1.04k ± 0% -49.29% (p=0.000 n=10+10)
StringAdd/4096-8 8.19k ± 0% 4.12k ± 0% -49.77% (p=0.000 n=10+10)
StringAdd/16384-8 32.8k ± 0% 16.4k ± 0% -49.97% (p=0.000 n=9+10)
StringAdd/65536-8 131k ± 0% 66k ± 0% -50.11% (p=0.000 n=10+10)
https://perf.golang.org/search?q=upload:20180105.2
Fixes #23348 (originally reported as cmd/vet failures in comments on #23222).
This makes constant.Values of Kind String no longer meaningful for ==, which
required fixes in go/types. While there, also fix go/types handling of constant.Values
of Kind Int (for uint64), Float, and Complex.
Change-Id: I80867bc9c4232c5c9b213443ff16645434a68b36
Reviewed-on: https://go-review.googlesource.com/86395
Run-TryBot: Russ Cox <[email protected]>
TryBot-Result: Gobot Gobot <[email protected]>
Reviewed-by: Robert Griesemer <[email protected]>
cooganb
pushed a commit
to cooganb/go-ethereum
that referenced
this issue
Jan 23, 2018
* go-ethereum/dashboard: update assets.go Use current rsc/go-bindata instead of jteeuwen/go-bindata, to get balanced tree in very long string concatenations. This works around problems in current Go distributions. For golang/go#23222. * dashboard: run last two go:generate steps for linter
@aclements the failure is at c:/go/src/runtime/mem_windows.go:124 so it is this call stdcall4(_VirtualAlloc, uintptr(v), n, _MEM_COMMIT, _PAGE_READWRITE)that failed. But according to VirtualAlloc documentation https://msdn.microsoft.com/en-us/library/windows/desktop/aa366887(v=vs.85).aspx: Alex |
gopherbot
pushed a commit
that referenced
this issue
Feb 15, 2018
This replaces the contiguous heap arena mapping with a potentially sparse mapping that can support heap mappings anywhere in the address space. This has several advantages over the current approach: * There is no longer any limit on the size of the Go heap. (Currently it's limited to 512GB.) Hence, this fixes #10460. * It eliminates many failures modes of heap initialization and growing. In particular it eliminates any possibility of panicking with an address space conflict. This can happen for many reasons and even causes a low but steady rate of TSAN test failures because of conflicts with the TSAN runtime. See #16936 and #11993. * It eliminates the notion of "non-reserved" heap, which was added because creating huge address space reservations (particularly on 64-bit) led to huge process VSIZE. This was at best confusing and at worst conflicted badly with ulimit -v. However, the non-reserved heap logic is complicated, can race with other mappings in non-pure Go binaries (e.g., #18976), and requires that the entire heap be either reserved or non-reserved. We currently maintain the latter property, but it's quite difficult to convince yourself of that, and hence difficult to keep correct. This logic is still present, but will be removed in the next CL. * It fixes problems on 32-bit where skipping over parts of the address space leads to mapping huge (and never-to-be-used) metadata structures. See #19831. This also completely rewrites and significantly simplifies mheap.sysAlloc, which has been a source of many bugs. E.g., #21044, #20259, #18651, and #13143 (and maybe #23222). This change also makes it possible to allocate individual objects larger than 512GB. As a result, a few tests that expected huge allocations to fail needed to be changed to make even larger allocations. However, at the moment attempting to allocate a humongous object may cause the program to freeze for several minutes on Linux as we fall back to probing every page with addrspace_free. That logic (and this failure mode) will be removed in the next CL. Fixes #10460. Fixes #22204 (since it rewrites the code involved). This slightly slows down compilebench and the x/benchmarks garbage benchmark. name old time/op new time/op delta Template 184ms ± 1% 185ms ± 1% ~ (p=0.065 n=10+9) Unicode 86.9ms ± 3% 86.3ms ± 1% ~ (p=0.631 n=10+10) GoTypes 599ms ± 0% 602ms ± 0% +0.56% (p=0.000 n=10+9) Compiler 2.87s ± 1% 2.89s ± 1% +0.51% (p=0.002 n=9+10) SSA 7.29s ± 1% 7.25s ± 1% ~ (p=0.182 n=10+9) Flate 118ms ± 2% 118ms ± 1% ~ (p=0.113 n=9+9) GoParser 147ms ± 1% 148ms ± 1% +1.07% (p=0.003 n=9+10) Reflect 401ms ± 1% 404ms ± 1% +0.71% (p=0.003 n=10+9) Tar 175ms ± 1% 175ms ± 1% ~ (p=0.604 n=9+10) XML 209ms ± 1% 210ms ± 1% ~ (p=0.052 n=10+10) (https://perf.golang.org/search?q=upload:20171231.4) name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.23ms ± 1% 2.25ms ± 1% +0.84% (p=0.000 n=19+19) (https://perf.golang.org/search?q=upload:20171231.3) Relative to the start of the sparse heap changes (starting at and including "runtime: fix various contiguous bitmap assumptions"), overall slowdown is roughly 1% on GC-intensive benchmarks: name old time/op new time/op delta Template 183ms ± 1% 185ms ± 1% +1.32% (p=0.000 n=9+9) Unicode 84.9ms ± 2% 86.3ms ± 1% +1.65% (p=0.000 n=9+10) GoTypes 595ms ± 1% 602ms ± 0% +1.19% (p=0.000 n=9+9) Compiler 2.86s ± 0% 2.89s ± 1% +0.91% (p=0.000 n=9+10) SSA 7.19s ± 0% 7.25s ± 1% +0.75% (p=0.000 n=8+9) Flate 117ms ± 1% 118ms ± 1% +1.10% (p=0.000 n=10+9) GoParser 146ms ± 2% 148ms ± 1% +1.48% (p=0.002 n=10+10) Reflect 398ms ± 1% 404ms ± 1% +1.51% (p=0.000 n=10+9) Tar 173ms ± 1% 175ms ± 1% +1.17% (p=0.000 n=10+10) XML 208ms ± 1% 210ms ± 1% +0.62% (p=0.011 n=10+10) [Geo mean] 369ms 373ms +1.17% (https://perf.golang.org/search?q=upload:20180101.2) name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.22ms ± 1% 2.25ms ± 1% +1.51% (p=0.000 n=20+19) (https://perf.golang.org/search?q=upload:20180101.3) Change-Id: I5daf4cfec24b252e5a57001f0a6c03f22479d0f0 Reviewed-on: https://go-review.googlesource.com/85887 Run-TryBot: Austin Clements <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Rick Hudson <[email protected]>
guilhermebr
pushed a commit
to gochain/gochain
that referenced
this issue
Feb 18, 2018
* go-ethereum/dashboard: update assets.go Use current rsc/go-bindata instead of jteeuwen/go-bindata, to get balanced tree in very long string concatenations. This works around problems in current Go distributions. For golang/go#23222. * dashboard: run last two go:generate steps for linter
cooganb
pushed a commit
to cooganb/go-ethereum
that referenced
this issue
Feb 23, 2018
* go-ethereum/dashboard: update assets.go Use current rsc/go-bindata instead of jteeuwen/go-bindata, to get balanced tree in very long string concatenations. This works around problems in current Go distributions. For golang/go#23222. * dashboard: run last two go:generate steps for linter
shuLhan
pushed a commit
to shuLhan/go-bindata
that referenced
this issue
Feb 27, 2018
The current code prints long string concatenations like a + b + c + d + e + f The new code prints ((a + (b + c)) + (d + (e + f))) This has the effect of minimizing the overall height of the expression tree, which makes recursive traversals of the tree a little happier and makes incremental computation of string constants as in go/types and go/vet MUCH happier. The compilers and go/types should be fixed so that this isn't necessary, but we can't fix the old releases, and this is very easy, so it seems worth doing. For golang/go#23222.
shuLhan
pushed a commit
to shuLhan/go-bindata
that referenced
this issue
Mar 2, 2018
The current code prints long string concatenations like a + b + c + d + e + f The new code prints ((a + (b + c)) + (d + (e + f))) This has the effect of minimizing the overall height of the expression tree, which makes recursive traversals of the tree a little happier and makes incremental computation of string constants as in go/types and go/vet MUCH happier. The compilers and go/types should be fixed so that this isn't necessary, but we can't fix the old releases, and this is very easy, so it seems worth doing. For golang/go#23222.
shuLhan
pushed a commit
to shuLhan/go-bindata
that referenced
this issue
May 19, 2018
The current code prints long string concatenations like a + b + c + d + e + f The new code prints ((a + (b + c)) + (d + (e + f))) This has the effect of minimizing the overall height of the expression tree, which makes recursive traversals of the tree a little happier and makes incremental computation of string constants as in go/types and go/vet MUCH happier. The compilers and go/types should be fixed so that this isn't necessary, but we can't fix the old releases, and this is very easy, so it seems worth doing. For golang/go#23222.
mariameda
pushed a commit
to NiluPlatform/go-nilu
that referenced
this issue
Aug 23, 2018
* go-ethereum/dashboard: update assets.go Use current rsc/go-bindata instead of jteeuwen/go-bindata, to get balanced tree in very long string concatenations. This works around problems in current Go distributions. For golang/go#23222. * dashboard: run last two go:generate steps for linter
hatricker
pushed a commit
to thundercore/thundercore-localchain
that referenced
this issue
Oct 8, 2018
* go-ethereum/dashboard: update assets.go Use current rsc/go-bindata instead of jteeuwen/go-bindata, to get balanced tree in very long string concatenations. This works around problems in current Go distributions. For golang/go#23222. * dashboard: run last two go:generate steps for linter
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What version of Go are you using (
go version)?go version go1.9.2 windows/386Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?What did you do?
We've noticed on our CI that 32bit Windows builds started crashing during compilation https://ci.appveyor.com/project/ethereum/go-ethereum/build/master.4791/job/hcsxkgy0uh018j05#L136.
After playing around with it trying to repro locally, we've managed to triage it to the 32bit Windows build of Go. To reproduce the issue yourself:
go get -v github.com/ethereum/go-ethereum/dashboardThe repository is quite large, so downloading it will take a while.
What did you expect to see?
A successful build.
What did you see instead?
Issue being tracked on the go-ethereum repo too: ethereum/go-ethereum#15738.
The text was updated successfully, but these errors were encountered: