shutdown bugfix: Prevent segfault in server if connection is broken during long function call #118
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s during method call Fix issue where if a client disconnects in the middle of a long running IPC method call, the server will segfault when the method eventually returns. This behavior was reported by tdb3 in bitcoin/bitcoin#31003 (review) including a stack trace showing where the crash happens in the PassField mp.Context overload while calling request_threads.erase.
|
Originally posted by @tdb3 in bitcoin/bitcoin#31003 (review):
|
ryanofsky
added a commit
to ryanofsky/bitcoin
that referenced
this pull request
Oct 8, 2024
Add recent changes and fixes for shutdown bugs: bitcoin-core/libmultiprocess#111: doc: Add internal design section bitcoin-core/libmultiprocess#113: Add missing include to util.h bitcoin-core/libmultiprocess#116: shutdown bugfix: destroy RPC system before running cleanup callbacks bitcoin-core/libmultiprocess#118: shutdown bugfix: Prevent segfault in server if connection is broken during long function call
ryanofsky
added a commit
to ryanofsky/bitcoin
that referenced
this pull request
Oct 8, 2024
Add recent changes and fixes for shutdown bugs: bitcoin-core/libmultiprocess#111: doc: Add internal design section bitcoin-core/libmultiprocess#113: Add missing include to util.h bitcoin-core/libmultiprocess#116: shutdown bugfix: destroy RPC system before running cleanup callbacks bitcoin-core/libmultiprocess#118: shutdown bugfix: Prevent segfault in server if connection is broken during long function call
|
Still encountering a segfault. Trace below. |
|
Following the instructions from bitcoin/bitcoin#31003 (review) (complicated but steps to reproduce were described very clearly!) I was able to reproduce the original segfault and a new segfault after applying the current fix 7f1d33f Will debug, but stack trace is: #0 std::operator== (__x=..., __y=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:330
#1 mp::EventLoop::post(std::function<void ()> const&) (this=0x0, fn=...) at /home/russ/work/mp/src/mp/proxy.cpp:219
#2 0x000055ac4fc57dc4 in mp::EventLoop::sync<mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}::operator()()::{lambda()#3}::operator()() const::{lambda()#1}>(mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}::operator()()::{lambda()#3}::operator()() const::{lambda()#1}&&) (this=0x0, callable=...) at /home/russ/work/mp/build/prefix/include/mp/proxy-io.h:155
#3 0x000055ac4fc57d44 in mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}::operator()()::{lambda()#3}::operator()() const (this=0x7f1c4affc7e8) at /home/russ/work/mp/build/prefix/include/mp/proxy-types.h:171
#4 0x000055ac4fc52267 in kj::runCatchingExceptions<mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}::operator()()::{lambda()#3}>(mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}::operator()()::{lambda()#3}&&) (func=...) at /nix/store/0rvzw1dahhd1h8qyhyxmwgr2l745fn3a-capnproto-1.0.2/include/kj/exception.h:371
#5 0x000055ac4fc51db9 in mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}::operator()() (this=0x7f1c8001c4d0) at /home/russ/work/mp/build/prefix/include/mp/proxy-types.h:171
#6 0x000055ac4fc51afe in mp::AsyncCallable<mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}>::operator()() const (this=0x7f1c8001c440) at /home/russ/work/mp/build/prefix/include/mp/util.h:189
#7 0x000055ac4fc51aa6 in std::__invoke_impl<void, mp::AsyncCallable<mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}>&>(std::__invoke_other, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&&) (__f=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:61
#8 0x000055ac4fc519f6 in std::__invoke_r<void, mp::AsyncCallable<mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}>&>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&&) (__fn=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:111
#9 0x000055ac4fc5182e in std::_Function_handler<void (), mp::AsyncCallable<mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > >, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > > >(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::Mining>, capnp::CallContext<ipc::capnp::messages::Mining::WaitFeesChangedParams, ipc::capnp::messages::Mining::WaitFeesChangedResults> >&, mp::ServerField<1, mp::Accessor<mp::mining_fields::CurrentTip, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::FeeThreshold, 1>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Timeout, 1>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 2>, mp::ServerCall> > > > > const&, mp::TypeList<uint256, long, node::BlockCreateOptions const&, std::chrono::duration<double, std::ratio<1l, 1000l> > >&&)::{lambda()#1}> >::_M_invoke(std::_Any_data const&) (__functor=...)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_function.h:290
#10 0x000055ac5016f360 in std::function<void ()>::operator()() const (this=<optimized out>) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_function.h:591
#11 mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1})::{lambda()#1}::operator()() const (this=<optimized out>) at /home/russ/work/mp/include/mp/proxy-io.h:261
#12 std::condition_variable::wait<mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1})::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1})::{lambda()#1}) (this=0x7f1c44001278, __lock=..., __p=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/condition_variable:104
#13 mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::{lambda()#1}) (this=0x7f1c44001250, lock=..., pred=...) at /home/russ/work/mp/include/mp/proxy-io.h:251
#14 mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const (this=<optimized out>) at /home/russ/work/mp/src/mp/proxy.cpp:364
#15 std::__invoke_impl<void, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(std::__invoke_other, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) (__f=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:61
#16 std::__invoke<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) (__fn=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:96
#17 std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:292
#18 std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> >::operator()() (this=<optimized out>)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:299
#19 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0> > >::_M_run() (this=<optimized out>)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:244
#20 0x00007f1c90ee8683 in execute_native_thread_routine () from /nix/store/agp6lqznayysqvqkx4k1ggr8n1rsyi8c-gcc-13.2.0-lib/lib/libstdc++.so.6
#21 0x00007f1c90bc0383 in start_thread () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6
#22 0x00007f1c90c4300c in clone3 () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6 |
…cts during method call This an imperfect workaround for a crash that can be triggered if a client disconnects in the middle of a long running IPC call. The workaround results in memory leak but avoids a crash. This can be improved later, and more details are in a code comment.
|
The second crash turned out to be more complicated to fix than the first one, so just pushing a workaround for now that avoids the crash but results in a small memory leak. I will open a separate PR with a more complete fix. I should also add a test for this issue making sure deleting server-side Connection* during an ongoing method call does not cause crashes. |
ryanofsky
added a commit
to ryanofsky/bitcoin
that referenced
this pull request
Oct 16, 2024
Add recent changes and fixes for shutdown bugs. bitcoin-core/libmultiprocess#111: doc: Add internal design section bitcoin-core/libmultiprocess#113: Add missing include to util.h bitcoin-core/libmultiprocess#116: shutdown bugfix: destroy RPC system before running cleanup callbacks bitcoin-core/libmultiprocess#118: shutdown bugfix: Prevent segfault in server if connection is broken during long function call bitcoin-core/libmultiprocess#119: cmake: avoid libatomic not found error on debian
fanquake
added a commit
to bitcoin/bitcoin
that referenced
this pull request
Oct 21, 2024
90b4055 Update libmultiprocess library (Ryan Ofsky) Pull request description: Add recent changes and fixes for shutdown bugs. bitcoin-core/libmultiprocess#111: doc: Add internal design section bitcoin-core/libmultiprocess#113: Add missing include to util.h bitcoin-core/libmultiprocess#116: shutdown bugfix: destroy RPC system before running cleanup callbacks bitcoin-core/libmultiprocess#118: shutdown bugfix: Prevent segfault in server if connection is broken during long function call bitcoin-core/libmultiprocess#119: cmake: avoid libatomic not found error on debian ACKs for top commit: fanquake: ACK 90b4055 TheCharlatan: ACK 90b4055 Tree-SHA512: 2c256667f0c16e00bb5a81b2c6d3db103fae211844e32b111bbed673ab2612ad1478e6b3ecd3a867a4e425cfa6e778b67388343626597a8fac800a15cea5e53a
janus
pushed a commit
to BitgesellOfficial/bitgesell
that referenced
this pull request
Jan 19, 2025
Add recent changes and fixes for shutdown bugs. bitcoin-core/libmultiprocess#111: doc: Add internal design section bitcoin-core/libmultiprocess#113: Add missing include to util.h bitcoin-core/libmultiprocess#116: shutdown bugfix: destroy RPC system before running cleanup callbacks bitcoin-core/libmultiprocess#118: shutdown bugfix: Prevent segfault in server if connection is broken during long function call bitcoin-core/libmultiprocess#119: cmake: avoid libatomic not found error on debian
ryanofsky
added a commit
to ryanofsky/libmultiprocess
that referenced
this pull request
Jul 1, 2025
This reverts commit 196e6fc from bitcoin-core#118 which added a workaround which is no longer needed after 315ff53 from bitcoin-core#160 When the workaround was introduced it prevented segfaults that happened when an IPC client disconnected during a long-running asynchronous call by leaking server objects instead of crashing. These leaks could sometimes cause the new "disconnecting and blocking" unit test introduced in bitcoin-core#160 to hang, and since the workaround is no longer necessary, the fix is to revert it. The problem with the hanged test was reported: - Sjors/bitcoin#90 (comment) - https://github.com/Sjors/bitcoin/actions/runs/15966265407/job/45027248310?pr=90 - Sjors/bitcoin#90 (comment) - https://cirrus-ci.com/task/4999408900636672 And the fix was posted: - Sjors/bitcoin#90 (comment)
ryanofsky
added a commit
to ryanofsky/libmultiprocess
that referenced
this pull request
Jul 1, 2025
This reverts commit 196e6fc from bitcoin-core#118 which added a workaround which is no longer needed after 315ff53 from bitcoin-core#160 When the workaround was introduced it prevented segfaults that happened when an IPC client disconnected during a long-running asynchronous call. But it prevented them by leaking server objects, and these leaks now sometimes cause the new "disconnecting and blocking" unit test introduced in bitcoin-core#160 to hang. Since the workaround is no longer necessary, revert it now to fix the test hangs. The problem with test hangs was reported: - Sjors/bitcoin#90 (comment) - https://github.com/Sjors/bitcoin/actions/runs/15966265407/job/45027248310?pr=90 - Sjors/bitcoin#90 (comment) - https://cirrus-ci.com/task/4999408900636672 And the fix was posted: - Sjors/bitcoin#90 (comment)
Note: this is done now. The second commit of this PR was reverted in #186. The workaround was no longer needed after #160, and it was actually causing intermittent test hangs in the new unit test added in #160 disconnecting the client during an IPC call. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix issue where if a client disconnects in the middle of a long running IPC method call, the server will segfault when the method eventually returns.
This issue was reported tdb3 in bitcoin/bitcoin#31003 (review) with a stack trace showing where the crash happens in the
PassFieldmp.Contextoverload while callingrequest_threads.erase.