http update + distributed table + wrong replication cluster = crash

[sanikolaev]

Bug Description:

If you try to run /update on a document in a distributed table and specify a cluster that the tables of the distributed table don't belong to, it will crash.

# mysql -v -P9306 -h0 -e "drop table if exists t; create table t(f int); drop table if exists d; CREATE TABLE d type='distributed' agent='127.0.0.1:9312:t'"

--------------
drop table if exists t
--------------

--------------
create table t(f int)
--------------

--------------
drop table if exists d
--------------

--------------
CREATE TABLE d type='distributed' agent='127.0.0.1:9312:t'
--------------

# curl -s -X POST http://localhost:9312/update -d '{"cluster": "unknown_cluster", "table": "d", "id": 2, "doc": {"f": 5}}'

Backtrace:

 0# sphBacktrace(int, bool) in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd
 1# HandleCrash(int) in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd
 2# _sigtramp in /usr/lib/system/libsystem_platform.dylib
 3# ProcessHttpQueryBuddy(HttpProcessResult_t&, std::__1::pair<char const*, int>, CSphOrderedHash<CSphString, CSphString, CSphStrHashFunc, 256>&, sph::Vector_T<unsigned char, sph::DefaultCopy_T<unsigned ch
ar>, sph::DefaultRelimit, sph::DefaultStorage_T<unsigned char>>&, bool, http_method) in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd
 4# sphProcessHttpQueryNoResponce(CSphString const&, CSphString const&, sph::Vector_T<unsigned char, sph::DefaultCopy_T<unsigned char>, sph::DefaultRelimit, sph::DefaultStorage_T<unsigned char>>&) in /opt/
homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd
 5# HandleCommandJson(ISphOutputBuffer&, unsigned short, InputBuffer_c&) in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd
 6# ExecuteApiCommand(SearchdCommand_e, unsigned short, int, InputBuffer_c&, GenericOutputBuffer_c&) in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd
 7# ApiServe(std::__1::unique_ptr<AsyncNetBuffer_c, std::__1::default_delete<AsyncNetBuffer_c>>) in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd
 8# MultiServe(std::__1::unique_ptr<AsyncNetBuffer_c, std::__1::default_delete<AsyncNetBuffer_c>>, std::__1::pair<int, unsigned char>, Proto_e) in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1
f8a194f0/bin/searchd
 9# std::__1::__function::__func<NetActionAccept_c::Impl_c::ProcessAccept()::$_3, std::__1::allocator<NetActionAccept_c::Impl_c::ProcessAccept()::$_3>, void ()>::operator()() in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd
10# Threads::CoRoutine_c::CreateContext(std::__1::function<void ()>, std::__1::pair<boost::context::stack_context, Threads::StackFlavour_E>)::'lambda'(boost::context::detail::transfer_t)::__invoke(boost::context::detail::transfer_t) in /opt/homebrew/Cellar/manticoresearch-dev/10.2.3-25061609-1f8a194f0/bin/searchd

Manticore Search Version:

Manticore 10.2.3 1f8a194f0@25061609 dev (columnar 4.2.2 31be562@25052112) (secondary 4.2.2 31be562@25052112)

Operating System Version:

Macos

Have you tried the latest development version?

Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed

[sanikolaev]

This issue blocks manticoresoftware/manticoresearch-buddy#559

[tomatolog]

I pushed the fix into #3484 The fix increase API protocol version that is why it could be a breaking change.

Now the daemon does not crash. However I see the different reply for the request while Buddy is not running and running along with the daemon

No Buddy running

C:\dev\sphinx\build\col_dbg\_deps\manticore-build\test>curl -s -X POST http://localhost:9312/update -d "{\"table\": \"t\", \"id\": 2, \"doc\": {\"f\": 5}}"
{"table":"t","id":2,"result":"noop"}
C:\dev\sphinx\build\col_dbg\_deps\manticore-build\test>curl -s -X POST http://localhost:9312/update -d "{\"cluster\": \"unknown_cluster\", \"table\": \"t\", \"id\": 2, \"doc\": {\"f\": 5}}"
{"error":{"type":"action_request_validation_exception","reason":"table t: table 't' is not in any cluster, use just 't'","table":"t"},"status":409}

Buddy running

C:\dev\sphinx\build\col_dbg\_deps\manticore-build\test>curl -s -X POST http://localhost:9312/update -d "{\"table\": \"t\", \"id\": 2, \"doc\": {\"f\": 5}}"
{"table":"t","id":2,"result":"noop"}
C:\dev\sphinx\build\col_dbg\_deps\manticore-build\test>curl -s -X POST http://localhost:9312/update -d "{\"cluster\": \"unknown_cluster\", \"table\": \"t\", \"id\": 2, \"doc\": {\"f\": 5}}"
{"error":"Trying to access array offset on null"}

Not quite sure why I see that behavior and should it be fixed prior to the merge of the branch at #3484 or this behavior should be just declared in the CLT test

[donhardman]

Now daemon returns noop on /update with proper cluster set

curl -s -X POST http://localhost:1308/update -d '{"cluster": "replication", "table": "test2", "id": 2, "doc": {"price": 1099.99, "discounted_price": 999.99}}' | jq
––– output –––
- "updated"
+ {
+   "table": "test2",
+   "id": 2,
+   "result": "noop"
+ }

Also, it returns noop with the incorrect cluster with buddy instead of an error, even when buddy returns a proper error.

What does "noop" mean in our terminology on update operation?

[donhardman]

Buddy should be run in fix/auto-sharding-rebalance where Something went wrong fixed

So test in the pull #3484 fails with noop instead of proxy error:

test/clt-tests/bugs/3481-http-update-distributed-table-wrong-cluster-crash.rec

Here the log:

CLT_DEBUG=1 clt test -d -t $test $docker_image; echo $?
Replaying data from the file: test/clt-tests/bugs/3481-http-update-distributed-table-wrong-cluster-crash.rec
The replay result will be stored to the file: test/clt-tests/bugs/3481-http-update-distributed-table-wrong-cluster-crash.rep

––– input –––
rm -f /var/log/manticore/searchd.log; stdbuf -oL searchd --stopwait > /dev/null; stdbuf -oL searchd --logdebugv > /dev/null
––– output –––
––– input –––
if timeout 10 grep -qm1 'accepting connections' <(tail -n 1000 -f /var/log/manticore/searchd.log); then echo 'Manticore started!'; else echo 'Timeout or failed!'; fi
––– output –––
Manticore started!
––– input –––
mysql -v -P9306 -h0 -e "drop table if exists t; create table t(f int); drop table if exists d; CREATE TABLE d type='distributed' agent='127.0.0.1:9312:t'"
––– output –––
--------------
drop table if exists t
--------------
--------------
create table t(f int)
--------------
--------------
drop table if exists d
--------------
--------------
CREATE TABLE d type='distributed' agent='127.0.0.1:9312:t'
--------------
––– input –––
curl -s -X POST http://localhost:9312/update -d '{"cluster": "unknown_cluster", "table": "d", "id": 2, "doc": {"f": 5}}'
––– output –––
- {"error":{"type":"action_request_validation_exception","reason":"","table":"d"},"status":409}
+ {"table":"d","id":2,"result":"noop"}
––– input –––
sleep 2; cat /var/log/manticore/searchd.log | grep 'response data' | cut -d' ' -f11-
––– output –––
{"version":3,"type":"json response","message":{"error":{"type":"action_request_validation_exception","reason":"table t: table 't' is not in any cluster, use just 't'","table":"t"},"status":409},"meta":null,"error_code":0}

So my concern here when BUDDY returns:

{"version":3,"type":"json response","message":{"error":{"type":"action_request_validation_exception","reason":"table t: table 't' is not in any cluster, use just 't'","table":"t"},"status":409},"meta":null,"error_code":0}

DAEMON still responds with

"table":"d","id":2,"result":"noop"}

INSTEAD of

{"error":{"type":"action_request_validation_exception","reason":"","table":"d"},"status":409}

[tomatolog]

the main issue related to the #3481 (comment) is #3537

[tomatolog]

I pushed the fix 312f8ab that fixes the case #3481 (comment) there is no

• other agents or local indexes at the distributed index
• agents has no retry options or mirrors

however for general fix the #3537 should be fixed