Migration to Buddy communication protocol v2

[donhardman]

We need to make updates to the Manticore Daemon and synchronize these changes with the Buddy updates for protocol version 2, as detailed here:

https://github.com/manticoresoftware/manticoresearch-buddy/tree/readme/new-protocol?tab=readme-ov-file#communication-protocol-v2

To review these updates, we can use the Buddy source from the readme/new-protocol branch.

[donhardman]

We can proceed with implementation on the daemon side.

[sanikolaev]

As discussed, we don't need to support the older version neither in Buddy nor in the daemon. It's enough that we just detect a version incompatibility and trigger an error about it (like with MCL versions).

[tomatolog]

the unclear case that part about the Buddy reply

• error_code: An integer representing the HTTP error code that the daemon should use to respond to the client. If it's 0, Manticore should return the original error code to the client.

currently we have such mapping of enum of HTTP_CODE into text that prints into HTTP reply packet

static const char * g_dHttpStatus[] = { "100 Continue", "200 OK", "206 Partial Content", "400 Bad Request", "403 Forbidden", "404 Not Found", "405 Method Not Allowed", "409 Conflict", "413 Request Entity Too Large", "500 Internal Server Error", "501 Not Implemented", "503 Service Unavailable", "526 Invalid SSL Certificate" };

and only these HTTP_CODES supported

enum ESphHttpStatus
{
	SPH_HTTP_STATUS_100,
	SPH_HTTP_STATUS_200,
	SPH_HTTP_STATUS_206,
	SPH_HTTP_STATUS_400,
	SPH_HTTP_STATUS_403,
	SPH_HTTP_STATUS_404,
	SPH_HTTP_STATUS_405,
	SPH_HTTP_STATUS_409,
	SPH_HTTP_STATUS_413,
	SPH_HTTP_STATUS_500,
	SPH_HTTP_STATUS_501,
	SPH_HTTP_STATUS_503,
	SPH_HTTP_STATUS_526,

should we add support of all error codes from the list https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml ?

[tomatolog]

another unclear moment that for SphinxQL query error_code got parsed but have no any use as ProcessSqlQueryBuddy has only original error string that daemon passed into buddy and in case of any error on communicating with the buddy daemon returns original binary packet into client

But after daemon receives reply from buddy it also do not uses original error string not error_code from buddy as it not clear what string should be produced from the integer error_code
Daemon checks for any error property at the ConvertJsonDataset function
https://github.com/manticoresoftware/manticoresearch/blob/master/src/searchdhttp.cpp#L1522 and if daemon sees any it retply to client with the error message with that error property text.

I left SphinxQL path as is the only change that it also parses error_code from the buddy reply.

[tomatolog]

I've pushed the change of the demon into the buddy_http_code branch

[donhardman]

Tried to build it locally but failed due to error:

Consolidate compiler generated dependencies of target lmanticore
[ 56%] Built target lmanticore
Consolidate compiler generated dependencies of target repli_stuff
[ 56%] Building CXX object src/replication/CMakeFiles/repli_stuff.dir/wsrep_v25.cpp.o
/manticoresearch/src/replication/wsrep_v25.cpp:359:15: error: declaration of 'using Status_e = enum class RAW25::Status_e' changes meaning of 'Status_e' [-fpermissive]
  359 |         using Status_e = Status_e;
      |               ^~~~~~~~
/manticoresearch/src/replication/wsrep_v25.cpp:18:12: note: 'Status_e' declared here as 'enum class RAW25::Status_e'
   18 | enum class Status_e : int {
      |            ^~~~~~~~
make[2]: *** [src/replication/CMakeFiles/repli_stuff.dir/build.make:342: src/replication/CMakeFiles/repli_stuff.dir/wsrep_v25.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1876: src/replication/CMakeFiles/repli_stuff.dir/all] Error 2

As a result created pull request to check if it works and also if it will I will get the binary from there to test: #1869

[tomatolog]

tried to reproduce this build issue locally following the steps from the Manual

by build in the container

docker run -it --rm -e SYSROOT_URL=https://repo.manticoresearch.com/repository/sysroots -e arch=x86_64 -e DISTR=rhel7 -e boost=boost_rhel_feb17 -e sysroot=roots_nov22 -v C:\dev\sphinx\manticore:/manticore_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa manticoresearch/external_toolchain:clang15_cmake3263 bash

and configure buddy htttp configure.txt and build buddy http build.txt finished well

Need the description of how to reproduce this build failure locally in docker container for example