Support ES|QL

[jsvd]

Description

Support executing ES|QL queries with the plugin.

Design note: https://docs.google.com/document/d/1jPtz3IeYx8LyoIyJua3RgjVsW4bC9hvWNw3CYhrHUIY

Upgrade elasticsearch-ruby

• needs upgrading elasticsearch-ruby client to 8.x to support ES|QL, ES|QL introduced in 8.11
• logstash core currently locks it at 7.x
• the manticore adapter has been moved to the "elastic-transport" gem and the namespace changed to Elastic:: instead of Elasticsearch:: (same with the requires)

this would be an alternative to the search or aggregations type of query.

Following issues and PR addressed the es-ruby client upgrades:

• migrate logstash-input-elasticsearch to elasticsearch-ruby 8.x elastic/logstash#17092
• Adds elastic-transport client support #223

[mashhurs]

High level thoughts

• to support ES|QL
  For the ES|QL, we can use either ES-ruby client helper or directly utilize the client.esql.
  ESQL capabilities were added in elasticsearch-v8.11 version, so we need at least 8.11 version to run ES|QL queries (add s.add_runtime_dependency 'elasticsearch', '>= 8.11.0' to plugin gemspec file)

  def query(client, query, params = {}, parser: {})
        response = client.esql.query({ body: { query: query }, format: 'json' }.merge(params))

        columns = response['columns']
        response['values'].map do |value|
          (value.length - 1).downto(0).map do |index|
            key = columns[index]['name']
            value[index] = parser[key].call(value[index]) if value[index] && parser[key]
            { key => value[index] }
          end.reduce({}, :merge)
        end
  end

and call it with defined parser

    parser = {
      '@timestamp' => Proc.new { |t| DateTime.parse(t) },
      'client.ip' => Proc.new { |i| IPAddr.new(i) },
      'event.duration' => Proc.new { |d| d.to_s }
    }
    @client = Elasticsearch::Client.new(@client_options)
    query = "FROM example-index"
    response = query(@client, query, parser: parser)

• Currently, the plugin bases ES-DSL search, we probably need to define a mode (query_mode => es-dsl | es-ql)

What would be the best practise to handle limitations?

• https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-limitations.html

Documentation

• document changed params for added ES|QL functionality
• add multiple examples (including how plugin behaves on the limitations)
• add a separate section for ES|QL and forward to ES|QL getting started and limitations pages