Create new detector / "mapper [query] cannot be changed from type [percolator_ext] to [text]" [BUG]

[jheidecker]

What is the bug?
Attempting to create a new detector (minimally configured) per. the documentation results in an error using both the dashboard or API.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Install fresh Opensearch w/Dashboards 2.4.0, using the helm chart on kubernetes.
2. Send data from a packetbeat (dns traffic) through logstash to an index (called packetbeat-8.5.0-2022.11.23 in this case)
3. Attempt to create any detector (GUI or API)

What is the expected behavior?
Detector is created, configured and runs.

What is your host/environment?

• OS: Kubernetes 1.24.6 on Fedora CoreOS 411.36.4
• Deployed with https://github.com/opensearch-project/helm-charts (nearly vanilla config)

Do you have any screenshots?
API log, Console Log and Dashboard screenshot attached.

Do you have any additional context?
Seems to be an underlying issue with mappings, but I am unable to troubleshoot further as the indexes are working exactly as expected with all other plugins.

at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:843) ~[opensearch-2.4.0.jar:2.4.0]
at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:414) ~[opensearch-2.4.0.jar:2.4.0]
at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:285) ~[opensearch-2.4.0.jar:2.4.0]
at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:196) ~[opensearch-2.4.0.jar:2.4.0]
at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:176) ~[opensearch-2.4.0.jar:2.4.0]
at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:214) ~[opensearch-2.4.0.jar:2.4.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.4.0.jar:2.4.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.4.0.jar:2.4.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.4.0.jar:2.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
uncaught exception in thread [DefaultDispatcher-worker-4]
RemoteTransportException[[scsp-internal-master-1][10.129.2.29:9300][indices:admin/mapping/put]]; nested: IllegalArgumentException[mapper [query] cannot be changed from type [percolator_ext] to [text]];
Caused by: java.lang.IllegalArgumentException: mapper [query] cannot be changed from type [percolator_ext] to [text]
at org.opensearch.index.mapper.ParametrizedFieldMapper.merge(ParametrizedFieldMapper.java:112)
at org.opensearch.index.mapper.ParametrizedFieldMapper.merge(ParametrizedFieldMapper.java:76)
at org.opensearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:616)
at org.opensearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:350)
at org.opensearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:575)
at org.opensearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:345)
at org.opensearch.index.mapper.Mapping.merge(Mapping.java:129)
at org.opensearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:307)
at org.opensearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.applyRequest(MetadataMappingService.java:271)
at org.opensearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.execute(MetadataMappingService.java:237)
at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:843)
at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:414)
at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:285)
at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:196)
at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:176)
at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:214)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
[2022-11-23T19:07:16,667][INFO ][o.o.j.s.JobSweeper ] [scsp-internal-master-3] Running full sweep
[2022-11-23T19:07:25,358][INFO ][o.o.a.a.AlertIndices ] [scsp-internal-master-3] Index mapping of .opensearch-sap-dns-alerts is updated
[2022-11-23T19:07:25,363][INFO ][o.o.a.a.AlertIndices ] [scsp-internal-master-3] Index mapping of .opensearch-sap-dns-alerts-history-2022.11.23-1 is updated
[2022-11-23T19:07:25,367][INFO ][o.o.a.a.AlertIndices ] [scsp-internal-master-3] Index mapping of .opensearch-sap-dns-findings-2022.11.23-1 is updated
[2022-11-23T19:07:25,375][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [scsp-internal-master-3] uncaught exception in thread [DefaultDispatcher-worker-5]
org.opensearch.transport.RemoteTransportException: [scsp-internal-master-1][10.129.2.29:9300][indices:admin/mapping/put]

POST _plugins/_security_analytics/detectors
{
  "enabled": true,
  "schedule": {
    "period": {
      "interval": 1,
      "unit": "MINUTES"
    }
  },
  "detector_type": "DNS",
  "type": "detector",
  "inputs": [
    {
      "detector_input": {
        "description": "dns detector for security analytics",
        "indices": [
          "packetbeat-8.5.0-2022.11.23"
        ],
        "pre_packaged_rules": [
          {
            "id": "8ae51330-899c-4641-8125-e39f2e07da72"
          }
        ]
      }
    }
  ],
  "name": "DNS Test"
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_analytics_exception",
        "reason" : "[scsp-internal-master-1][10.129.2.29:9300][indices:admin/mapping/put]"
      }
    ],
    "type" : "security_analytics_exception",
    "reason" : "[scsp-internal-master-1][10.129.2.29:9300][indices:admin/mapping/put]",
    "caused_by" : {
      "type" : "exception",
      "reason" : "org.opensearch.alerting.util.AlertingException: [scsp-internal-master-1][10.129.2.29:9300][indices:admin/mapping/put]"
    }
  },
  "status" : 500
}

[sbcd90]

hi @jheidecker , can you please check if the packetbeat data you're sending also has a field called query of type text?

[jheidecker]

hi @jheidecker , can you please check if the packetbeat data you're sending also has a field called query of type text?

It does!

        },
        "query" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },

'query' would almost always be a field in an index with DNS queries, I think. I'll try another index.

[sbcd90]

@jheidecker , we will try to address it in next release.

[jheidecker]

Just closing the loop. Removing the query field in the target index does prevent this error and the detector is working.

[petardz]

@jheidecker

Can you please provide index mappings:

GET packetbeat-8.5.0-2022.11.23/_mapping

[raj-chak]

@jheidecker

Can you please provide index mappings:

GET packetbeat-8.5.0-2022.11.23/_mapping

It would be very helpful

[jheidecker]

That index is gone, but this should be the same:

{ "packetbeat-8.5.0-2022.12.13" : { "mappings" : { "properties" : { "@timestamp" : { "type" : "date" }, "@version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "agent" : { "properties" : { "ephemeral_id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "client" : { "properties" : { "bytes" : { "type" : "long" }, "ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "port" : { "type" : "long" } } }, "destination" : { "properties" : { "bytes" : { "type" : "long" }, "domain" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "mac" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "packets" : { "type" : "long" }, "port" : { "type" : "long" } } }, "dns" : { "properties" : { "additionals_count" : { "type" : "long" }, "answers" : { "properties" : { "algorithm" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "class" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "data" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "expiration" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "expire" : { "type" : "long" }, "flags" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "inception" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "key_tag" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "labels" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "minimum" : { "type" : "long" }, "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "original_ttl" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "port" : { "type" : "long" }, "preference" : { "type" : "long" }, "priority" : { "type" : "long" }, "protocol" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "refresh" : { "type" : "long" }, "retry" : { "type" : "long" }, "rname" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "serial" : { "type" : "long" }, "signer_name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "ttl" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type_covered" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "weight" : { "type" : "long" } } }, "answers_count" : { "type" : "long" }, "authorities_count" : { "type" : "long" }, "flags" : { "properties" : { "authentic_data" : { "type" : "boolean" }, "authoritative" : { "type" : "boolean" }, "checking_disabled" : { "type" : "boolean" }, "recursion_available" : { "type" : "boolean" }, "recursion_desired" : { "type" : "boolean" }, "truncated_response" : { "type" : "boolean" } } }, "header_flags" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "id" : { "type" : "long" }, "op_code" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "opt" : { "properties" : { "do" : { "type" : "boolean" }, "ext_rcode" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "udp_size" : { "type" : "long" }, "version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "question" : { "properties" : { "class" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "etld_plus_one" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "registered_domain" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "subdomain" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "top_level_domain" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "resolved_ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "response_code" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "ecs" : { "properties" : { "version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "error" : { "properties" : { "message" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "event" : { "properties" : { "action" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "category" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "dataset" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "duration" : { "type" : "long" }, "end" : { "type" : "date" }, "kind" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "start" : { "type" : "date" }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "flow" : { "properties" : { "final" : { "type" : "boolean" }, "id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "host" : { "properties" : { "architecture" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "containerized" : { "type" : "boolean" }, "hostname" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "mac" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "os" : { "properties" : { "codename" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "family" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "kernel" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "platform" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } }, "http" : { "properties" : { "request" : { "properties" : { "bytes" : { "type" : "long" }, "headers" : { "properties" : { "content-length" : { "type" : "long" } } }, "method" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "response" : { "properties" : { "body" : { "properties" : { "bytes" : { "type" : "long" } } }, "bytes" : { "type" : "long" }, "headers" : { "properties" : { "content-length" : { "type" : "long" }, "content-type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "status_code" : { "type" : "long" }, "status_phrase" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "method" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "network" : { "properties" : { "bytes" : { "type" : "long" }, "community_id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "direction" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "packets" : { "type" : "long" }, "protocol" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "transport" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "related" : { "properties" : { "hosts" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "resource" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "server" : { "properties" : { "bytes" : { "type" : "long" }, "domain" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "port" : { "type" : "long" } } }, "source" : { "properties" : { "bytes" : { "type" : "long" }, "ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "mac" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "packets" : { "type" : "long" }, "port" : { "type" : "long" } } }, "status" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "tags" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "url" : { "properties" : { "domain" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "extension" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "full" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "path" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "port" : { "type" : "long" }, "scheme" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "user_agent" : { "properties" : { "original" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } } } }