Skip to content

adding ignore_malformed field parameter #9513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions _field-types/mapping-parameters/ignore-malformed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
---
layout: default
title: Ignore malformed
parent: Mapping parameters
grand_parent: Mapping and field types
nav_order: 45
has_children: false
has_toc: false
---

# Ignore malformed

The `ignore_malformed` mapping parameter instructs the indexing engine to ignore values that do not match the field's expected format. When enabled, malformed values are not indexed, preventing entire-document rejection because of data format issues. This ensures that documents are still stored even if one or more fields contain data that cannot be parsed.

By default, `ignore_malformed` is disabled, which means that if a value cannot be parsed according to the field type, indexing will fail for the entire document.

## Example: ignore_malformed off

Create an index named `people_no_ignore` containing an `age` field of type `integer`. By default, `ignore_malformed` is set to `false`:

```json
PUT /people_no_ignore
{
"mappings": {
"properties": {
"age": {
"type": "integer"
}
}
}
}
```
{% include copy-curl.html %}

Index a document with a malformed value:

```json
PUT /people_no_ignore/_doc/1
{
"age": "twenty"
}
```
{% include copy-curl.html %}

The request fails because of the malformed value:

```json
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "failed to parse field [age] of type [integer] in document with id '1'. Preview of field's value: 'twenty'"
}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse field [age] of type [integer] in document with id '1'. Preview of field's value: 'twenty'",
"caused_by": {
"type": "number_format_exception",
"reason": "For input string: \"twenty\""
}
},
"status": 400
}
```

## Example: ignore_malformed on

Create an index named `people_ignore` in which the `age` field has `ignore_malformed` set to `true`:

```json
PUT /people_ignore
{
"mappings": {
"properties": {
"age": {
"type": "integer",
"ignore_malformed": true
}
}
}
}
```
{% include copy-curl.html %}

Index a document with a malformed value:

```json
PUT /people_ignore/_doc/1
{
"age": "twenty"
}
```
{% include copy-curl.html %}

Retrieve the document:

```json
GET /people_ignore/_doc/1
```
{% include copy-curl.html %}

The response shows that the document was indexed successfully, despite having a malformed value:

```json
{
"_index": "people_ignore",
"_id": "1",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"found": true,
"_source": {
"age": "twenty"
}
}
```


2 changes: 1 addition & 1 deletion _field-types/mapping-parameters/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ Parameter | Description
`enabled` | Specifies whether the field is enabled or disabled. Default value is `true`, which means that the field is enabled. Allowed values are `true` or `false`.
`format` | Specifies the date format for date fields. There is no default value for this parameter. Allowed values are any valid date format string, such as `yyyy-MM-dd` or `epoch_millis`.
`ignore_above` | Skips indexing values that exceed the specified length. Default value is `2147483647`, which means that there is no limit on the field value length. Allowed values are any positive integer.
`ignore_malformed` | Specifies whether malformed values should be ignored. Default value is `false`, which means that malformed values are not ignored. Allowed values are `true` or `false`.
[`ignore_malformed`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/ignore-malformed/) | Specifies whether malformed values should be ignored. Default value is `false`, which means that malformed values are not ignored. Allowed values are `true` or `false`.
`index` | Specifies whether a field should be indexed. Default value is `true`, which means that the field is indexed. Allowed values are `true` or `false`.
`index_options` | Specifies what information should be stored in an index for scoring purposes. Default value is `docs`, which means that only the document numbers are stored in the index. Allowed values are `docs`, `freqs`, `positions`, or `offsets`.
Loading