Skip to content

Commit 33f86e7

Browse files
original-brownbearoriginal-brownbear
committed
Merge remote-tracking branch 'elastic/main' into track-length-on-buffered-index-input
2 parents 9d3a593 + c2a3ec4 commit 33f86e7

File tree

295 files changed

+6890
-2040
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

295 files changed

+6890
-2040
lines changed

‎docs/changelog/107121.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 107121
2+
summary: Add a flag to re-enable writes on the final index after an ILM shrink action.
3+
area: ILM+SLM
4+
type: enhancement
5+
issues:
6+
- 106599

‎docs/changelog/107178.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 107178
2+
summary: "Add support for Azure OpenAI embeddings to inference service"
3+
area: Machine Learning
4+
type: feature
5+
issues: [ ]

‎docs/changelog/107272.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 107272
2+
summary: "ESQL: extend BUCKET with spans"
3+
area: ES|QL
4+
type: enhancement
5+
issues: []

‎docs/changelog/107358.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 107358
2+
summary: Check node shutdown before fail
3+
area: Transform
4+
type: enhancement
5+
issues:
6+
- 100891

‎docs/changelog/107411.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 107411
2+
summary: Invalidating cross cluster API keys requires `manage_security`
3+
area: Security
4+
type: enhancement
5+
issues: []

‎docs/changelog/107447.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 107447
2+
summary: "Fix regression in get index settings (human=true) where the version was not displayed in human-readable format"
3+
area: Infra/Core
4+
type: bug
5+
issues: []

‎docs/changelog/107449.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 107449
2+
summary: Leverage ordinals in enrich lookup
3+
area: ES|QL
4+
type: enhancement
5+
issues: []

‎docs/changelog/107467.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 107467
2+
summary: "[Connector API] Fix bug with filtering validation toXContent"
3+
area: Application
4+
type: bug
5+
issues: []

‎docs/changelog/107537.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 107537
2+
summary: "Revert \"[ES|QL] Moving argument compatibility checking for Equals\""
3+
area: ES|QL
4+
type: bug
5+
issues: []

‎docs/reference/aggregations/bucket/random-sampler-aggregation.asciidoc

+12-1
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,17 @@ higher sampling rates, the relative error is still low.
9494

9595
NOTE: This represents the result of aggregations against a typical positively skewed APM data set which also has outliers in the upper tail. The linear dependence of the relative error on the sample size is found to hold widely, but the slope depends on the variation in the quantity being aggregated. As such, the variance in your own data may
9696
cause relative error rates to increase or decrease at a different rate.
97+
[[random-sampler-consistency]]
98+
==== Random sampler consistency
99+
100+
For a given `probability` and `seed`, the random sampler aggregation is consistent when sampling unchanged data from the same shard.
101+
However, this is background random sampling if a particular document is included in the sampled set or not is dependent on current number of segments.
102+
103+
Meaning, replica vs. primary shards could return different values as different particular documents are sampled.
104+
105+
If the shard changes in via doc addition, update, deletion, or segment merging, the particular documents sampled could change, and thus the resulting statistics could change.
106+
107+
The resulting statistics used from the random sampler aggregation are approximate and should be treated as such.
97108

98109
[[random-sampler-special-cases]]
99110
==== Random sampling special cases
@@ -105,6 +116,6 @@ for a bucket is `10,000` with `probability: 0.1`, the actual number of documents
105116

106117
An exception to this is <<search-aggregations-metrics-cardinality-aggregation, cardinality aggregation>>. Unique item
107118
counts are not suitable for automatic scaling. When interpreting the cardinality count, compare it
108-
to the number of sampled docs provided in the top level `doc_count` within the random_sampler aggregation. It gives
119+
to the number of sampled docs provided in the top level `doc_count` within the random_sampler aggregation. It gives
109120
you an idea of unique values as a percentage of total values. It may not reflect, however, the exact number of unique values
110121
for the given field.

‎docs/reference/esql/esql-async-query-api.asciidoc

+2-1
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ POST /_query/async
2424
| SORT year
2525
| LIMIT 5
2626
""",
27-
"wait_for_completion_timeout": "2s"
27+
"wait_for_completion_timeout": "2s",
28+
"version": "2024.04.01"
2829
}
2930
----
3031
// TEST[setup:library]

‎docs/reference/esql/esql-get-started.asciidoc

+3-3
Original file line numberDiff line numberDiff line change
@@ -248,22 +248,22 @@ For example, to create hourly buckets for the data on October 23rd:
248248

249249
[source,esql]
250250
----
251-
include::{esql-specs}/date.csv-spec[tag=gs-bucket]
251+
include::{esql-specs}/bucket.csv-spec[tag=gs-bucket]
252252
----
253253

254254
Combine `BUCKET` with <<esql-stats-by>> to create a histogram. For example,
255255
to count the number of events per hour:
256256

257257
[source,esql]
258258
----
259-
include::{esql-specs}/date.csv-spec[tag=gs-bucket-stats-by]
259+
include::{esql-specs}/bucket.csv-spec[tag=gs-bucket-stats-by]
260260
----
261261

262262
Or the median duration per hour:
263263

264264
[source,esql]
265265
----
266-
include::{esql-specs}/date.csv-spec[tag=gs-bucket-stats-by-median]
266+
include::{esql-specs}/bucket.csv-spec[tag=gs-bucket-stats-by-median]
267267
----
268268

269269
[discrete]

‎docs/reference/esql/esql-query-api.asciidoc

+7-2
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ POST /_query
1616
| STATS MAX(page_count) BY year
1717
| SORT year
1818
| LIMIT 5
19-
"""
19+
""",
20+
"version": "2024.04.01"
2021
}
2122
----
2223
// TEST[setup:library]
@@ -76,7 +77,11 @@ For syntax, refer to <<esql-locale-param>>.
7677
<<esql-rest-params>>.
7778

7879
`query`::
79-
(Required, object) {esql} query to run. For syntax, refer to <<esql-syntax>>.
80+
(Required, string) {esql} query to run. For syntax, refer to <<esql-syntax>>.
81+
82+
`version`::
83+
(Required, string) {esql} language version. Can be sent in short or long form, e.g.
84+
`2024.04.01` or `2024.04.01.🚀`. See <<esql-version>> for details.
8085

8186
[discrete]
8287
[role="child_attributes"]

‎docs/reference/esql/esql-rest.asciidoc

+16-8
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ The <<esql-query-api,{esql} query API>> accepts an {esql} query string in the
1616
----
1717
POST /_query?format=txt
1818
{
19-
"query": "FROM library | KEEP author, name, page_count, release_date | SORT page_count DESC | LIMIT 5"
19+
"query": "FROM library | KEEP author, name, page_count, release_date | SORT page_count DESC | LIMIT 5",
20+
"version": "2024.04.01"
2021
}
2122
----
2223
// TEST[setup:library]
@@ -55,7 +56,8 @@ POST /_query?format=txt
5556
| KEEP author, name, page_count, release_date
5657
| SORT page_count DESC
5758
| LIMIT 5
58-
"""
59+
""",
60+
"version": "2024.04.01"
5961
}
6062
----
6163
// TEST[setup:library]
@@ -143,7 +145,8 @@ POST /_query?format=txt
143145
"lte": 200
144146
}
145147
}
146-
}
148+
},
149+
"version": "2024.04.01"
147150
}
148151
----
149152
// TEST[setup:library]
@@ -179,7 +182,8 @@ POST /_query?format=json
179182
| SORT page_count DESC
180183
| LIMIT 5
181184
""",
182-
"columnar": true
185+
"columnar": true,
186+
"version": "2024.04.01"
183187
}
184188
----
185189
// TEST[setup:library]
@@ -226,7 +230,8 @@ POST /_query
226230
| EVAL birth_date = date_parse(birth_date_string)
227231
| EVAL month_of_birth = DATE_FORMAT("MMMM",birth_date)
228232
| LIMIT 5
229-
"""
233+
""",
234+
"version": "2024.04.01"
230235
}
231236
----
232237
// TEST[setup:library]
@@ -249,7 +254,8 @@ POST /_query
249254
| STATS count = COUNT(*) by year
250255
| WHERE count > 0
251256
| LIMIT 5
252-
"""
257+
""",
258+
"version": "2024.04.01"
253259
}
254260
----
255261
// TEST[setup:library]
@@ -270,7 +276,8 @@ POST /_query
270276
| WHERE count > ?
271277
| LIMIT 5
272278
""",
273-
"params": [300, "Frank Herbert", 0]
279+
"params": [300, "Frank Herbert", 0],
280+
"version": "2024.04.01"
274281
}
275282
----
276283
// TEST[setup:library]
@@ -304,7 +311,8 @@ POST /_query/async
304311
| SORT year
305312
| LIMIT 5
306313
""",
307-
"wait_for_completion_timeout": "2s"
314+
"wait_for_completion_timeout": "2s",
315+
"version": "2024.04.01"
308316
}
309317
----
310318
// TEST[setup:library]

‎docs/reference/esql/esql-using.asciidoc

+4
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,12 @@ Using {esql} to query across multiple clusters.
1818
<<esql-task-management>>::
1919
Using the <<tasks,task management API>> to list and cancel {esql} queries.
2020

21+
<<esql-version>>::
22+
Information about {esql} language versions.
23+
2124
include::esql-rest.asciidoc[]
2225
include::esql-kibana.asciidoc[]
2326
include::esql-security-solution.asciidoc[]
2427
include::esql-across-clusters.asciidoc[]
2528
include::task-management.asciidoc[]
29+
include::esql-version.asciidoc[]

‎docs/reference/esql/esql-version.asciidoc

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
[[esql-version]]
2+
=== {esql} language versions
3+
4+
++++
5+
<titleabbrev>Language versions</titleabbrev>
6+
++++
7+
8+
[discrete]
9+
[[esql-versions-released]]
10+
==== Released versions
11+
12+
* Version `2024.04.01`
13+
14+
[discrete]
15+
[[esql-versions-explanation]]
16+
=== How versions work
17+
18+
{esql} language versions are independent of {es} versions.
19+
Versioning the language ensures that your queries will always
20+
remain valid, independent of new {es} and {esql} releases. And it lets us
21+
evolve ESQL as we learn more from people using it. We don't plan to make
22+
huge changes to it, but we know we've made mistakes and we don't want those
23+
to live forever.
24+
25+
For instance, the following query will remain valid, even if a future
26+
version of {esql} introduces syntax changes or changes how the used
27+
commands or functions work.
28+
29+
[source,console]
30+
----
31+
POST /_query?format=txt
32+
{
33+
"version": "2024.04.01",
34+
"query": """
35+
FROM library
36+
| EVAL release_month = DATE_TRUNC(1 month, release_date)
37+
| KEEP release_month
38+
| SORT release_month ASC
39+
| LIMIT 3
40+
"""
41+
}
42+
----
43+
// TEST[setup:library]
44+
45+
We won't make breaking changes to released {esql} versions and
46+
versions will remain supported until they are deprecated.
47+
New features, bug fixes, and performance improvements
48+
will be continue to be added to released {esql} versions,
49+
provided they do not involve breaking changes.

‎docs/reference/esql/functions/bucket.asciidoc

+11-11
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,11 @@ in monthly buckets:
3535

3636
[source.merge.styled,esql]
3737
----
38-
include::{esql-specs}/date.csv-spec[tag=docsBucketMonth]
38+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketMonth]
3939
----
4040
[%header.monospaced.styled,format=dsv,separator=|]
4141
|===
42-
include::{esql-specs}/date.csv-spec[tag=docsBucketMonth-result]
42+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketMonth-result]
4343
|===
4444

4545
The goal isn't to provide *exactly* the target number of buckets, it's to pick a
@@ -51,11 +51,11 @@ Combine `BUCKET` with
5151

5252
[source.merge.styled,esql]
5353
----
54-
include::{esql-specs}/date.csv-spec[tag=docsBucketMonthlyHistogram]
54+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketMonthlyHistogram]
5555
----
5656
[%header.monospaced.styled,format=dsv,separator=|]
5757
|===
58-
include::{esql-specs}/date.csv-spec[tag=docsBucketMonthlyHistogram-result]
58+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketMonthlyHistogram-result]
5959
|===
6060

6161
NOTE: `BUCKET` does not create buckets that don't match any documents.
@@ -66,11 +66,11 @@ at most 100 buckets in a year results in weekly buckets:
6666

6767
[source.merge.styled,esql]
6868
----
69-
include::{esql-specs}/date.csv-spec[tag=docsBucketWeeklyHistogram]
69+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketWeeklyHistogram]
7070
----
7171
[%header.monospaced.styled,format=dsv,separator=|]
7272
|===
73-
include::{esql-specs}/date.csv-spec[tag=docsBucketWeeklyHistogram-result]
73+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketWeeklyHistogram-result]
7474
|===
7575

7676
NOTE: `BUCKET` does not filter any rows. It only uses the provided range to
@@ -83,11 +83,11 @@ salary histogram:
8383

8484
[source.merge.styled,esql]
8585
----
86-
include::{esql-specs}/ints.csv-spec[tag=docsBucketNumeric]
86+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketNumeric]
8787
----
8888
[%header.monospaced.styled,format=dsv,separator=|]
8989
|===
90-
include::{esql-specs}/ints.csv-spec[tag=docsBucketNumeric-result]
90+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketNumeric-result]
9191
|===
9292

9393
Unlike the earlier example that intentionally filters on a date range, you
@@ -102,17 +102,17 @@ per hour:
102102

103103
[source.styled,esql]
104104
----
105-
include::{esql-specs}/date.csv-spec[tag=docsBucketLast24hr]
105+
include::{esql-specs}/bucket.csv-spec[tag=docsBucketLast24hr]
106106
----
107107

108108
Create monthly buckets for the year 1985, and calculate the average salary by
109109
hiring month:
110110

111111
[source.merge.styled,esql]
112112
----
113-
include::{esql-specs}/date.csv-spec[tag=bucket_in_agg]
113+
include::{esql-specs}/bucket.csv-spec[tag=bucket_in_agg]
114114
----
115115
[%header.monospaced.styled,format=dsv,separator=|]
116116
|===
117-
include::{esql-specs}/date.csv-spec[tag=bucket_in_agg-result]
117+
include::{esql-specs}/bucket.csv-spec[tag=bucket_in_agg-result]
118118
|===

‎docs/reference/esql/functions/description/bucket.asciidoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22

33
*Description*
44

5-
Creates human-friendly buckets and returns a datetime value for each row that corresponds to the resulting bucket the row falls into.
5+
Creates groups of values - buckets - out of a datetime or numeric input. The size of the buckets can either be provided directly, or chosen based on a recommended count and values range.

‎docs/reference/esql/functions/description/from_base64.asciidoc

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
// This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
*Description*
4+
5+
Decode a base64 string.

‎docs/reference/esql/functions/description/to_base64.asciidoc

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
// This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
*Description*
4+
5+
Encode a string to a base64 string.

‎docs/reference/esql/functions/examples/from_base64.asciidoc

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
// This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
2+
3+
*Example*
4+
5+
[source.merge.styled,esql]
6+
----
7+
include::{esql-specs}/string.csv-spec[tag=from_base64]
8+
----
9+
[%header.monospaced.styled,format=dsv,separator=|]
10+
|===
11+
include::{esql-specs}/string.csv-spec[tag=from_base64-result]
12+
|===
13+

0 commit comments

Comments
 (0)