[backport to 2.19-dev] Backport calcite prs #3752
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xinyual
changed the title
xinyual
force-pushed
the
backportForSignoff
branch
from
5fbc9ed to
69763f6
Compare
|
The link checker failure seems due to |
|
…t#3110) Signed-off-by: currantw <[email protected]> Signed-off-by: xinyual <[email protected]>
…ject#3075) Signed-off-by: Norman Jordan <[email protected]> Signed-off-by: normanj-bitquill <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: xinyual <[email protected]>
* Add trendline (With SWA) PPL command --------- Signed-off-by: James Duong <[email protected]> Signed-off-by: Andrew Carbonetto <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: xinyual <[email protected]>
Signed-off-by: currantw <[email protected]> * Add support for casting (`cast(field_name to ip)`) and remove existing unused sorting syntax. Signed-off-by: currantw <[email protected]> * Update comparison logic to compare in IPv6 Signed-off-by: currantw <[email protected]> * Fix bug casting to IP Signed-off-by: currantw <[email protected]> * Fix failing tests Signed-off-by: currantw <[email protected]> * Assert that comparison only valid if same type, update tests accordingly Signed-off-by: currantw <[email protected]> * Add additional tests to increase code coverage Signed-off-by: currantw <[email protected]> * Integrate `cidrmatch` changes Signed-off-by: currantw <[email protected]> * Remove `OpenSearchIPType` data type Signed-off-by: currantw <[email protected]> * Fix more failing tests Signed-off-by: currantw <[email protected]> * Minor cleanup Signed-off-by: currantw <[email protected]> * Add new tests for IP data type to `SortCommandIT`, and update `weblogs` test data. Signed-off-by: currantw <[email protected]> * Fixing IT test failure. Signed-off-by: currantw <[email protected]> * Spotless and update test to sort in SQL Signed-off-by: currantw <[email protected]> * Fix broken link Signed-off-by: currantw <[email protected]> * Fix failing code coverage Signed-off-by: currantw <[email protected]> * Fix failing doctest Signed-off-by: currantw <[email protected]> * Fix failing `ip.rst` doctest Signed-off-by: currantw <[email protected]> * Fix test failure due to merge. Signed-off-by: currantw <[email protected]> * Fix spotless Signed-off-by: currantw <[email protected]> * Add missing `url` field Signed-off-by: currantw <[email protected]> * Address minor review comments. Signed-off-by: currantw <[email protected]> * Revert sort syntax changes Signed-off-by: currantw <[email protected]> * Minor doc update Signed-off-by: currantw <[email protected]> * FIx failing `ip.rst` doctest Signed-off-by: currantw <[email protected]> * Add `IPComparisonIT` tests for comparison operators, rename modules and weblogs test index to make plural for consistency. Signed-off-by: currantw <[email protected]> --------- Signed-off-by: currantw <[email protected]> Signed-off-by: xinyual <[email protected]>
…er (opensearch-project#3249) * First commit for Calcite integration Signed-off-by: Lantao Jin <[email protected]> * disable java security manager in IT Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: xinyual <[email protected]>
…ject#3258) * [POC] Make Calcite execute successfully Signed-off-by: Heng Qian <[email protected]> * [POC] Change caching schema to simple schema and avoid registering table when visitRelation. Signed-off-by: Heng Qian <[email protected]> * spotlessApply Signed-off-by: Heng Qian <[email protected]> * address comments Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: xinyual <[email protected]>
* Make basic aggregation working (partial) Signed-off-by: Lantao Jin <[email protected]> * add a settings to enable calcite Signed-off-by: Lantao Jin <[email protected]> * add more UTs Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: xinyual <[email protected]>
opensearch-project#3327) * Support Filter and Project pushdown Signed-off-by: Heng Qian <[email protected]> * Support Filter and Project pushdown v2 Signed-off-by: Heng Qian <[email protected]> * Address comments Signed-off-by: Heng Qian <[email protected]> * Add original license for PredicateAnalyzer Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: xinyual <[email protected]>
* Build integration test framework Signed-off-by: Lantao Jin <[email protected]> * make local work Signed-off-by: Lantao Jin <[email protected]> * Fix the timestamp issue Signed-off-by: Lantao Jin <[email protected]> * address comments Signed-off-by: Lantao Jin <[email protected]> * fix java style and rename CalcitePPLTestCase back to CalcitePPLIntegTestCase Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: xinyual <[email protected]>
…nsearch-project#3349) Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: xinyual <[email protected]>
…oject#3355) * Add more aggregation tests Signed-off-by: Lantao Jin <[email protected]> * delete unrelavant code Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: xinyual <[email protected]>
* Transform to calcite plan before executing Signed-off-by: Heng Qian <[email protected]> * Fix bug for single column row Signed-off-by: Heng Qian <[email protected]> * Add settings for calcite pushdown Signed-off-by: Heng Qian <[email protected]> * Lazily construct OpenSearchRequestBuilder and do push down Signed-off-by: Heng Qian <[email protected]> * Address comments and disable push down Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: xinyual <[email protected]>
Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: xinyual <[email protected]>
Signed-off-by: Heng Qian <[email protected]> Signed-off-by: xinyual <[email protected]>
* Fix PredicateAnalyzer for in and notIn Signed-off-by: Heng Qian <[email protected]> * Change text field to keyword since we don't support push down for that type Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: xinyual <[email protected]>
…3371) Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
…3376) * [BugFix] Fix text field push down Signed-off-by: Heng Qian <[email protected]> * Ignore CalciteSortCommandIT.testSortWithNullValue Signed-off-by: Heng Qian <[email protected]> * Refine code: only get keyword subfield for termQuery builder Signed-off-by: Heng Qian <[email protected]> * Refine code Signed-off-by: Heng Qian <[email protected]> * remove ignore tests in CalcitePPLInSubqueryIT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: xinyual <[email protected]>
* add udf/udaf interface and take/sqrt function Signed-off-by: xinyual <[email protected]> * add UT Signed-off-by: xinyual <[email protected]> * add POW, Atan, Atan2 and corresponding UT Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> * fix table for join it Signed-off-by: xinyual <[email protected]> * add java doc Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]>
…ry impl (opensearch-project#3377) Signed-off-by: xinyual <[email protected]>
…ect#3378) Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: xinyual <[email protected]>
…t#3388) Signed-off-by: xinyual <[email protected]>
…t#3392) * Implement ppl scalar subquery command with Calcite Signed-off-by: Lantao Jin <[email protected]> * more general subquery checker Signed-off-by: Lantao Jin <[email protected]> * support correlated IN subquery Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: xinyual <[email protected]>
* Change push down to logical index scan Signed-off-by: Heng Qian <[email protected]> * Support Aggregate Push Down Signed-off-by: Heng Qian <[email protected]> * Rebase and resolve conflict Signed-off-by: Heng Qian <[email protected]> * Add TODO Signed-off-by: Heng Qian <[email protected]> * Address comments Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: xinyual <[email protected]>
* add string udfs Signed-off-by: xinyual <[email protected]> * add it to string Signed-off-by: xinyual <[email protected]> * add IT for string function Signed-off-by: xinyual <[email protected]> * remove change for local test Signed-off-by: xinyual <[email protected]> * revert change Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
LantaoJin
requested review from
ps48,
kavithacm,
derek-ho,
joshuali925,
dai-chen,
YANG-DB,
mengweieric,
Swiddis,
penghuo,
seankao-az,
MaxKsyunz,
Yury-Fridlyand,
anirudha,
forestmvey,
acarbonetto,
GumpacG and
ykmr1224
as code owners
LantaoJin
previously approved these changes
Jun 12, 2025
|
@penghuo please double confirm the "?" item in #3752 (comment) and merge it. |
#3715, not required. |
Signed-off-by: xinyual <[email protected]>
|
@qianheng-aws Please review it. |
|
@noCharger Do we need to backport async-query-core to 2.19-dev? 2 related PRs |
It's good to have them on 2.19 to benefit people using 2.19 cluster. Not must-have though. |
penghuo
approved these changes
Jun 13, 2025
LantaoJin
approved these changes
Jun 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The PR backport all calcite feature PRs and some pre-required PR to 2.19-dev.
The PR is based on 2.19.0-SNAPSHOT and delete maven-publish to avoid to publish to maven repo.
Some dependencies' version are fixed from core
Backporting all PRs in main branch Starting from #3249 to now.
It also includes some PRs not related to calcite but as pre-required to simplify conflict resolve:
#3036
#3032
#3071
#3145
#3230
#3246
Remaining PR from #3249 not backported
#3256 CI: Fix broken url for link-checker CI build
#3254 Unused PPL EARLIEST and LATEST aggregations
#3255 CI: BWC testsuite
#3244 CI: Geo-Spatial Plugin integration test (need for geo IP)
#3259 CI: Syntax fix for Jekyll build
#3228 PPL command expression implementation for geoip(need for geo ip)
#3274 Update PPL parse command doc with limitation
#3299 Fixed week of week based year handling
#3300 [Release 2.19] Add release notes for 2.19
#3294 Build: Centralise dependencies version - Pt1
#3304 Add other functions to SQL query validator
#3306 [v3.0.0] Remove SparkSQL support
#3243 PPL: Add json function and cast(x as json) function
#2891 Remove dependency from async-query-core to datasources
#3326 Remove opendistro settings and endpoints
#3319 [Release 3.0] Bump gradle 8.10.2 / JDK23 / 3.0.0.0-alpha1 on SQL plugin
#3337 [v3.0.0] Deprecate SQL Delete statement
#3278 Clean up syntax error reporting
#3367 [v3.0.0] Deprecate OpenSearch DSL format
#3414 Add release notes on version 3.0.0.0-alpha1
#3434 Merge main for OpenSearch 3.0 release.
#3469 Set bouncycastle version inline
#3484 CVE-2024-57699 High: Fix json-smart vulnerability
#3489 Bump SQL main to version 3.0.0.0-beta1
#3515 Fix: CSV handling of embedded crlf
#3539 Fix build due to phasing off SecurityManager usage in favor of Java Agent
#3269 Add SQLQuery Utils support for Vaccum queries
#3551 Using java-agent gradle plugin to phase off Security M anager in favor of Java-agent.
#3442 Fix issue 2489 [backported]
#3649 create a new directory org/opensearch/direct-query/
#3671 [Fix CVE-2025-47273]Bump setuptools to 78.1.1 [backported]
#3604 Implement geoip udf with Calcite (need for geo ip) [backported]
#3660 Fix: Long IN-lists causes crash [backported]
#3715 Modified workflow: Grammar Files & Async Query Core
#3588 [AUTO] Increment version to 3.1.0-SNAPSHOT
#3589 Remove beta1 qualifier
#3592 Add release notes on version 3.0.0.0
The difference in functionality between this PR and main:
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.