Skip to content

Merge 2.x to main after 2.4 release #1063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 178 commits into from
Dec 13, 2022

Conversation

penghuo
Copy link
Collaborator

@penghuo penghuo commented Nov 9, 2022

Description

Merge 2.x to main after 2.4 release.

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-trigger-bot bot and others added 30 commits August 5, 2022 18:10
…oject#731)

Signed-off-by: Joshua Li <[email protected]>
(cherry picked from commit 1df89e3)

Co-authored-by: Joshua Li <[email protected]>
…project#759)

* Backport Spring and jackson version bump

Signed-off-by: Chen Dai <[email protected]>

* Remove BWC tests (opensearch-project#721)

Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Chen Dai <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>
…nsearch-project#753)

Signed-off-by: Yury Fridlyand <[email protected]>
(cherry picked from commit deececb)

Co-authored-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Joshua Li <[email protected]>
…h-project#776)

* Bump checkstyle version and fix violations

Signed-off-by: Chen Dai <[email protected]>

* Fix checkstyle violations in test code

Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Chen Dai <[email protected]>
(cherry picked from commit 8103d9f)

Co-authored-by: Chen Dai <[email protected]>
…earch-project#797)

* Change master timeout to new API

Signed-off-by: Chen Dai <[email protected]>

* Change param name to cluster_manager_timeout

Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Chen Dai <[email protected]>
(cherry picked from commit 4e40ed2)

Co-authored-by: Chen Dai <[email protected]>
**Reorganize GitHub workflows:**

* Remove obsoleve workflows, mostly related to releaseing ODFE artifacts.
* Update actions in rest workflows.
* Limit workflow scopes:
  * Reduce file scope, for example, JDBC workflow will be triggered on JDBC changes only.
  * Reduce action scope, so workflows shouldn't be triggered twice on PR.
* Strictly define env version to avoid unexpected failures.
* Upload test reports for SQL plugin workflow.
* Activate tests for SQL CLI.

Signed-off-by: Yury-Fridlyand <[email protected]>
* Add create index function in opensearch node client

Signed-off-by: Chen Dai <[email protected]>

* Add create index function in opensearch rest client

Signed-off-by: Chen Dai <[email protected]>

* Add create index function in OpenSearchIndex table

Signed-off-by: Chen Dai <[email protected]>

* Separate index exist API from create index function

Signed-off-by: Chen Dai <[email protected]>

* Separate table exist API from create table function

Signed-off-by: Chen Dai <[email protected]>

* Add UT and Todo comment for OpenSearch system index

Signed-off-by: Chen Dai <[email protected]>

* Fix broken UT in core due to new API in Table interface

Signed-off-by: Chen Dai <[email protected]>

* Add more UT and javadoc for OpenSearch data type mapping

Signed-off-by: Chen Dai <[email protected]>

* Rename isExist to exists which looks more readable

Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Chen Dai <[email protected]>
* Add new test workflow.

Signed-off-by: Yury-Fridlyand <[email protected]>
…ions (opensearch-project#754)

* Add implementation of `now`, `sysdate`, `localtime` and similar functions (opensearch-project#92)

Signed-off-by: Yury-Fridlyand <[email protected]>

* Rework on `now` function implementation (opensearch-project#113)

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor SQL ANTLR clean-up.

Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>
…n-2.13.4

Update com.fasterxml.jackson to 2.13.4 to match opensearch repo.
…ch-project#835)

* Add datetime functions `FROM_UNIXTIME` and `UNIX_TIMESTAMP` (opensearch-project#114)

* Add implementation for `FROM_UNIXTIME` and `UNIX_TIMESTAMP` functions, UT and IT.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Collent all DateTime formatters into one place.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Rename `DateFormatters` -> `DateTimeFormatters`.

Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
…nize-workflows

Reorganize GitHub workflows.
…ch-project#848)

* Added `CONVERT_TZ` to the PPL lexer/parser and the SQL lexer/parser.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added `convert_tz` to the BuiltinFunctionName.java.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added convert_tz to DateTimeFunction.java register, private FunctionResolver convert_tz and ExprValue exprConvert_TZ. It implements the functionality for converting between time zones.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added IT PPL and SQL tests for various conditions including time zones that are outside the existing range (consistent with MySQL standard). Added implementation for convert_tz consistent with MySQL implementation.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added DATETIME to OpenSearchPPLParser.g4 and OpenSearchSQLParser.g4.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Rebase merge conflict resolution.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added ppl doctest for convert_tz

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Completed implementation for datetime and convert_Tz.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed SQL test from PPL IT test

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed redundant convert to string.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed doctests and reverted changes to adddate function in DateTimeFunctionTest.java.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed doctest.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed doctest.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added additional integration tests for PPL and SQL tests for convert_tz.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added null for timezones outside of basic range for DATETIME.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added test cases for null with the datetime function. Made DateTime function call conevrt_tz function.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Seperated out test from DateTimeFunction.java.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Updated tests. Fixed exception to be less general.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed changes.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed rel timezone issue.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added support for non valid datetime to return null for convert_tz.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Made exception more verbose.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed unneeded format changes in DateTimeFunction.java.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added more doctests.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed formatting changes.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Reverting sql/ppl DateTimeFunctionsIT.java.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Reverted changes to DateTimeFunctionTest.java

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added more information about invalid date for convert_tz

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Converted "from Field" and "To Field" to use "Fieldn" where n is the field number.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added date validation. Added test cases in IT to cover cases. Added test in ConvertTZTest.java.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed date validation function. Broke up some unit tests.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed formatting.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added DateTime tests, broke up functions.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added space in DateTimeTest.java.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Tidied up code and tests.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed local date time rst test.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed nested try/catch exceptions.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* removed exprConvertTZ function call from within try catch statement.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Reverted change from parse localdate

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed extra casting around fromTz variable.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Updated wording for functions.rst

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Updated wording for datetime.rst to describe null for conert_tz

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added more test cases for functions.rst

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed doctests.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed extra import.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Renamed isValidTimeZone function to isValidMySqlTimeZoneId.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Has ExprDatetimeValue doing work for exprConvertTZ call.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Deleted fromTZ

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Moved fixed variables to top of class.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added Null to support of exprDateTime.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added missing expr functions for makedate/time

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* cleaning up after rebase merge.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Rebase merge conflict resolution.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed missing DATETIME in SQL Parser.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed IT test.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Addressed PR comments

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Added missing variables after rebase

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Fixed checkstyle errors after rebase.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Moved formatter for date time over.

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Changed function resolved to default

Signed-off-by: MitchellGale-BitQuill <[email protected]>

* Removed unneeded code

Signed-off-by: MitchellGale-BitQuill <[email protected]>

Signed-off-by: MitchellGale-BitQuill <[email protected]>
…earch-project#870)

* Do not remove opensearch from the list of registered catalogs.

Signed-off-by: MaxKsyunz <[email protected]>

* New Changes to handle bug:opensearch-project#866

Signed-off-by: vamsi-amazon <[email protected]>

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: vamsi-amazon <[email protected]>
Co-authored-by: MaxKsyunz <[email protected]>
penghuo and others added 12 commits November 24, 2022 18:38
…-project#1099)

Support opensearch-sql:run
Update developer_guide doc. add prometheus and filesystem description.
Add .java-version to .gitignore

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: vamsi-amazon <[email protected]>

Signed-off-by: vamsi-amazon <[email protected]>
…earch-project#1090)

* Added Unit Tests

Signed-off-by: GabeFernandez310 <[email protected]>

* Added Implementation

Signed-off-by: GabeFernandez310 <[email protected]>

* Added Integration Tests

Signed-off-by: GabeFernandez310 <[email protected]>

* Added Documentation

Signed-off-by: GabeFernandez310 <[email protected]>

* Made Changes To Address Failing Checkstyle And Fixed Documentation

Signed-off-by: GabeFernandez310 <[email protected]>

* Made Changes To Docs

Signed-off-by: GabeFernandez310 <[email protected]>

* Added IT Test To Compare Alternate Syntaxes

Signed-off-by: GabeFernandez310 <[email protected]>

* Added Unit Tests

Signed-off-by: GabeFernandez310 <[email protected]>

* Added Implementation

Signed-off-by: GabeFernandez310 <[email protected]>

* Added Integration Tests

Signed-off-by: GabeFernandez310 <[email protected]>

* Added Documentation

Signed-off-by: GabeFernandez310 <[email protected]>

* Made Changes To Address Failing Checkstyle And Fixed Documentation

Signed-off-by: GabeFernandez310 <[email protected]>

* Made Changes To Docs

Signed-off-by: GabeFernandez310 <[email protected]>

* Added IT Test To Compare Alternate Syntaxes

Signed-off-by: GabeFernandez310 <[email protected]>

* Simplified Documentation

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
* Add cbrt function to the PPL (opensearch-project#171)

Signed-off-by: Margarit Hakobyan <[email protected]>
…pensearch-project#1103)

* Added Tests

Signed-off-by: GabeFernandez310 <[email protected]>

* Changed Documentation

Signed-off-by: GabeFernandez310 <[email protected]>

* Added Implementation.

Signed-off-by: GabeFernandez310 <[email protected]>

* Modified Documentation

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
…t#1102)

* Adding alternate multi_match syntax.

Signed-off-by: forestmvey <[email protected]>

* Adding documentation for alternate multi-match syntax options.

Signed-off-by: forestmvey <[email protected]>

Signed-off-by: forestmvey <[email protected]>
POSITION function is a synonym to LOCATE. It's syntax is POSITION(<substr> IN <string>)

Signed-off-by: Margarit Hakobyan <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Comment on lines +27 to +38
"%\\{"
+ "(?<name>"
+ "(?<pattern>[A-z0-9]+)"
+ "(?::(?<subname>[A-z0-9_:;,\\-\\/\\s\\.']+))?"
+ ")"
+ "(?:=(?<definition>"
+ "(?:"
+ "(?:[^{}]+|\\.+)+"
+ ")+"
+ ")"
+ ")?"
+ "\\}");

Check failure

Code scanning / CodeQL

Inefficient regular expression

This part of the regular expression may cause exponential backtracking on strings starting with '%{{0=' and containing many repetitions of 'z'.
Comment on lines +27 to +38
"%\\{"
+ "(?<name>"
+ "(?<pattern>[A-z0-9]+)"
+ "(?::(?<subname>[A-z0-9_:;,\\-\\/\\s\\.']+))?"
+ ")"
+ "(?:=(?<definition>"
+ "(?:"
+ "(?:[^{}]+|\\.+)+"
+ ")+"
+ ")"
+ ")?"
+ "\\}");

Check failure

Code scanning / CodeQL

Inefficient regular expression

This part of the regular expression may cause exponential backtracking on strings starting with '%{{0=' and containing many repetitions of '.'.
public class GrokCompiler implements Serializable {

// We don't want \n and commented line
private static final Pattern patternLinePattern = Pattern.compile("^([A-z0-9_]+)\\s+(.*)$");

Check warning

Code scanning / CodeQL

Overly permissive regular expression range

Suspicious character range that is equivalent to \[A-Z\\[\\\\]^_`a-z\].
Comment on lines +27 to +38
"%\\{"
+ "(?<name>"
+ "(?<pattern>[A-z0-9]+)"
+ "(?::(?<subname>[A-z0-9_:;,\\-\\/\\s\\.']+))?"
+ ")"
+ "(?:=(?<definition>"
+ "(?:"
+ "(?:[^{}]+|\\.+)+"
+ ")+"
+ ")"
+ ")?"
+ "\\}");

Check warning

Code scanning / CodeQL

Overly permissive regular expression range

Suspicious character range that is equivalent to \[A-Z\\[\\\\]^_`a-z\].
Comment on lines +27 to +38
"%\\{"
+ "(?<name>"
+ "(?<pattern>[A-z0-9]+)"
+ "(?::(?<subname>[A-z0-9_:;,\\-\\/\\s\\.']+))?"
+ ")"
+ "(?:=(?<definition>"
+ "(?:"
+ "(?:[^{}]+|\\.+)+"
+ ")+"
+ ")"
+ ")?"
+ "\\}");

Check warning

Code scanning / CodeQL

Overly permissive regular expression range

Suspicious character range that is equivalent to \[A-Z\\[\\\\]^_`a-z\].
penghuo and others added 6 commits December 6, 2022 13:12
* Add position() string function to PPL (opensearch-project#184)

Signed-off-by: Margarit Hakobyan <[email protected]>
…pensearch-project#1091)

* Add new table scan builder and optimizer rules

Signed-off-by: Chen Dai <[email protected]>

* Fix jacoco test coverage

Signed-off-by: Chen Dai <[email protected]>

* Update javadoc with more details

Signed-off-by: Chen Dai <[email protected]>

* Fix highlight pushdown issue

Signed-off-by: Chen Dai <[email protected]>

* Rename new class more properly

Signed-off-by: Chen Dai <[email protected]>

* Fix default sort by doc issue

Signed-off-by: Chen Dai <[email protected]>

* Rename visit method and javadoc

Signed-off-by: Chen Dai <[email protected]>

* Move table scan builder and optimize rule to read package

Signed-off-by: Chen Dai <[email protected]>

* Fix sort push down issue

Signed-off-by: Chen Dai <[email protected]>

* Move sortByFields to parent scan builder

Signed-off-by: Chen Dai <[email protected]>

* Add back old test

Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Chen Dai <[email protected]>
* Add reverse() string function to V2 SQL engine

Signed-off-by: Margarit Hakobyan <[email protected]>
penghuo and others added 2 commits December 12, 2022 15:47
Signed-off-by: opensearch-ci-bot <[email protected]>

Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
@penghuo
Copy link
Collaborator Author

penghuo commented Dec 13, 2022

Pending on opensearch-project/opensearch-build#2999.

@penghuo penghuo marked this pull request as ready for review December 13, 2022 20:57
@penghuo penghuo requested a review from a team as a code owner December 13, 2022 20:57
@penghuo
Copy link
Collaborator Author

penghuo commented Dec 13, 2022

Ignore CodeQL issue. Tracking in #1168.

@penghuo penghuo merged commit 034027c into opensearch-project:main Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.