Skip to content

Add support for wildcard_query function #156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 35 commits into from
Nov 25, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
a96a740
Implemented wildcard_query and added tests in core
Nov 2, 2022
885f04c
Implemented and added tests for sql
Nov 2, 2022
4d0dadb
Implemented and added tests for ppl
Nov 3, 2022
9c25e66
Implemented and added tests for lucene
Nov 3, 2022
efe0ee5
Fixed test for like expression
Nov 3, 2022
bedf662
Added parameters to wildcard_query
Nov 4, 2022
afbeb44
Added integration tests for ppl and sql
Nov 4, 2022
e25d4da
Added docs for doctests
Nov 4, 2022
84fcf88
Pulled from 2.x
Nov 7, 2022
4708ae2
Fixed issues introduced during merging
Nov 7, 2022
e05e5c5
Addressed PR comment
Nov 7, 2022
988b40b
Added annotation that was deleted from merging
Nov 8, 2022
5c225ce
Fixed merge conflict issues
Nov 8, 2022
0b9752c
Addressed some PR comments and handled escaping wildcards
Nov 9, 2022
a2ca906
Added tests for wildcard conversion and created data for testing
Nov 11, 2022
68c97d6
Added javadoc
Nov 14, 2022
a873b27
Changed index name
Nov 14, 2022
518c09f
Temporarily changed jackson_version to run GH actions
Nov 14, 2022
1101ca3
Added comparison test for wildcard conversion
Nov 14, 2022
fc8883c
Removed PPL implementation of wildcard_query
Nov 14, 2022
ae46f96
Reverted ppl docs change
Nov 14, 2022
a9f5be7
Made namedArgument a static function
Nov 14, 2022
bba63ce
Removed extra space
Nov 14, 2022
8b7b442
Merge branch 'integ-add-wildcardquery' of github.com:Bit-Quill/opense…
Nov 15, 2022
4af9eef
Fixed LIKE query
Nov 17, 2022
57aac8e
Fixed LIKE tests and added more tests
Nov 18, 2022
0a4af9d
Addressed PR comments
Nov 18, 2022
288e29c
Implemented converting text field to keyword. Still needs testing
Nov 21, 2022
1fb4973
Added test cases for LIKE in sql and ppl
Nov 22, 2022
34ffe13
Addressed PR comments regarding docs
Nov 24, 2022
eed3294
Fixed backslashes in docs
Nov 24, 2022
86fa737
Added missed backticks in docs
Nov 24, 2022
b54a934
Moved escaping wildcard test to common/utils
Nov 25, 2022
fede123
Fixed merge conflicts
Nov 25, 2022
9da6140
Fixed checkstyle error
Nov 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions docs/user/dql/functions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3137,17 +3137,17 @@ Description

``wildcard_query(field_expression, query_expression[, option=<option_value>]*)``

The wildcard_query function maps to the wildcard_query query used in search engine. It returns documents that match provided text in the specified field.
Supported wildcard characters can be found here: https://opensearch.org/docs/latest/opensearch/query-dsl/term/#wildcards
SQL wildcard % is converted to * as well as _ to ?. You may include % and _ in the search by escaping with a backslash prefix.
The `wildcard_query` function maps to the `wildcard_query` query used in search engine. It returns documents that match provided text in the specified field.
OpenSearch supports wildcard characters `*` and `?`. See the full description here: https://opensearch.org/docs/latest/opensearch/query-dsl/term/#wildcards.
SQL wildcard character `%` is converted to `*`. SQL character wildcard `_` is converted to `?`. You may include a backslash `\` to escape wildcard characters.

Available parameters include:

- boost
- case_insensitive
- rewrite

For backward compatibility, wildcardquery is also supported and mapped to wildcard_query query as well.
For backward compatibility, `wildcardquery` is also supported and mapped to `wildcard_query` query as well.

Example with only ``field`` and ``query`` expressions, and all other parameters are set default values::

Expand Down Expand Up @@ -3180,3 +3180,4 @@ Another example to show how to set custom values for the optional parameters::
| test wildcard in _ the middle of the text |
| test wildcard __ beside each other |
+-------------------------------------------+

88 changes: 88 additions & 0 deletions integ-test/src/test/java/org/opensearch/sql/ppl/LikeQueryIT.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/


package org.opensearch.sql.ppl;

import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_WILDCARD;
import static org.opensearch.sql.util.MatcherUtils.rows;
import static org.opensearch.sql.util.MatcherUtils.verifyDataRows;

import java.io.IOException;
import org.json.JSONObject;
import org.junit.Test;

public class LikeQueryIT extends PPLIntegTestCase {

@Override
public void init() throws IOException {
loadIndex(Index.WILDCARD);
}

@Test
public void test_like_with_percent() throws IOException {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ I like how you split tests by scenario.

String query = "source=" + TEST_INDEX_WILDCARD + " | WHERE Like(KeywordBody, 'test wildcard%') | fields KeywordBody";
JSONObject result = executeQuery(query);
verifyDataRows(result,
rows("test wildcard"),
rows("test wildcard in the end of the text%"),
rows("test wildcard in % the middle of the text"),
rows("test wildcard %% beside each other"),
rows("test wildcard in the end of the text_"),
rows("test wildcard in _ the middle of the text"),
rows("test wildcard __ beside each other"));
}

@Test
public void test_like_with_escaped_percent() throws IOException {
String query = "source=" + TEST_INDEX_WILDCARD + " | WHERE Like(KeywordBody, '\\\\%test wildcard%') | fields KeywordBody";
JSONObject result = executeQuery(query);
verifyDataRows(result,
rows("%test wildcard in the beginning of the text"));
}

@Test
public void test_like_in_where_with_escaped_underscore() throws IOException {
String query = "source=" + TEST_INDEX_WILDCARD + " | WHERE Like(KeywordBody, '\\\\_test wildcard%') | fields KeywordBody";
JSONObject result = executeQuery(query);
verifyDataRows(result,
rows("_test wildcard in the beginning of the text"));
}

@Test
public void test_like_on_text_field_with_one_word() throws IOException {
String query = "source=" + TEST_INDEX_WILDCARD + " | WHERE Like(TextBody, 'test*') | fields TextBody";
JSONObject result = executeQuery(query);
assertEquals(9, result.getInt("total"));
}

@Test
public void test_like_on_text_keyword_field_with_one_word() throws IOException {
String query = "source=" + TEST_INDEX_WILDCARD + " | WHERE Like(TextKeywordBody, 'test*') | fields TextKeywordBody";
JSONObject result = executeQuery(query);
assertEquals(8, result.getInt("total"));
}

@Test
public void test_like_on_text_keyword_field_with_greater_than_one_word() throws IOException {
String query = "source=" + TEST_INDEX_WILDCARD + " | WHERE Like(TextKeywordBody, 'test wild*') | fields TextKeywordBody";
JSONObject result = executeQuery(query);
assertEquals(7, result.getInt("total"));
}

@Test
public void test_like_on_text_field_with_greater_than_one_word() throws IOException {
String query = "source=" + TEST_INDEX_WILDCARD + " | WHERE Like(TextBody, 'test wild*') | fields TextBody";
JSONObject result = executeQuery(query);
assertEquals(0, result.getInt("total"));
}

@Test
public void test_convert_field_text_to_keyword() throws IOException {
String query = "source=" + TEST_INDEX_WILDCARD + " | WHERE Like(TextKeywordBody, '*') | fields TextKeywordBody";
String result = explainQueryToString(query);
assertTrue(result.contains("TextKeywordBody.keyword"));
}
}
140 changes: 140 additions & 0 deletions integ-test/src/test/java/org/opensearch/sql/sql/LikeQueryIT.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/


package org.opensearch.sql.sql;

import org.json.JSONObject;
import org.junit.Test;
import org.opensearch.sql.legacy.SQLIntegTestCase;

import java.io.IOException;

import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_WILDCARD;
import static org.opensearch.sql.util.MatcherUtils.rows;
import static org.opensearch.sql.util.MatcherUtils.verifyDataRows;

public class LikeQueryIT extends SQLIntegTestCase {
@Override
protected void init() throws Exception {
loadIndex(Index.WILDCARD);
}

@Test
public void test_like_in_select() throws IOException {
String query = "SELECT KeywordBody, KeywordBody LIKE 'test wildcard%' FROM " + TEST_INDEX_WILDCARD;
JSONObject result = executeJdbcRequest(query);
verifyDataRows(result,
rows("test wildcard", true),
rows("test wildcard in the end of the text%", true),
rows("%test wildcard in the beginning of the text", false),
rows("test wildcard in % the middle of the text", true),
rows("test wildcard %% beside each other", true),
rows("test wildcard in the end of the text_", true),
rows("_test wildcard in the beginning of the text", false),
rows("test wildcard in _ the middle of the text", true),
rows("test wildcard __ beside each other", true),
rows("test backslash wildcard \\_", false));
}

@Test
public void test_like_in_select_with_escaped_percent() throws IOException {
String query = "SELECT KeywordBody, KeywordBody LIKE '\\\\%test wildcard%' FROM " + TEST_INDEX_WILDCARD;
JSONObject result = executeJdbcRequest(query);
verifyDataRows(result,
rows("test wildcard", false),
rows("test wildcard in the end of the text%", false),
rows("%test wildcard in the beginning of the text", true),
rows("test wildcard in % the middle of the text", false),
rows("test wildcard %% beside each other", false),
rows("test wildcard in the end of the text_", false),
rows("_test wildcard in the beginning of the text", false),
rows("test wildcard in _ the middle of the text", false),
rows("test wildcard __ beside each other", false),
rows("test backslash wildcard \\_", false));
}

@Test
public void test_like_in_select_with_escaped_underscore() throws IOException {
String query = "SELECT KeywordBody, KeywordBody LIKE '\\\\_test wildcard%' FROM " + TEST_INDEX_WILDCARD;
JSONObject result = executeJdbcRequest(query);
verifyDataRows(result,
rows("test wildcard", false),
rows("test wildcard in the end of the text%", false),
rows("%test wildcard in the beginning of the text", false),
rows("test wildcard in % the middle of the text", false),
rows("test wildcard %% beside each other", false),
rows("test wildcard in the end of the text_", false),
rows("_test wildcard in the beginning of the text", true),
rows("test wildcard in _ the middle of the text", false),
rows("test wildcard __ beside each other", false),
rows("test backslash wildcard \\_", false));
}

@Test
public void test_like_in_where() throws IOException {
String query = "SELECT KeywordBody FROM " + TEST_INDEX_WILDCARD + " WHERE KeywordBody LIKE 'test wildcard%'";
JSONObject result = executeJdbcRequest(query);
verifyDataRows(result,
rows("test wildcard"),
rows("test wildcard in the end of the text%"),
rows("test wildcard in % the middle of the text"),
rows("test wildcard %% beside each other"),
rows("test wildcard in the end of the text_"),
rows("test wildcard in _ the middle of the text"),
rows("test wildcard __ beside each other"));
}

@Test
public void test_like_in_where_with_escaped_percent() throws IOException {
String query = "SELECT KeywordBody FROM " + TEST_INDEX_WILDCARD + " WHERE KeywordBody LIKE '\\\\%test wildcard%'";
JSONObject result = executeJdbcRequest(query);
verifyDataRows(result,
rows("%test wildcard in the beginning of the text"));
}

@Test
public void test_like_in_where_with_escaped_underscore() throws IOException {
String query = "SELECT KeywordBody FROM " + TEST_INDEX_WILDCARD + " WHERE KeywordBody LIKE '\\\\_test wildcard%'";
JSONObject result = executeJdbcRequest(query);
verifyDataRows(result,
rows("_test wildcard in the beginning of the text"));
}

@Test
public void test_like_on_text_field_with_one_word() throws IOException {
String query = "SELECT * FROM " + TEST_INDEX_WILDCARD + " WHERE TextBody LIKE 'test*'";
JSONObject result = executeJdbcRequest(query);
assertEquals(9, result.getInt("total"));
}

@Test
public void test_like_on_text_keyword_field_with_one_word() throws IOException {
String query = "SELECT * FROM " + TEST_INDEX_WILDCARD + " WHERE TextKeywordBody LIKE 'test*'";
JSONObject result = executeJdbcRequest(query);
assertEquals(8, result.getInt("total"));
}

@Test
public void test_like_on_text_keyword_field_with_greater_than_one_word() throws IOException {
String query = "SELECT * FROM " + TEST_INDEX_WILDCARD + " WHERE TextKeywordBody LIKE 'test wild*'";
JSONObject result = executeJdbcRequest(query);
assertEquals(7, result.getInt("total"));
}

@Test
public void test_like_on_text_field_with_greater_than_one_word() throws IOException {
String query = "SELECT * FROM " + TEST_INDEX_WILDCARD + " WHERE TextBody LIKE 'test wild*'";
JSONObject result = executeJdbcRequest(query);
assertEquals(0, result.getInt("total"));
}

@Test
public void test_convert_field_text_to_keyword() throws IOException {
String query = "SELECT * FROM " + TEST_INDEX_WILDCARD + " WHERE TextKeywordBody LIKE '*'";
String result = explainQuery(query);
assertTrue(result.contains("TextKeywordBody.keyword"));
}
}
Loading