Skip to content

PPL: Add json function and cast(x as json) function #3243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 34 commits into from
Feb 11, 2025
Merged

PPL: Add json function and cast(x as json) function #3243

merged 34 commits into from
Feb 11, 2025

Conversation

acarbonetto
Copy link
Collaborator

@acarbonetto acarbonetto commented Jan 13, 2025 •
edited
Loading

Description

For OpenSearch-PPL, adds the json() and cast(expression as json) functions. Both of these functions can be used to convert a json-encoded string into an expression object. These functions return any expression type that is valid json (object, array, or scalar). Unfortunately, because the return type is unknown, the expression's return type is UNDEFINED.

The user can use cast to convert the types where necessary for scalar types. For example:

// converts json-encoded integer to an int expression
cast(json(json_string) as int)

The following casts are available to convert json-encoded scalars: int, long, double, float, boolean.

JSON
----------

Description
>>>>>>>>>>>

Usage: `json(value)` Evaluates whether a string can be parsed as a json-encoded string and casted as an expression. Returns the JSON value if valid, null otherwise.

Argument type: STRING

Return type: BOOLEAN/DOUBLE/INTEGER/NULL/STRUCT/ARRAY

Example::

    > source=json_test | where json_valid(json_string) | eval json=json(json_string) | fields test_name, json_string, json
    fetched rows / total rows = 4/4
    +---------------------+------------------------------+---------------+
    | test_name           | json_string                  | json          |
    |---------------------|------------------------------|---------------|
    | json object         | {"a":"1","b":"2"}            | {a:"1",b:"2"} |
    | json array          | [1, 2, 3, 4]                 | [1,2,3,4]     |
    | json scalar string  | "abc"                        | "abc"         |
    | json empty string   |                              | null          |
    +---------------------+------------------------------+---------------+

Related Issues

Resolves #3209

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@acarbonetto acarbonetto force-pushed the feature/acarbo_json_cast_ppl branch 2 times, most recently from 2ccf4ab to 3ec16e0 Compare January 15, 2025 18:33
kenrickyap
kenrickyap reviewed Jan 16, 2025
View reviewed changes
14yapkc1 and others added 5 commits January 16, 2025 15:55
kenrickyap
kenrickyap reviewed Jan 21, 2025
View reviewed changes
kenrickyap
kenrickyap reviewed Jan 21, 2025
View reviewed changes
Copy link
Contributor

@kenrickyap kenrickyap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM minor nits

@acarbonetto
Remove conflicted files
c6c6cc1 
Signed-off-by: Andrew Carbonetto <[email protected]>
kenrickyap
kenrickyap reviewed Jan 21, 2025
View reviewed changes
@acarbonetto
Add doctext row
a5652ea 
Signed-off-by: Andrew Carbonetto <[email protected]>
@kenrickyap kenrickyap mentioned this pull request Jan 23, 2025
Closed
7 tasks
YANG-DB
YANG-DB reviewed Jan 28, 2025
View reviewed changes
integ-test/src/test/resources/json_test.json Outdated
@@ -1,14 +1,24 @@
{"index":{"_id":"0"}}
{"test_name":"json nested object", "json_string":"{\"a\":\"1\",\"b\":{\"c\":\"2\",\"d\":\"3\"}}"}
{"test_name":"json nested object", "json_string":"{\"a\":\"1\", \"b\": {\"c\": \"3\"}, \"d\": [1, 2, 3]}"}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can u also add a use case for inner json obj that has an array ?
{"a":{"b":{"c":[....]}}}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"d" is mapped to an array. Do you want an object within in an array?

YANG-DB
YANG-DB reviewed Jan 28, 2025
View reviewed changes
Copy link
Member

@YANG-DB YANG-DB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO looking good - added some minor comments...

@acarbonetto
Adding more json tests
65faa90 
Signed-off-by: Andrew Carbonetto <[email protected]>
@acarbonetto
adding more json IT tests
25b0f90 
Signed-off-by: Andrew Carbonetto <[email protected]>
@acarbonetto
Copy link
Collaborator Author

IMO looking good - added some minor comments...

Tests added. Thank you for looking.

@YANG-DB
Copy link
Member

YANG-DB commented Jan 30, 2025

@acarbonetto can u plz resolve the conflicts ?
thanks

YANG-DB
YANG-DB previously approved these changes Feb 3, 2025
View reviewed changes
LantaoJin
LantaoJin reviewed Feb 11, 2025
View reviewed changes
core/src/main/java/org/opensearch/sql/utils/JsonUtils.java
jsonNode = objectMapper.readTree(json.stringValue());
} catch (JsonProcessingException e) {
final String errorFormat = "JSON string '%s' is not valid. Error details: %s";
throw new SemanticCheckException(String.format(errorFormat, json, e.getMessage()), e);
Copy link
Member

@LantaoJin LantaoJin Feb 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: SemanticCheckExcpetion is not a proper exception to throw here. Maybe change it to ExpressionEvaluationException (add a two-parameters construction)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Lantao. I will address shortly.

LantaoJin
LantaoJin approved these changes Feb 11, 2025
View reviewed changes
Copy link
Member

@LantaoJin LantaoJin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except a minor comment

@LantaoJin
Copy link
Member

Merging to main. Thanks for contribution. @acarbonetto please address this minor comment #3243 (comment) in your followup PR.

@LantaoJin LantaoJin merged commit 5aa86d7 into opensearch-project:main Feb 11, 2025
15 checks passed
@acarbonetto acarbonetto deleted the feature/acarbo_json_cast_ppl branch February 11, 2025 18:02
@penghuo penghuo mentioned this pull request Feb 19, 2025
Closed
@xinyual xinyual mentioned this pull request Jun 11, 2025
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LantaoJin LantaoJin LantaoJin approved these changes

@YANG-DB YANG-DB YANG-DB approved these changes

@kenrickyap kenrickyap kenrickyap approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE] Add json function to cast string to json object
5 participants
@acarbonetto @YANG-DB @LantaoJin @kenrickyap @14yapkc1