Add compare_ip operator udfs #3821
Conversation
Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Xinyu Hao <[email protected]>
ishaoxy
requested review from
ps48,
kavithacm,
derek-ho,
joshuali925,
dai-chen,
YANG-DB,
mengweieric,
Swiddis,
penghuo,
seankao-az,
MaxKsyunz,
Yury-Fridlyand,
anirudha,
forestmvey,
acarbonetto,
GumpacG,
ykmr1224,
LantaoJin,
noCharger and
qianheng-aws
as code owners
Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Xinyu Hao <[email protected]>
|
I think we should implement After that, for Ip comparison, we can convert string to ExprIP. And then we can implement some generic comparison functions for types of comparable, via calling their But current approach is still acceptable if we won't have other UDT in the future. |
| // UserDefinedFunctionUtils.convertRelDataTypeToSqlTypeName | ||
| return UDFOperandMetadata.wrap(OperandTypes.STRING_STRING); | ||
| return UDFOperandMetadata.wrap( |
There was a problem hiding this comment.
Maybe we should use new PPLTypeChecker here and override its checkOperandTypes method to only allow accept string or ip.
| @@ -400,6 +400,10 @@ private static List<ExprType> getExprTypes(SqlTypeFamily family) { | |||
| OpenSearchTypeFactory.TYPE_FACTORY.createSqlType(SqlTypeName.INTEGER)); | |||
| case ANY, IGNORE -> List.of( | |||
| OpenSearchTypeFactory.TYPE_FACTORY.createSqlType(SqlTypeName.ANY)); | |||
| // We borrow SqlTypeFamily.NULL to represent EXPR_IP. This is a workaround | |||
There was a problem hiding this comment.
If we stop use CompositeOperandTypeChecker and create a new TypeChecker ourself, maybe we can avoid using SqlTypeFamily.NULL to represent EXPR_IP and check operands on RelDataType level.
| @@ -614,6 +614,14 @@ public PPLTypeChecker getTypeChecker() { | |||
| } | |||
|
|
|||
| void populate() { | |||
| // register operators for IP comparing | |||
| registerOperator(NOTEQUAL, PPLBuiltinOperators.NOT_EQUALS_IP); | |||
There was a problem hiding this comment.
Questions, Do we need to support IP as UDT in PPL engine?
In OpenSearch PPL, IP handling depends on index field type. CIDR-based filtering should use ip_range queries for ip fields, script pushdown / in-memory processing for keyword, text, and runtime string fields.
| Field Type | Use Case | Expectation |
|---|---|---|
| IP Field | search index=log ip="192.168.0.0/16" | Rewrite as term query |
| search index | where cidrmatch("192.168.0.0/16", ip) | Rewrite as term query | |
| Keyword Field | search index=log ip="192.168.0.0/16" | Rewrite as term query, extactally keyword match |
| search index | where cidrmatch("192.168.0.0/16", ip) | Script pushdown — ip field is a string, not rewrite as term query. | |
| Text Field | search index=log ip="192.168.0.0/16" | Rewrite as query_string query, full text search |
| search index | where cidrmatch("192.168.0.0/16", ip) | Script pushdown — ip field is a string, not rewrite as term query. | |
| Runtime Field | search index=log | parse ip=regex(...)| where ip="192.168.0.0/16" | Script pushdown — ip field is a string, it is a string comparsion query |
| search index=log | parse ip=regex(...)| where cidrmatch("192.168.0.0/16", ip) | Script pushdown — ip field is a string, not rewrite as term query. |
Signed-off-by: Xinyu Hao <[email protected]>
Handled. |
Signed-off-by: Xinyu Hao <[email protected]>
Description
added six udfs related to IP comparison.
modify ip.sqlTypeName from OTHER to NULL in type checker.
Related Issues
Resolves #3776
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.