Add sanity test script #3564
Add sanity test script #3564
Conversation
Signed-off-by: Heng Qian <[email protected]>
qianheng-aws
requested review from
ps48,
kavithacm,
derek-ho,
joshuali925,
dai-chen,
YANG-DB,
mengweieric,
Swiddis,
penghuo,
seankao-az,
MaxKsyunz,
Yury-Fridlyand,
anirudha,
forestmvey,
acarbonetto,
GumpacG,
ykmr1224,
LantaoJin and
noCharger
as code owners
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
scripts/SanityTestScript/README.md
Outdated
| "start_time": "2025-04-22 16:30:22.461505", | ||
| "end_time": "2025-04-22 16:30:22.863651", | ||
| "error": { | ||
| "error": "404 Client Error: Not Found for url: http://k8s-calcitep-opensear-8312a971dd-1309739395.us-west-1.elb.amazonaws.com/_plugins/_ppl", |
There was a problem hiding this comment.
can you replace this url to localhost?
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
| @@ -0,0 +1,3 @@ | |||
| query_name,query,expected_status | |||
| Successful Demo,"source=opensearch_dashboards_sample_data_flights | patterns FlightDelayType | stats count() by patterns_field",SUCCESS | |||
There was a problem hiding this comment.
It was migrated from opensearch-spark repo and didn't support checking results currently. Plan to support such feature later.
There was a problem hiding this comment.
It bring me that we can assert results comparing to enable/disable Calcite in script.
| # Sanity Test Script | ||
|
|
||
| ### Description | ||
| This Python script executes test queries from a CSV file using an asynchronous query API and generates comprehensive test reports. |
There was a problem hiding this comment.
What is the difference with existing IT? When should we use SanityTest script?
There was a problem hiding this comment.
Without this script, we need to do sanity test manually by submitting PPL queries one by one and recording the results.
In the sanity test for 3.0.0, we uses the same PPL queries and datas as opensearch-spark https://github.com/opensearch-project/opensearch-spark/blob/main/integ-test/script/test_cases.csv. But we plans to build more factual tests next time, which should be the major difference from existing IT,
There was a problem hiding this comment.
Without this script, we need to do sanity test manually by submitting PPL queries one by one and recording the results.
We can run IT/YamlIT against remote cluster. ./gradlew ':integ-test:yamlRestTest' -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="yamlRestTest-0"
@dai-chen ask similar question in here.
My thoughts,
- ExistingIT can leave with IT.
- New command / functions, bug fix could migrate to YamlIT.
- The tests in the PR can be considered a comprehensive test set, which can be leveraged to verify PPL behavior across both Spark and OpenSearch.
There was a problem hiding this comment.
It seems yamlRestTest can satisfy our requirements of sanity test already, including the feature of results matching. What's your opinion? @LantaoJin
@penghuo By the way, does this test framework support concurrency testing?
There was a problem hiding this comment.
@penghuo 1. do we have any doc how to run yaml test (prepare data and queries)? 2. are yamlTest and sanity test equivalent? (can we sign-off the sanity requirement if yarmTest passed?)
There was a problem hiding this comment.
@penghuo By the way, does this test framework support concurrency testing?
Not sure, I think it should, borrow it from opensearch-core.
- do we have any doc how to run yaml test (prepare data and queries)?
data and query is self contained for each feature. for instance. in opensearch-core, each aggregation feature is sepereate file.
- are yamlTest and sanity test equivalent?
I think so, we can add sanityTest as yamlTest, and run yamlTest against remote cluster, then signoff.
Description
Add sanity test script for submitting queries in batch and generating test reports automatically.
How to use:
Related Issues
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.