fix broken build flag, move build to one directory #2442
Conversation
sam-herman
requested review from
heemin32,
navneet1v,
VijayanB,
vamshin,
jmazanec15,
naveentatikonda,
junqiu-lei,
martin-gaievski,
ryanbogan,
luyuncheng,
shatejas and
0ctopus13prime
as code owners
sam-herman
force-pushed
the
fix-syntax-typo-breaking-build
branch
from
3f12185 to
515c986
Compare
|
@navneet1v @vamshin @jmazanec15 can someone review this fix? Also, I am not sure why the 2 BWC checks are failing and can't re-run the workflows to get them to see if it's a transient things. Can you help me try re-running those? |
Triggered a re run. |
|
@sam-herman can you rebase your code from main and then we can trigger a run again. I see some changes have been made in main branch which fixes some issues. |
sam-herman
force-pushed
the
fix-syntax-typo-breaking-build
branch
from
25c356a to
53943ec
Compare
@navneet1v looks like after the rebase there are even more tests failing, including some of the non-bwc ones. maybe try re-run to check if it's transient? |
|
@sam-herman i think for main branch since 3.0 is in progress there are a lot of breaking changes. |
|
Tagging 3.0 release owner here to know when the main branch will be stable since a lot of prs are struck. |
| @@ -111,10 +111,10 @@ endif() | |||
| if(NOT DEFINED AVX512_SPR_ENABLED) | |||
| # Check if the system is Intel(R) Sapphire Rapids or a newer-generation processor | |||
| execute_process(COMMAND bash -c "lscpu | grep -q 'GenuineIntel' && lscpu | grep -i 'avx512_fp16' | grep -i 'avx512_bf16' | grep -i 'avx512_vpopcntdq'" OUTPUT_VARIABLE SPR_FLAGS OUTPUT_STRIP_TRAILING_WHITESPACE) | |||
| if (AND NOT "${SPR_FLAGS}" STREQUAL "") | |||
There was a problem hiding this comment.
Good catch @sam-herman . @mulugetam Can you check this and make sure this change makes sense?
There was a problem hiding this comment.
It is a valid change. I fixed it in 2.x and 2.19 but somehow missed backporting it to main
| @@ -333,7 +333,7 @@ task cmakeJniLib(type:Exec) { | |||
| workingDir 'jni' | |||
| def args = [] | |||
| args.add("cmake") | |||
| args.add(".") | |||
There was a problem hiding this comment.
Good update. This makes cleaning some much easier. That being said, in the buildJniLib, instead of changing workingDir, can we just do:
make -Cbuild
I worry if we add "/" we might end up having issues with path resolution on different platforms (unless gradle handles this for us)
There was a problem hiding this comment.
The windows build seems to work fine https://github.com/opensearch-project/k-NN/actions/runs/13020810027/job/36325044749?pr=2442
|
thanks @sam-herman. main is messed up right now due to lucene upgrade. On other PRs, weve been developing directly on 2.x and backporting to main to unblock (#2438 ) |
|
@navneet1v @Vikasht34 Any updates on fixes to BWC and integ tests? |
|
@sam-herman , Tentative changes to be merged in main by End of Next week . |
|
@sam-herman I think this looks good - if you want to expedite, you can change branch merging to to 2.x and then we can backport to main. |
Signed-off-by: Samuel Herman <[email protected]>
Signed-off-by: Samuel Herman <[email protected]>
Signed-off-by: Samuel Herman <[email protected]>
sam-herman
force-pushed
the
fix-syntax-typo-breaking-build
branch
from
f09882e to
0b86cea
Compare
Signed-off-by: Samuel Herman <[email protected]>
sam-herman
force-pushed
the
fix-syntax-typo-breaking-build
branch
from
0b86cea to
dd9a595
Compare
Signed-off-by: Samuel Herman <[email protected]>
|
@jmazanec15 @navneet1v I added a couple of fixes for issues I just spotted in main:
Everything is building now locally and working fine, however now all the integ tests of the build in the GitHub workflow seems to be failing to start a cluster even though it all works and passes locally. Do you know what could be the reason? |
|
@sam-herman From https://productionresultssa3.blob.core.windows.net/actions-results/181b3bc8-6d78-4be2-850b-157d78dc8cab/workflow-job-run-189e3ba7-4a1d-5257-bc22-4748799b99b3/logs/job/job-logs.txt?rsct=text%2Fplain&se=2025-02-10T16%3A39%3A47Z&sig=L0VyXZ3q6KC9fYlzYh5SqUaL0lv9fSCAbTzI6vHkMWM%3D&ske=2025-02-11T04%3A09%3A32Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2025-02-10T16%3A09%3A32Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&st=2025-02-10T16%3A29%3A42Z&sv=2025-01-05, Im seeing a link issue. @naveentatikonda Do you know what might be causing this? |
| } | ||
|
|
||
| task buildJniLib(type:Exec) { | ||
| dependsOn cmakeJniLib | ||
| workingDir 'jni' | ||
| commandLine 'make', 'opensearchknn_nmslib', 'opensearchknn_faiss', 'opensearchknn_common', '-j', "${nproc_count}" | ||
| def outputStream = new ByteArrayOutputStream() |
There was a problem hiding this comment.
Shall we move this block of code from line 376 to 393 into a separate gradle task like makeJniLib ?
There was a problem hiding this comment.
let me refactor that part, I think I can completely avoid the outputStream and just pull from System.getEnv
@jmazanec15 I think I see the issue: |
@sam-herman Yes, it could be the reason. But, does building the libraries inside build directory doesn't show the build progress and which libraries are being built in the logs ? If it doesn't show the build progress in the logs can we add a |
Yeah, I think there is another issue, I am able to repro locally now. It's because of the |
Signed-off-by: Samuel Herman <[email protected]>
|
@jmazanec15 @naveentatikonda can you guys take a look at the latest changes? |
There was a problem hiding this comment.
Looks good to me. Seems its breaking with:
2025-02-10T20:46:05.7887456Z » /__t/Java_Temurin-Hotspot_jdk/21.0.6-7.0.LTS/x64/bin/java: symbol lookup error: /__w/k-NN/k-NN/jni/release/libopensearchknn_nmslib.so: undefined symbol: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev
which is unrelated. Approved
@naveentatikonda @navneet1v can one of you approve as well so we can merge this? |
…t#2442) * move build to one directory, fix broken flag Signed-off-by: Samuel Herman <[email protected]> * fix make path Signed-off-by: Samuel Herman <[email protected]> * changelog update Signed-off-by: Samuel Herman <[email protected]> * add fix for classpath change and for cmake discovery on macos Signed-off-by: Samuel Herman <[email protected]> * fix make discovery for gradle Signed-off-by: Samuel Herman <[email protected]> * fix cmake path for macOS Signed-off-by: Samuel Herman <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]>
Signed-off-by: Peter Zhu <[email protected]>
Signed-off-by: Peter Zhu <[email protected]>
…vice (#2532) * fix broken build flag, move build to one directory (#2442) * move build to one directory, fix broken flag Signed-off-by: Samuel Herman <[email protected]> * fix make path Signed-off-by: Samuel Herman <[email protected]> * changelog update Signed-off-by: Samuel Herman <[email protected]> * add fix for classpath change and for cmake discovery on macos Signed-off-by: Samuel Herman <[email protected]> * fix make discovery for gradle Signed-off-by: Samuel Herman <[email protected]> * fix cmake path for macOS Signed-off-by: Samuel Herman <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> * Update package name to fix compilation issue (#2513) * Update package name to fix compilation issue Core renamed this package in opensearch-project/OpenSearch#17272 This commit renames package accordingly Signed-off-by: Balasubramanian <[email protected]> * Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions (#2506) (#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <[email protected]> --------- Signed-off-by: Balasubramanian <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <[email protected]> * Adds debug logs for KNNQuery and KNNWeight (#2466) (#2470) * Adds debug logs for KNNQuery and KNNWeight Signed-off-by: Tejas Shah <[email protected]> * Adds check to see if log is enabled to start and stop StopWatch Signed-off-by: Tejas Shah <[email protected]> * Addressing comments on the PR Signed-off-by: Tejas Shah <[email protected]> * Adds shard and segment info in the logs Signed-off-by: Tejas Shah <[email protected]> * Removes unnecessary segment name param from exact search Signed-off-by: Tejas Shah <[email protected]> * Fixes the build Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Tejas Shah <[email protected]> (cherry picked from commit f322e27) Co-authored-by: Tejas Shah <[email protected]> * Clean JNI artifacts with ./gradlew clean (#2516) * clean JNI artifacts with ./gradlew clean Signed-off-by: Samuel Herman <[email protected]> * nest release under build directory Signed-off-by: Samuel Herman <[email protected]> * adjust all references to the old release path Signed-off-by: Samuel Herman <[email protected]> * remove irrelevant paths from gitignore, add jni/build Signed-off-by: Samuel Herman <[email protected]> * fix logging on linux Signed-off-by: Samuel Herman <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> * Backport to main (#2520) * Remove skip building graph check for quantization use case (#2430) For quantization indices, we don't have to apply building graph check since it is already faster, this is now only applied for fp32/16 indices and where threshold is configured. Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update default to 0 to always build graph as default behavior (#2452) Signed-off-by: Balasubramanian <[email protected]> * Update changelog Signed-off-by: Balasubramanian <[email protected]> --------- Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Balasubramanian <[email protected]> * [Backport main] Add release notes for 2.19.0 (#2503) * Add release notes for 2.19.0 Signed-off-by: Kunal Kotwani <[email protected]> * Fix links for release notes Co-authored-by: John Mazanec <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> --------- Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: John Mazanec <[email protected]> * Fix main knnlib dir in build script based on #2442 (#2526) Signed-off-by: Peter Zhu <[email protected]> * Minor performance improvments in KNNQueryBuilder (#2528) Signed-off-by: Tejas Shah <[email protected]> * Initial implementation of control flow Signed-off-by: Rohan Chitale <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]> Signed-off-by: Balasubramanian <[email protected]> Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Peter Zhu <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Signed-off-by: Rohan Chitale <[email protected]> Co-authored-by: sam-herman <[email protected]> Co-authored-by: Vijayan Balasubramanian <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <[email protected]> Co-authored-by: Tejas Shah <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]> Co-authored-by: John Mazanec <[email protected]>
Description
Addresses a bug and house cleaning during build.
jnidirectory.Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.