Merge branch 'main' into source-field-optimization

Signed-off-by: Michael Froh <[email protected]>

Author: msfroh

.ci/bwcVersions

@@ -42,4 +42,5 @@ BWC_VERSION:
   - "2.17.2"
   - "2.18.0"
   - "2.18.1"
-  - "2.19.0"
+  - "2.19.0"
+  - "2.20.0"

.github/workflows/dependabot_pr.yml

@@ -23,10 +23,10 @@ jobs:
           token: ${{ steps.github_app_token.outputs.token }}
 
       # See please https://docs.gradle.org/8.10/userguide/upgrading_version_8.html#minimum_daemon_jvm_version
-      - name: Set up JDK 17
+      - name: Set up JDK 21
         uses: actions/setup-java@v4
         with:
-          java-version: 17
+          java-version: 21
           distribution: temurin
 
       - name: Update Gradle SHAs

CHANGELOG-3.0.md

@@ -11,8 +11,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Implement on behalf of token passing for extensions ([#8679](https://github.com/opensearch-project/OpenSearch/pull/8679), [#10664](https://github.com/opensearch-project/OpenSearch/pull/10664))
 - Provide service accounts tokens to extensions ([#9618](https://github.com/opensearch-project/OpenSearch/pull/9618))
 - GHA to verify checklist items completion in PR descriptions ([#10800](https://github.com/opensearch-project/OpenSearch/pull/10800))
+- [WLM] Add WLM support for search scroll API ([#16981](https://github.com/opensearch-project/OpenSearch/pull/16981))
 - Allow to pass the list settings through environment variables (like [], ["a", "b", "c"], ...) ([#10625](https://github.com/opensearch-project/OpenSearch/pull/10625))
 - Views, simplify data access and manipulation by providing a virtual layer over one or more indices ([#11957](https://github.com/opensearch-project/OpenSearch/pull/11957))
+- Added pull-based Ingestion (APIs, for ingestion source, a Kafka plugin, and IngestionEngine that pulls data from the ingestion source) ([#16958](https://github.com/opensearch-project/OpenSearch/pull/16958))
 
 ### Dependencies
 - Update Apache Lucene to 10.1.0 ([#16366](https://github.com/opensearch-project/OpenSearch/pull/16366))
@@ -27,8 +29,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add task completion count in search backpressure stats API ([#10028](https://github.com/opensearch-project/OpenSearch/pull/10028/))
 - Deprecate CamelCase `PathHierarchy` tokenizer name in favor to lowercase `path_hierarchy` ([#10894](https://github.com/opensearch-project/OpenSearch/pull/10894))
 - Breaking change: Do not request "search_pipelines" metrics by default in NodesInfoRequest ([#12497](https://github.com/opensearch-project/OpenSearch/pull/12497))
-- Refactor `:libs` module `bootstrap` package to eliminate top level split packages [#17117](https://github.com/opensearch-project/OpenSearch/pull/17117))
 - Use simpler matching logic for source fields when explicit field names (no wildcards or dot-paths) are specified ([#17160](https://github.com/opensearch-project/OpenSearch/pull/17160))
+- Refactor `:libs` module `bootstrap` package to eliminate top level split packages for JPMS support [#17117](https://github.com/opensearch-project/OpenSearch/pull/17117))
+- Refactor the codebase to eliminate top level split packages for JPMS support. [#17153](https://github.com/opensearch-project/OpenSearch/pull/17153))
+
 ### Deprecated
 
 ### Removed

CHANGELOG.md

@@ -24,6 +24,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add stats for remote publication failure and move download failure stats to remote methods([#16682](https://github.com/opensearch-project/OpenSearch/pull/16682/))
 - Update script supports java.lang.String.sha1() and java.lang.String.sha256() methods ([#16923](https://github.com/opensearch-project/OpenSearch/pull/16923))
 - Added a precaution to handle extreme date values during sorting to prevent `arithmetic_exception: long overflow` ([#16812](https://github.com/opensearch-project/OpenSearch/pull/16812)).
+- Add support for append only indices([#17039](https://github.com/opensearch-project/OpenSearch/pull/17039))
 - Add `verbose_pipeline` parameter to output each processor's execution details ([#16843](https://github.com/opensearch-project/OpenSearch/pull/16843)).
 - Add search replica stats to segment replication stats API ([#16678](https://github.com/opensearch-project/OpenSearch/pull/16678))
 - Introduce a setting to disable download of full cluster state from remote on term mismatch([#16798](https://github.com/opensearch-project/OpenSearch/pull/16798/))
@@ -39,6 +40,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Introduce Template query ([#16818](https://github.com/opensearch-project/OpenSearch/pull/16818))
 - Propagate the sourceIncludes and excludes fields from fetchSourceContext to FieldsVisitor. ([#17080](https://github.com/opensearch-project/OpenSearch/pull/17080))
 - [Star Tree] [Search] Resolving Date histogram with metric aggregation using star-tree ([#16674](https://github.com/opensearch-project/OpenSearch/pull/16674))
+- [Star Tree] [Search] Extensible design to support different query and field types ([#17137](https://github.com/opensearch-project/OpenSearch/pull/17137))
 
 ### Dependencies
 - Bump `com.google.cloud:google-cloud-core-http` from 2.23.0 to 2.47.0 ([#16504](https://github.com/opensearch-project/OpenSearch/pull/16504))
@@ -57,7 +59,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `codecov/codecov-action` from 4 to 5 ([#16667](https://github.com/opensearch-project/OpenSearch/pull/16667))
 - Bump `org.apache.logging.log4j:log4j-core` from 2.24.1 to 2.24.3 ([#16718](https://github.com/opensearch-project/OpenSearch/pull/16718), [#16858](https://github.com/opensearch-project/OpenSearch/pull/16858))
 - Bump `jackson` from 2.17.2 to 2.18.2 ([#16733](https://github.com/opensearch-project/OpenSearch/pull/16733))
-- Bump `ch.qos.logback:logback-classic` from 1.2.13 to 1.5.15 ([#16716](https://github.com/opensearch-project/OpenSearch/pull/16716), [#16898](https://github.com/opensearch-project/OpenSearch/pull/16898))
+- Bump `ch.qos.logback:logback-classic` from 1.2.13 to 1.5.16 ([#16716](https://github.com/opensearch-project/OpenSearch/pull/16716), [#16898](https://github.com/opensearch-project/OpenSearch/pull/16898), [#17133](https://github.com/opensearch-project/OpenSearch/pull/17133))
 - Bump `com.azure:azure-identity` from 1.13.2 to 1.14.2 ([#16778](https://github.com/opensearch-project/OpenSearch/pull/16778))
 - Bump Apache Lucene from 9.12.0 to 9.12.1 ([#16846](https://github.com/opensearch-project/OpenSearch/pull/16846))
 - Bump `com.gradle.develocity` from 3.18.2 to 3.19 ([#16855](https://github.com/opensearch-project/OpenSearch/pull/16855))
@@ -114,8 +116,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - The `phone-search` analyzer no longer emits the tel/sip prefix, international calling code, extension numbers and unformatted input as a token ([#16993](https://github.com/opensearch-project/OpenSearch/pull/16993))
 - Stop processing search requests when _msearch request is cancelled ([#17005](https://github.com/opensearch-project/OpenSearch/pull/17005))
 - Fix GRPC AUX_TRANSPORT_PORT and SETTING_GRPC_PORT settings and remove lingering HTTP terminology ([#17037](https://github.com/opensearch-project/OpenSearch/pull/17037))
+- [WLM] Fix the QueryGroupTask logging bug ([#17169](https://github.com/opensearch-project/OpenSearch/pull/17169))
 - Fix exists queries on nested flat_object fields throws exception ([#16803](https://github.com/opensearch-project/OpenSearch/pull/16803))
 - Use OpenSearch version to deserialize remote custom metadata([#16494](https://github.com/opensearch-project/OpenSearch/pull/16494))
+- Fix AutoDateHistogramAggregator rounding assertion failure ([#17023](https://github.com/opensearch-project/OpenSearch/pull/17023))
+- Add highlighting for wildcard search on `match_only_text` field ([#17101](https://github.com/opensearch-project/OpenSearch/pull/17101))
+- Fix the failing CI's with `Failed to load eclipse jdt formatter` error ([#17172](https://github.com/opensearch-project/OpenSearch/pull/17172))
 
 ### Security
 

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java

@@ -1188,10 +1188,6 @@ private void createConfiguration() {
         // Don't wait for state, just start up quickly. This will also allow new and old nodes in the BWC case to become the master
         baseConfig.put("discovery.initial_state_timeout", "0s");
 
-        // TODO: Remove these once https://github.com/elastic/elasticsearch/issues/46091 is fixed
-        baseConfig.put("logger.org.opensearch.action.support.master", "DEBUG");
-        baseConfig.put("logger.org.opensearch.cluster.coordination", "DEBUG");
-
         HashSet<String> overriden = new HashSet<>(baseConfig.keySet());
         overriden.retainAll(settings.keySet());
         OVERRIDABLE_SETTINGS.forEach(overriden::remove);

client/rest-high-level/src/main/java/org/opensearch/client/RequestConverters.java

@@ -946,14 +946,6 @@ Params withFields(String[] fields) {
             return this;
         }
 
-        /**
-         * @deprecated As of 2.0, because supporting inclusive language, replaced by {@link #withClusterManagerTimeout(TimeValue)}
-         */
-        @Deprecated
-        Params withMasterTimeout(TimeValue clusterManagerTimeout) {
-            return putParam("master_timeout", clusterManagerTimeout);
-        }
-
         Params withClusterManagerTimeout(TimeValue clusterManagerTimeout) {
             return putParam("cluster_manager_timeout", clusterManagerTimeout);
         }

client/rest-high-level/src/main/java/org/opensearch/client/TimedRequest.java

@@ -45,9 +45,6 @@ public abstract class TimedRequest implements Validatable {
 
     public static final TimeValue DEFAULT_ACK_TIMEOUT = timeValueSeconds(30);
     public static final TimeValue DEFAULT_CLUSTER_MANAGER_NODE_TIMEOUT = TimeValue.timeValueSeconds(30);
-    /** @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #DEFAULT_CLUSTER_MANAGER_NODE_TIMEOUT} */
-    @Deprecated
-    public static final TimeValue DEFAULT_MASTER_NODE_TIMEOUT = DEFAULT_CLUSTER_MANAGER_NODE_TIMEOUT;
 
     private TimeValue timeout = DEFAULT_ACK_TIMEOUT;
     private TimeValue clusterManagerTimeout = DEFAULT_CLUSTER_MANAGER_NODE_TIMEOUT;
@@ -68,16 +65,6 @@ public void setClusterManagerTimeout(TimeValue clusterManagerTimeout) {
         this.clusterManagerTimeout = clusterManagerTimeout;
     }
 
-    /**
-     * Sets the timeout to connect to the cluster-manager node
-     * @param clusterManagerTimeout timeout as a {@link TimeValue}
-     * @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #setClusterManagerTimeout(TimeValue)}
-     */
-    @Deprecated
-    public void setMasterTimeout(TimeValue clusterManagerTimeout) {
-        setClusterManagerTimeout(clusterManagerTimeout);
-    }
-
     /**
      * Returns the request timeout
      */
@@ -91,13 +78,4 @@ public TimeValue timeout() {
     public TimeValue clusterManagerNodeTimeout() {
         return clusterManagerTimeout;
     }
-
-    /**
-     * Returns the timeout for the request to be completed on the cluster-manager node
-     * @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #clusterManagerNodeTimeout()}
-     */
-    @Deprecated
-    public TimeValue masterNodeTimeout() {
-        return clusterManagerNodeTimeout();
-    }
 }

client/rest-high-level/src/main/java/org/opensearch/client/indices/GetComponentTemplatesRequest.java

@@ -71,25 +71,10 @@ public TimeValue getClusterManagerNodeTimeout() {
         return clusterManagerNodeTimeout;
     }
 
-    /**
-     * @return the timeout for waiting for the cluster-manager node to respond
-     * @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #getMasterNodeTimeout()}
-     */
-    @Deprecated
-    public TimeValue getMasterNodeTimeout() {
-        return getClusterManagerNodeTimeout();
-    }
-
     public void setClusterManagerNodeTimeout(@Nullable TimeValue clusterManagerNodeTimeout) {
         this.clusterManagerNodeTimeout = clusterManagerNodeTimeout;
     }
 
-    /** @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #setClusterManagerNodeTimeout(TimeValue)} */
-    @Deprecated
-    public void setMasterNodeTimeout(@Nullable TimeValue clusterManagerNodeTimeout) {
-        setClusterManagerNodeTimeout(clusterManagerNodeTimeout);
-    }
-
     public void setClusterManagerNodeTimeout(String clusterManagerNodeTimeout) {
         final TimeValue timeValue = TimeValue.parseTimeValue(
             clusterManagerNodeTimeout,
@@ -98,12 +83,6 @@ public void setClusterManagerNodeTimeout(String clusterManagerNodeTimeout) {
         setClusterManagerNodeTimeout(timeValue);
     }
 
-    /** @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #setClusterManagerNodeTimeout(String)} */
-    @Deprecated
-    public void setMasterNodeTimeout(String clusterManagerNodeTimeout) {
-        setClusterManagerNodeTimeout(clusterManagerNodeTimeout);
-    }
-
     /**
      * @return true if this request is to read from the local cluster state, rather than the cluster-manager node - false otherwise
      */

client/rest-high-level/src/main/java/org/opensearch/client/indices/GetComposableIndexTemplateRequest.java

@@ -71,25 +71,10 @@ public TimeValue getClusterManagerNodeTimeout() {
         return clusterManagerNodeTimeout;
     }
 
-    /**
-     * @return the timeout for waiting for the cluster-manager node to respond
-     * @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #getMasterNodeTimeout()}
-     */
-    @Deprecated
-    public TimeValue getMasterNodeTimeout() {
-        return getClusterManagerNodeTimeout();
-    }
-
     public void setClusterManagerNodeTimeout(@Nullable TimeValue clusterManagerNodeTimeout) {
         this.clusterManagerNodeTimeout = clusterManagerNodeTimeout;
     }
 
-    /** @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #setClusterManagerNodeTimeout(TimeValue)} */
-    @Deprecated
-    public void setMasterNodeTimeout(@Nullable TimeValue clusterManagerNodeTimeout) {
-        setClusterManagerNodeTimeout(clusterManagerNodeTimeout);
-    }
-
     public void setClusterManagerNodeTimeout(String clusterManagerNodeTimeout) {
         final TimeValue timeValue = TimeValue.parseTimeValue(
             clusterManagerNodeTimeout,
@@ -98,12 +83,6 @@ public void setClusterManagerNodeTimeout(String clusterManagerNodeTimeout) {
         setClusterManagerNodeTimeout(timeValue);
     }
 
-    /** @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #setClusterManagerNodeTimeout(String)} */
-    @Deprecated
-    public void setMasterNodeTimeout(String clusterManagerNodeTimeout) {
-        setClusterManagerNodeTimeout(clusterManagerNodeTimeout);
-    }
-
     /**
      * @return true if this request is to read from the local cluster state, rather than the cluster-manager node - false otherwise
      */

client/rest-high-level/src/main/java/org/opensearch/client/indices/GetIndexTemplatesRequest.java

@@ -90,25 +90,10 @@ public TimeValue getClusterManagerNodeTimeout() {
         return clusterManagerNodeTimeout;
     }
 
-    /**
-     * @return the timeout for waiting for the cluster-manager node to respond
-     * @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #getMasterNodeTimeout()}
-     */
-    @Deprecated
-    public TimeValue getMasterNodeTimeout() {
-        return getClusterManagerNodeTimeout();
-    }
-
     public void setClusterManagerNodeTimeout(@Nullable TimeValue clusterManagerNodeTimeout) {
         this.clusterManagerNodeTimeout = clusterManagerNodeTimeout;
     }
 
-    /** @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #setClusterManagerNodeTimeout(TimeValue)} */
-    @Deprecated
-    public void setMasterNodeTimeout(@Nullable TimeValue clusterManagerNodeTimeout) {
-        setClusterManagerNodeTimeout(clusterManagerNodeTimeout);
-    }
-
     public void setClusterManagerNodeTimeout(String clusterManagerNodeTimeout) {
         final TimeValue timeValue = TimeValue.parseTimeValue(
             clusterManagerNodeTimeout,
@@ -117,12 +102,6 @@ public void setClusterManagerNodeTimeout(String clusterManagerNodeTimeout) {
         setClusterManagerNodeTimeout(timeValue);
     }
 
-    /** @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #setClusterManagerNodeTimeout(String)} */
-    @Deprecated
-    public void setMasterNodeTimeout(String clusterManagerNodeTimeout) {
-        setClusterManagerNodeTimeout(clusterManagerNodeTimeout);
-    }
-
     /**
      * @return true if this request is to read from the local cluster state, rather than the cluster-manager node - false otherwise
      */

client/rest-high-level/src/test/java/org/opensearch/client/IndicesClientIT.java

@@ -134,6 +134,7 @@
 import java.util.Map;
 import java.util.stream.Collectors;
 
+import static org.opensearch.cluster.metadata.IndexMetadata.SETTING_CREATION_DATE;
 import static org.opensearch.cluster.metadata.IndexMetadata.SETTING_NUMBER_OF_REPLICAS;
 import static org.opensearch.cluster.metadata.IndexMetadata.SETTING_NUMBER_OF_SHARDS;
 import static org.opensearch.common.xcontent.support.XContentMapValues.extractRawValues;
@@ -256,6 +257,26 @@ public void testCreateIndex() throws IOException {
         }
     }
 
+    public void testCreateIndexFailPrivateSetting() throws IOException {
+        {
+            // Create index with private setting
+            String indexName = "private_index";
+            assertFalse(indexExists(indexName));
+
+            CreateIndexRequest createIndexRequest = new CreateIndexRequest(indexName);
+
+            Settings.Builder settings = Settings.builder();
+            settings.put(SETTING_CREATION_DATE, -1);
+            createIndexRequest.settings(settings);
+
+            OpenSearchStatusException exception = expectThrows(
+                OpenSearchStatusException.class,
+                () -> execute(createIndexRequest, highLevelClient().indices()::create, highLevelClient().indices()::createAsync)
+            );
+            assertTrue(exception.getMessage().contains("private index setting [index.creation_date] can not be set explicitly"));
+        }
+    }
+
     public void testGetSettings() throws IOException {
         String indexName = "get_settings_index";
         Settings basicSettings = Settings.builder().put("number_of_shards", 1).put("number_of_replicas", 0).build();
@@ -281,6 +302,24 @@ public void testGetSettings() throws IOException {
         assertEquals("30s", updatedResponse.getSetting(indexName, "index.refresh_interval"));
     }
 
+    public void testGetPrivateSettings() throws IOException {
+        String indexName = "get_settings_index";
+        Settings basicSettings = Settings.builder().put("number_of_shards", 1).put("number_of_replicas", 0).build();
+
+        createIndex(indexName, basicSettings);
+
+        GetSettingsRequest getSettingsRequest = new GetSettingsRequest().indices(indexName);
+        GetSettingsResponse getSettingsResponse = execute(
+            getSettingsRequest,
+            highLevelClient().indices()::getSettings,
+            highLevelClient().indices()::getSettingsAsync
+        );
+
+        assertNull(getSettingsResponse.getSetting(indexName, "index.refresh_interval"));
+        assertNotNull(getSettingsResponse.getSetting(indexName, "index.creation_date"));
+        assertNotNull(getSettingsResponse.getSetting(indexName, "index.uuid"));
+    }
+
     public void testGetSettingsNonExistentIndex() throws IOException {
         String nonExistentIndex = "index_that_doesnt_exist";
         assertFalse(indexExists(nonExistentIndex));

client/rest/src/main/java/org/opensearch/client/Node.java

@@ -226,15 +226,6 @@ public boolean isClusterManagerEligible() {
             return roles.contains("master") || roles.contains("cluster_manager");
         }
 
-        /**
-         * Returns whether or not the node <strong>could</strong> be elected cluster-manager.
-         * @deprecated As of 2.2, because supporting inclusive language, replaced by {@link #isClusterManagerEligible()}
-         */
-        @Deprecated
-        public boolean isMasterEligible() {
-            return isClusterManagerEligible();
-        }
-
         /**
          * Returns whether or not the node stores data.
          */

distribution/build.gradle

@@ -302,7 +302,7 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
           it.version = VersionProperties.getBundledJre(platform, architecture)
           it.vendor = VersionProperties.bundledJdkVendor
           it.architecture = architecture
-       }
+        }
       }
     }
   }
@@ -481,7 +481,7 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
         }
       }
     }
-    
+
     jreFiles = { Project project, String platform, String architecture ->
       return copySpec {
         /*