Add FIPS build tooling (#17907)

* Add FIPS build-tooling

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

# Conflicts:
#	CHANGELOG.md

# Conflicts:
#	CHANGELOG.md

* Update distribution/tools/plugin-cli/src/test/java/org/opensearch/tools/cli/plugin/InstallPluginCommandTests.java

Co-authored-by: Andriy Redko <[email protected]>
Signed-off-by: Iwan Igonin <[email protected]>

* Update server/src/main/java/org/opensearch/bootstrap/SecurityProviderManager.java

Co-authored-by: Andriy Redko <[email protected]>
Signed-off-by: Iwan Igonin <[email protected]>

* add additional '-Ptests.fips.enabled=true' parameter; set KEYSTORE_PASSWORD when running docker tests.

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

* prevent bc deps propagation outside their modules

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

* change TestUtilsTests to test for cryptographic equality

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

* use 'fipsOnly' for all BC declarations

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

* fix: do not allow other values than 'FIPS-140-3' for 'crypto.standard'

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

* exclude BC from client & server compile scope

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

* Refactor BootstrapForTesting & Randomness

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

* revert Randomness.java

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

# Conflicts:
#	server/src/main/java/org/opensearch/common/Randomness.java

* rename SecurityProviderManager#excludeSunJCE to SecurityProviderManager#removeNonCompliantFipsProviders

Signed-off-by: Igonin <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

---------

Signed-off-by: Iwan Igonin <[email protected]>
Signed-off-by: Igonin <[email protected]>
Co-authored-by: Igonin <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>
Co-authored-by: Benny Goerzig <[email protected]>
Co-authored-by: Karsten Schnitter <[email protected]>
Co-authored-by: Kai Sternad <[email protected]>

Author: 6 people

CHANGELOG.md

@@ -35,6 +35,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Added File Cache Stats - Involves Block level as well as full file level stats ([#17538](https://github.com/opensearch-project/OpenSearch/issues/17479))
 - Added File Cache Pinning ([#17617](https://github.com/opensearch-project/OpenSearch/issues/13648))
 - Support consumer reset in Resume API for pull-based ingestion. This PR includes a breaking change for the experimental pull-based ingestion feature. ([#18332](https://github.com/opensearch-project/OpenSearch/pull/18332))
+- Add FIPS build tooling ([#4254](https://github.com/opensearch-project/security/issues/4254))
 
 ### Changed
 - Create generic DocRequest to better categorize ActionRequests ([#18269](https://github.com/opensearch-project/OpenSearch/pull/18269)))

build.gradle

@@ -66,7 +66,6 @@ apply from: 'gradle/ide.gradle'
 apply from: 'gradle/forbidden-dependencies.gradle'
 apply from: 'gradle/formatting.gradle'
 apply from: 'gradle/local-distribution.gradle'
-apply from: 'gradle/fips.gradle'
 apply from: 'gradle/run.gradle'
 apply from: 'gradle/missing-javadoc.gradle'
 apply from: 'gradle/code-coverage.gradle'
@@ -427,13 +426,19 @@ gradle.projectsEvaluated {
 
     project.tasks.withType(Test) { task ->
       if (task != null) {
-        task.jvmArgs += ["--add-modules=jdk.incubator.vector"]
+        task.jvmArgs += [
+            "--add-modules=jdk.incubator.vector",
+            "--add-exports=java.base/com.sun.crypto.provider=ALL-UNNAMED"
+        ]
 
         // Add Java Agent for security sandboxing
         if (!(project.path in [':build-tools', ":libs:agent-sm:bootstrap", ":libs:agent-sm:agent"])) {
           dependsOn(project(':libs:agent-sm:agent').prepareAgent)
           jvmArgs += ["-javaagent:" + project(':libs:agent-sm:agent').jar.archiveFile.get()]
         }
+        if (BuildParams.inFipsJvm) {
+          task.jvmArgs += ["-Dorg.bouncycastle.fips.approved_only=true"]
+        }
       }
     }
 
@@ -703,6 +708,14 @@ allprojects {
   plugins.withId('lifecycle-base') {
     checkPart1.configure { dependsOn 'check' }
   }
+
+  plugins.withId('opensearch.testclusters') {
+    testClusters.configureEach {
+      if (BuildParams.inFipsJvm) {
+        keystorePassword 'notarealpasswordphrase'
+      }
+    }
+  }
 }
 
 subprojects {

buildSrc/src/main/java/org/opensearch/gradle/OpenSearchTestBasePlugin.java

@@ -161,11 +161,13 @@ public void execute(Task t) {
                 test.systemProperty("tests.seed", BuildParams.getTestSeed());
             }
 
-            var securityFile = "java.security";
-            test.systemProperty(
-                "java.security.properties",
-                project.getRootProject().getLayout().getProjectDirectory() + "/distribution/src/config/" + securityFile
-            );
+            if (BuildParams.isInFipsJvm()) {
+                test.systemProperty(
+                    "java.security.properties",
+                    project.getRootProject().getLayout().getProjectDirectory() + "/distribution/src/config/fips_java.security"
+                );
+            }
+
             // don't track these as inputs since they contain absolute paths and break cache relocatability
             File gradleHome = project.getGradle().getGradleUserHomeDir();
             String gradleVersion = project.getGradle().getGradleVersion();

buildSrc/src/main/java/org/opensearch/gradle/info/FipsBuildParams.java

@@ -0,0 +1,43 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.gradle.info;
+
+import java.util.function.Function;
+
+public class FipsBuildParams {
+
+    @Deprecated
+    public static final String FIPS_BUILD_PARAM_FOR_TESTS = "tests.fips.enabled";
+    public static final String FIPS_BUILD_PARAM = "crypto.standard";
+    public static final String DEFAULT_FIPS_MODE = "FIPS-140-3";
+
+    private static String fipsMode;
+
+    public static void init(Function<String, Object> fipsValue) {
+        var fipsBuildParamForTests = Boolean.parseBoolean((String) fipsValue.apply(FIPS_BUILD_PARAM_FOR_TESTS));
+        var fipsBuildParam = (String) fipsValue.apply(FIPS_BUILD_PARAM);
+
+        if (fipsBuildParamForTests || DEFAULT_FIPS_MODE.equals(fipsBuildParam)) {
+            fipsMode = DEFAULT_FIPS_MODE;
+        } else {
+            fipsMode = "any-supported";
+        }
+    }
+
+    private FipsBuildParams() {}
+
+    public static boolean isInFipsMode() {
+        return DEFAULT_FIPS_MODE.equals(fipsMode);
+    }
+
+    public static String getFipsMode() {
+        return fipsMode;
+    }
+
+}

buildSrc/src/main/java/org/opensearch/gradle/info/GlobalBuildInfoPlugin.java

@@ -109,6 +109,8 @@ public void apply(Project project) {
         File rootDir = project.getRootDir();
         GitInfo gitInfo = gitInfo(rootDir);
 
+        FipsBuildParams.init(project::findProperty);
+
         BuildParams.init(params -> {
             // Initialize global build parameters
             boolean isInternal = GlobalBuildInfoPlugin.class.getResource("/buildSrc.marker") != null;
@@ -129,7 +131,7 @@ public void apply(Project project) {
             params.setIsCi(System.getenv("JENKINS_URL") != null);
             params.setIsInternal(isInternal);
             params.setDefaultParallel(findDefaultParallel(project));
-            params.setInFipsJvm(Util.getBooleanProperty("tests.fips.enabled", false));
+            params.setInFipsJvm(FipsBuildParams.isInFipsMode());
             params.setIsSnapshotBuild(Util.getBooleanProperty("build.snapshot", true));
             if (isInternal) {
                 params.setBwcVersions(resolveBwcVersions(rootDir));
@@ -179,7 +181,7 @@ private void logGlobalBuildInfo() {
             LOGGER.quiet("  JAVA_HOME             : " + gradleJvm.getJavaHome());
         }
         LOGGER.quiet("  Random Testing Seed   : " + BuildParams.getTestSeed());
-        LOGGER.quiet("  In FIPS 140 mode      : " + BuildParams.isInFipsJvm());
+        LOGGER.quiet("  Crypto Standard       : " + FipsBuildParams.getFipsMode());
         LOGGER.quiet("=======================================");
     }
 

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java

@@ -46,6 +46,7 @@
 import org.opensearch.gradle.Version;
 import org.opensearch.gradle.VersionProperties;
 import org.opensearch.gradle.info.BuildParams;
+import org.opensearch.gradle.info.FipsBuildParams;
 import org.gradle.api.Action;
 import org.gradle.api.Named;
 import org.gradle.api.NamedDomainObjectContainer;
@@ -546,6 +547,10 @@ public synchronized void start() {
             logToProcessStdout("installed plugins");
         }
 
+        if (FipsBuildParams.isInFipsMode() && keystorePassword.isEmpty()) {
+            throw new TestClustersException("Can not start " + this + " in FIPS JVM, missing keystore password");
+        }
+
         logToProcessStdout("Creating opensearch keystore with password set to [" + keystorePassword + "]");
         if (keystorePassword.length() > 0) {
             runOpenSearchBinScriptWithInput(keystorePassword + "\n" + keystorePassword + "\n", "opensearch-keystore", "create", "-p");