Add SPLUNK_MEMORY_TOTAL_MIB (open-telemetry#48)

Today, SPLUNK_BALLAST_SIZE_MIB needs to be computed and manually
configured by a user. Most users do not know or care about the ballast
and under normal circumstances they should not need to. Users will
understand the total amount of memory allocated to the Collector. As
such, introduce a SPLUNK_MEMORY_TOTAL_MIB env var that can be used
instead of or in addition to SPLUNK_BALLAST_SIZE_MIB. This has the added
bonus that now memory spike and limit env vars are no longer required
for non-linux systems. This change is backwards compatible.

A few other minor enhancements:

- Memory spike/limit env vars are now applied even if SPLUNK_CONFIG is
  passed
- Add logging for custom behavior so configuration is clear
- Add accessible endpoints to README

Author: flands

README.md

@@ -41,19 +41,35 @@ which require the following environment variables:
 
 - `SPLUNK_REALM` (no default): Which realm to send the data to (for example: `us0`)
 - `SPLUNK_ACCESS_TOKEN` (no default): Access token to authenticate requests
-- `SPLUNK_BALLAST_SIZE_MIB` (no default): How much memory to allocate to the ballast. This should be set to 1/3 to 1/2 of configured memory.
+- `SPLUNK_MEMORY_TOTAL_MIB` (no default): Total memory allocated to the Collector.
 
-In addition, the following environment variables are optional:
+The following environment variables are optional:
 
 - `SPLUNK_CONFIG` (default = `/etc/otel/collector/splunk-config_linux.yaml`): Which configuration to load.
-- `SPLUNK_MEMORY_LIMIT_PERCENTAGE` (default = `90`): Maximum total memory to be allocated by the process heap.
-- `SPLUNK_MEMORY_SPIKE_PERCENTAGE` (default = `20`): Maximum spike between the measurements of memory usage.
-
-When running on a non-linux system, the following environment variables are required:
-
-- `SPLUNK_CONFIG` (default = `/etc/otel/collector/splunk-config_non_linux.yaml`): Configuration to load.
-- `SPLUNK_MEMORY_LIMIT_MIB` (no default): Maximum total memory to be allocated by the process heap.
-- `SPLUNK_MEMORY_SPIKE_MIB` (no default): Maximum spike between the measurements of memory usage.
+- `SPLUNK_BALLAST_SIZE_MIB` (no default): How much memory to allocate to the ballast.
+- For Linux systems:
+  - `SPLUNK_MEMORY_LIMIT_PERCENTAGE` (default = `90`): Maximum total memory to be allocated by the process heap.
+  - `SPLUNK_MEMORY_SPIKE_PERCENTAGE` (default = `20`): Maximum spike between the measurements of memory usage.
+- For non-Linux systems:
+  - `SPLUNK_MEMORY_LIMIT_MIB` (no default): Maximum total memory to be allocated by the process heap.
+  - `SPLUNK_MEMORY_SPIKE_MIB` (no default): Maximum spike between the measurements of memory usage.
+
+> `SPLUNK_MEMORY_TOTAL_MIB` automatically configures the ballast, memory limit,
+> and memory spike. If the optional environment variables are defined, they
+> will override the value calculated from `SPLUNK_MEMORY_TOTAL_MIB`.
+
+With the Collector configured, the following endpoints are accessible:
+
+- `http(s)://<collectorFQDN>:13133/` Health endpoint useful for load balancer monitoring
+- `http(s)://<collectorFQDN>:[14250|14268]` Jaeger [gRPC|Thrift HTTP] receiver
+- `http(s)://<collectorFQDN>:55678` OpenCensus gRPC and HTTP receiver
+- `http(s)://localhost:55679/debug/[tracez|pipelinez]` zPages monitoring
+- `http(s)://<collectorFQDN>:55680` OpenTelemetry gRPC receiver
+- `http(s)://<collectorFQDN>:6060` HTTP Forwarder used to receive Smart Agent `apiUrl` data
+- `http(s)://<collectorFQDN>:7276` SignalFx Infrastructure Monitoring gRPC receiver
+- `http(s)://localhost:8888/metrics` Prometheus metrics for the Collector
+- `http(s)://<collectorFQDN>:9411/api/[v1|v2]/spans` Zipkin JSON (can be set to proto)receiver
+- `http(s)://<collectorFQDN>:9943/v2/trace` SignalFx APM receiver
 
 The following sections describe how to deploy the Collector in supported environments.
 
@@ -62,7 +78,7 @@ The following sections describe how to deploy the Collector in supported environ
 Deploy from a Docker container. Replace `0.1.0` with the latest stable version number:
 
 ```bash
-$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_BALLAST_SIZE_MIB=683 \
+$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_MEMORY_TOTAL_MIB=1024 \
     -e SPLUNK_REALM=us0 -p 13133:13133 -p 14250:14250 -p 14268:14268 -p 55678-55680:55678-55680 \
     -p 6060:6060 -p 7276:7276 -p 8888:8888 -p 9411:9411 -p 9943:9943 \
     --name otelcol quay.io/signalfx/splunk-otel-collector:0.1.0
@@ -80,7 +96,7 @@ file on GitHub.
 
 ```bash
 $ make otelcol
-$ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_BALLAST_SIZE_MIB=683 \
+$ SPLUNK_REALM=us0 SPLUNK_ACCESS_TOKEN=12345 SPLUNK_MEMORY_TOTAL_MIB=1024 \
     ./bin/otelcol
 ```
 
@@ -197,7 +213,7 @@ least a CPU core per Collector. Multiple Collectors can deployed behind a
 simple round-robin load balancer. Each Collector runs independently, so scale
 increases linearly with the number of Collectors you deploy.
 
-The Collector does not persist data to disk so no disk space is required.
+> The Collector does not persist data to disk so no disk space is required.
 
 ## Advanced Configuration
 
@@ -209,7 +225,7 @@ specified. Command line arguments take priority over environment variables.
 For example in Docker:
 
 ```bash
-$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_BALLAST_SIZE_MIB=683 \
+$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_MEMORY_TOTAL_MIB=1024 \
     -e SPLUNK_REALM=us0 -p 13133:13133 -p 14250:14250 -p 14268:14268 -p 55678-55680:55678-55680 \
     -p 6060:6060 -p 7276:7276 -p 8888:8888 -p 9411:9411 -p 9943:9943 \
     --name otelcol quay.io/signalfx/splunk-otel-collector:0.1.0 \
@@ -230,7 +246,7 @@ the `--config` command line argument to provide a custom configuration.
 For example in Docker:
 
 ```bash
-$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_BALLAST_SIZE_MIB=683 \
+$ docker run --rm -e SPLUNK_ACCESS_TOKEN=12345 -e SPLUNK_MEMORY_TOTAL_MIB=1024 \
     -e SPLUNK_REALM=us0 -e SPLUNK_CONFIG=/etc/collector.yaml -p 13133:13133 -p 14250:14250 \
     -p 14268:14268 -p 55678-55680:55678-55680 -p 6060:6060 -p 7276:7276 -p 8888:8888 \
     -p 9411:9411 -p 9943:9943 -v collector.yaml:/etc/collector.yaml:ro \

cmd/otelcol/main.go

@@ -39,6 +39,7 @@ const (
 	memLimitMiBEnvVarName = "SPLUNK_MEMORY_LIMIT_MIB"
 	memSpikeEnvVarName    = "SPLUNK_MEMORY_SPIKE_PERCENTAGE"
 	memSpikeMiBEnvVarName = "SPLUNK_MEMORY_SPIKE_MIB"
+	memTotalEnvVarName    = "SPLUNK_MEMORY_TOTAL_MIB"
 	realmEnvVarName       = "SPLUNK_REALM"
 	tokenEnvVarName       = "SPLUNK_ACCESS_TOKEN"
 
@@ -50,22 +51,50 @@ const (
 	defaultLocalSAPMNonLinuxConfig  = "cmd/otelcol/config/collector/splunk_config_non_linux.yaml"
 	defaultLocalOTLPLinuxConfig     = "cmd/otelcol/config/collector/otlp_config_linux.yaml"
 	defaultLocalOTLPNonLinuxConfig  = "cmd/otelcol/config/collector/otlp_config_non_linux.yaml"
+	defaultMemoryBallastPercentage  = 50
 	defaultMemoryLimitPercentage    = 90
+	defaultMemoryLimitMaxMiB        = 2048
 	defaultMemorySpikePercentage    = 25
+	defaultMemorySpikeMaxMiB        = 2048
 )
 
 func main() {
+	// TODO: Use same format as the collector
 	log.SetFlags(log.Ldate | log.Ltime | log.Lshortfile)
 
+	// Check if the total memory is specified via the env var.
+	memTotalEnvVarVal := os.Getenv(memTotalEnvVarName)
+	memTotalSizeMiB := 0
+	if memTotalEnvVarVal != "" {
+		// Check if it is a numeric value.
+		val, err := strconv.Atoi(memTotalEnvVarVal)
+		if err != nil {
+			log.Fatalf("Expected a number in %s env variable but got %s", memTotalEnvVarName, memTotalEnvVarVal)
+		}
+		if 10 > val {
+			log.Fatalf("Expected a number greater than 10 for %s env variable but got %s", memTotalEnvVarName, memTotalEnvVarVal)
+		}
+		memTotalSizeMiB = val
+	}
+
 	// Check runtime parameters
 	// Runtime parameters take priority over environment variables
 	// Runtime parameters are not validated
 	args := os.Args[1:]
 	if !contains(args, "--mem-ballast-size-mib") {
-		useBallastSizeFromEnvVar()
+		useMemorySizeFromEnvVar(memTotalSizeMiB)
+	} else {
+		log.Printf("Ballast CLI argument found, ignoring %s if set", ballastEnvVarName)
 	}
 	if !contains(args, "--config") {
 		useConfigFromEnvVar()
+	} else {
+		log.Printf("Config CLI argument found, please ensure memory_limiter settings are correct")
+	}
+	if runtime.GOOS == "linux" {
+		useMemorySettingsPercentageFromEnvVar()
+	} else {
+		useMemorySettingsMiBFromEnvVar(memTotalSizeMiB)
 	}
 
 	factories, err := components.Get()
@@ -98,18 +127,27 @@ func contains(arr []string, str string) bool {
 	return false
 }
 
-func useBallastSizeFromEnvVar() {
+func useMemorySizeFromEnvVar(memTotalSizeMiB int) {
 	// Check if the ballast is specified via the env var.
 	ballastSize := os.Getenv(ballastEnvVarName)
 	if ballastSize != "" {
 		// Check if it is a numeric value.
-		_, err := strconv.Atoi(ballastSize)
+		val, err := strconv.Atoi(ballastSize)
 		if err != nil {
 			log.Fatalf("Expected a number in %s env variable but got %s", ballastEnvVarName, ballastSize)
 		}
+		if 0 > val {
+			log.Fatalf("Expected a number greater than 0 for %s env variable but got %s", ballastEnvVarName, ballastSize)
+		}
 
 		// Inject the command line flag that controls the ballast size.
 		os.Args = append(os.Args, "--mem-ballast-size-mib="+ballastSize)
+	} else if memTotalSizeMiB > 0 {
+		halfMem := strconv.Itoa(memTotalSizeMiB * defaultMemoryBallastPercentage / 100)
+		log.Printf("Set ballast to %s MiB", halfMem)
+		// Inject the command line flag that controls the ballast size.
+		os.Args = append(os.Args, "--mem-ballast-size-mib="+halfMem)
+		os.Setenv(ballastEnvVarName, halfMem)
 	}
 }
 
@@ -130,7 +168,6 @@ func useConfigFromEnvVar() {
 			if config == "" {
 				log.Fatalf("Unable to find the default configuration file, ensure %s environment variable is set properly", configEnvVarName)
 			}
-			useMemorySettingsPercentageFromEnvVar()
 		} else {
 			_, err := os.Stat(defaultDockerSAPMNonLinuxConfig)
 			if err == nil {
@@ -143,7 +180,6 @@ func useConfigFromEnvVar() {
 			if config == "" {
 				log.Fatalf("Unable to find the default configuration file, ensure %s environment variable is set properly", configEnvVarName)
 			}
-			useMemorySettingsMiBFromEnvVar()
 		}
 	} else {
 		// Check if file exists.
@@ -165,25 +201,31 @@ func useConfigFromEnvVar() {
 		defaultLocalOTLPNonLinuxConfig:
 		// The following environment variables are required.
 		// If any are missing stop here.
-		requiredEnvVars := []string{ballastEnvVarName, realmEnvVarName, tokenEnvVarName}
+		requiredEnvVars := []string{realmEnvVarName, tokenEnvVarName}
 		for _, v := range requiredEnvVars {
 			if len(os.Getenv(v)) == 0 {
-				log.Printf("Usage: %s=12345 %s=us0 %s=684 %s", tokenEnvVarName, realmEnvVarName, ballastEnvVarName, os.Args[0])
+				log.Printf("Usage: %s=12345 %s=us0 %s=1024 %s", tokenEnvVarName, realmEnvVarName, memTotalEnvVarName, os.Args[0])
 				log.Fatalf("ERROR: Missing environment variable %s", v)
 			}
 		}
+		// Needed for backwards compatibility
+		if len(os.Getenv(memTotalEnvVarName)) == 0 && len(os.Getenv(ballastEnvVarName)) == 0 {
+			log.Printf("Usage: %s=12345 %s=us0 %s=1024 %s", tokenEnvVarName, realmEnvVarName, memTotalEnvVarName, os.Args[0])
+			log.Fatalf("ERROR: Missing environment variable %s", memTotalEnvVarName)
+		}
 	}
 
 	// Inject the command line flag that controls the configuration.
 	os.Args = append(os.Args, "--config="+config)
 }
 
-func checkMemorySettingsMiBFromEnvVar(envVar string) int {
+func checkMemorySettingsMiBFromEnvVar(envVar string, memTotalSizeMiB int) int {
 	// Check if the memory limit is specified via the env var
 	// Ensure memory limit is valid
-	var envVarResult int
+	var envVarResult int = 0
 	envVarVal := os.Getenv(envVar)
-	if envVarVal != "" {
+	switch {
+	case envVarVal != "":
 		// Check if it is a numeric value.
 		val, err := strconv.Atoi(envVarVal)
 		if err != nil {
@@ -193,16 +235,54 @@ func checkMemorySettingsMiBFromEnvVar(envVar string) int {
 			log.Fatalf("Expected a number greater than 0 for %s env variable but got %s", envVar, envVarVal)
 		}
 		envVarResult = val
-	} else {
+	case memTotalSizeMiB > 0:
+		break
+	default:
 		log.Printf("Usage: %s=12345 %s=us0 %s=684 %s=1024 %s=256 %s", tokenEnvVarName, realmEnvVarName, ballastEnvVarName, memLimitMiBEnvVarName, memSpikeMiBEnvVarName, os.Args[0])
 		log.Fatalf("ERROR: Missing environment variable %s", envVar)
 	}
 	return envVarResult
 }
 
-func useMemorySettingsMiBFromEnvVar() {
-	memLimit := checkMemorySettingsMiBFromEnvVar(memLimitMiBEnvVarName)
-	memSpike := checkMemorySettingsMiBFromEnvVar(memSpikeMiBEnvVarName)
+func useMemorySettingsMiBFromEnvVar(memTotalSizeMiB int) {
+	// Check if memory limit is specified via environment variable
+	memLimit := checkMemorySettingsMiBFromEnvVar(memLimitMiBEnvVarName, memTotalSizeMiB)
+	// Use if set, otherwise memory total size must be specified
+	if memLimit == 0 {
+		if memTotalSizeMiB == 0 {
+			panic("PANIC: Both memory limit MiB and memory total size are set to zero. This should never happen.")
+		}
+		// If not set, compute based on memory total size specified
+		// and default memory limit percentage const
+		memLimitMiB := memTotalSizeMiB * defaultMemoryLimitPercentage / 100
+		// The memory limit should be set to defaultMemoryLimitPercentage of total memory
+		// while reserving a maximum of defaultMemoryLimitMaxMiB of memory.
+		if (memTotalSizeMiB - memLimitMiB) < defaultMemoryLimitMaxMiB {
+			memLimit = memLimitMiB
+		} else {
+			memLimit = (memTotalSizeMiB - defaultMemoryLimitMaxMiB)
+		}
+		log.Printf("Set memory limit to %d MiB", memLimit)
+	}
+	// Check if memory spike is specified via environment variable
+	memSpike := checkMemorySettingsMiBFromEnvVar(memSpikeMiBEnvVarName, memTotalSizeMiB)
+	// Use if set, otherwise memory total size must be specified
+	if memSpike == 0 {
+		if memTotalSizeMiB == 0 {
+			panic("PANIC: Both memory limit MiB and memory total size are set to zero. This should never happen.")
+		}
+		// If not set, compute based on memory total size specified
+		// and default memory spike percentage const
+		memSpikeMiB := memTotalSizeMiB * defaultMemorySpikePercentage / 100
+		// The memory spike should be set to defaultMemorySpikePercentage of total memory
+		// while specifying a maximum of defaultMemorySpikeMaxMiB of memory.
+		if memSpikeMiB < defaultMemorySpikeMaxMiB {
+			memSpike = memSpikeMiB
+		} else {
+			memSpike = defaultMemorySpikeMaxMiB
+		}
+		log.Printf("Set memory spike limit to %d MiB", memSpike)
+	}
 	setMemorySettingsToEnvVar(memLimit, memLimitMiBEnvVarName, memSpike, memSpikeMiBEnvVarName)
 }