[BUG] DEB systemd test failed as swap syscall not being blocked #18083

peterzhuamazon · 2025-04-25T20:47:04Z

Describe the bug

[BUG] DEB systemd test failed as swap syscall not being blocked

> Task :qa:systemd-test:integTest

SystemdIntegTests > testSystemCallFilter FAILED
    java.lang.AssertionError: Swap system call should be blocked
        at __randomizedtesting.SeedInfo.seed([D94D25E53E38C947:850EC0A576478592]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.opensearch.systemdinteg.SystemdIntegTests.testSystemCallFilter(SystemdIntegTests.java:148)

Tests with failures:
 - org.opensearch.systemdinteg.SystemdIntegTests.testSystemCallFilter

> Task :qa:systemd-test:integTest FAILED
60 actionable tasks: 1 executed, 59 up-to-date

Thanks.

Related component

Other

To Reproduce

See above

Expected behavior

See above

Additional Details

See above

The text was updated successfully, but these errors were encountered:

peterzhuamazon · 2025-04-25T20:47:25Z

Hi @kumargu @RajatGupta02 could you take a look on this?

This is particularly showing up on DEB only, not on rpm.

Thanks.

kumargu · 2025-04-27T07:12:27Z

It looks like swap might not be enabled on all DEBs which fails the assertions. could we check the existence of swap before executing swapon -a

The existence of swap devices can be checked by reading contents of /proc/swap

  private boolean isSwapEnabled() throws IOException {
        String content = Files.readString(Paths.get("/proc/swaps"));
       // check if there are more than one entry 
        return content.lines().skip(1).findFirst().isPresent();
    }

peterzhuamazon · 2025-04-28T18:13:54Z

DEB seems very easy to stop the cluster after systemctl start:

java.lang.UnsupportedOperationException: seccomp(BOGUS_OPERATION): Operation not permitted
        at org.opensearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:311) ~[opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:666) ~[opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:281) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:128) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:130) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:192) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:405) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.0.0-beta1.jar:3.0.0-beta1]

peterzhuamazon · 2025-04-28T18:16:29Z

opensearchcluster1.log

peterzhuamazon · 2025-04-28T19:41:54Z

Three steps:

Skip the swap check on systemd test for the time being. @RajatGupta02
i. Disable SystemCallFilter test temporarily #18115
ii. Remove testSystemCallFilter temporarily #18123
Add access for /dev/shm for the time being @RajatGupta02
i. Add shared memory access #18114
Make sure deb is using the /etc/default folder instead of /etc/sysconfig for env @peterzhuamazon
i. Make sure deb is correctly adding env file in OpenSearch opensearch-build#5479

Thanks.

peterzhuamazon · 2025-04-29T04:56:57Z

With #18114, both deb/rpm will fail with this error:

Apr 29 04:54:29 1e3522934a87 systemd[1]: Starting OpenSearch...
Apr 29 04:54:29 1e3522934a87 systemd[1]: opensearch.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 29 04:54:29 1e3522934a87 systemd[1]: opensearch.service: Failed with result 'exit-code'.
Apr 29 04:54:29 1e3522934a87 systemd[1]: Failed to start OpenSearch.

-- Unit opensearch.service has begun starting up.
Apr 29 04:54:29 1e3522934a87 systemd[1357]: opensearch.service: Failed to set up mount namespacing: No such file or directory
Apr 29 04:54:29 1e3522934a87 systemd[1357]: opensearch.service: Failed at step NAMESPACE spawning /usr/share/opensearch/bin/systemd-entrypoint: No such file or directory
-- Subject: Process /usr/share/opensearch/bin/systemd-entrypoint could not be executed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The process /usr/share/opensearch/bin/systemd-entrypoint could not be executed and failed.
--
-- The error number returned by this process is 2.
Apr 29 04:54:29 1e3522934a87 systemd[1]: opensearch.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 29 04:54:29 1e3522934a87 systemd[1]: opensearch.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit opensearch.service has entered the 'failed' state with result 'exit-code'.
Apr 29 04:54:29 1e3522934a87 systemd[1]: Failed to start OpenSearch.
-- Subject: Unit opensearch.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit opensearch.service has failed.
--
-- The result is failed.

The thing is, /usr/share/opensearch/bin/systemd-entrypoint is on disk and would only fail if ReadWritePaths=/dev/shm/performanceanalyzer present in systemd service file.

This location is not existing by the start of the cluster and will only be created when PA is invoke, which could be the cause as the folder is not even there.

Thanks.

peterzhuamazon · 2025-04-29T05:00:16Z

By changing from ReadWritePaths=/dev/shm/performanceanalyzer to ReadWritePaths=/dev/shm the cluster is able to start now.

peterzhuamazon · 2025-04-29T05:19:55Z

Will fix these, actually not taking effect due to systemd file does not interpret # after first char:


Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:154: Failed to parse boolean value, ignoring: true           # Prevent creating writable executable memory mappings
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "#", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "Allow", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "only", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "system", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "calls", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:156: Failed to parse keyring mode, ignoring: private                   # Service does not share key material with other services
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:157: Failed to parse boolean value, ignoring: true                  # Prevent changing ABI personality
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:158: Failed to parse boolean value, ignoring: true                 # Prevent creating SUID/SGID files
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:159: Failed to parse boolean value, ignoring: true                 # Prevent acquiring realtime scheduling
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:160: Failed to parse boolean value, ignoring: true                  # Prevent changes to system hostname
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:161: Failed to parse boolean value, ignoring: true                # Prevent reading/writing kernel logs
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:162: Failed to parse boolean value, ignoring: true                     # Prevent tampering with the system clock
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:154: Failed to parse boolean value, ignoring: true           # Prevent creating writable executable memory mappings
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "#", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "Allow", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "only", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "system", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:155: Failed to parse system call architecture "calls", ignoring: Invalid argument
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:156: Failed to parse keyring mode, ignoring: private                   # Service does not share key material with other services
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:157: Failed to parse boolean value, ignoring: true                  # Prevent changing ABI personality
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:158: Failed to parse boolean value, ignoring: true                 # Prevent creating SUID/SGID files
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:159: Failed to parse boolean value, ignoring: true                 # Prevent acquiring realtime scheduling
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:160: Failed to parse boolean value, ignoring: true                  # Prevent changes to system hostname
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:161: Failed to parse boolean value, ignoring: true                # Prevent reading/writing kernel logs
Apr 29 05:13:38 30b88f5a700d systemd[1]: /usr/lib/systemd/system/opensearch.service:162: Failed to parse boolean value, ignoring: true                     # Prevent tampering with the system clock

peterzhuamazon · 2025-04-29T06:07:50Z

Working copy:

# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0
#
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

# Description:
# Default opensearch.service file

[Unit]
Description=OpenSearch
Documentation=https://opensearch.org/
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
RuntimeDirectory=opensearch
PrivateTmp=true
EnvironmentFile=-/etc/default/opensearch
EnvironmentFile=-/etc/sysconfig/opensearch

WorkingDirectory=/usr/share/opensearch

User=opensearch
Group=opensearch

ExecStartPre=/bin/mkdir -p /dev/shm/performanceanalyzer
ExecStartPre=/bin/chown opensearch:opensearch /dev/shm/performanceanalyzer

ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# opensearch logging system is initialized. OpenSearch
# stores its logs in /var/log/opensearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit
SyslogIdentifier=opensearch

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

# Allow a slow startup before the systemd notifier module kicks in to extend the timeout
TimeoutStartSec=75

# Prevent modifications to the control group filesystem
ProtectControlGroups=true

# Prevent loading or reading kernel modules
ProtectKernelModules=true

# Prevent altering kernel tunables (sysctl parameters)
ProtectKernelTunables=true

# Set device access policy to 'closed', allowing access only to specific devices
DevicePolicy=closed

# Make /proc invisible to the service, enhancing isolation
ProtectProc=invisible

# Make /usr, /boot, and /etc read-only (less restrictive than 'strict')
ProtectSystem=full

# Prevent changes to control groups (redundant with earlier setting, can be removed)
ProtectControlGroups=yes

# Prevent changing the execution domain
LockPersonality=yes


# System call filtering
# System call filterings which restricts which system calls a process can make
# @ means allowed
# ~ means not allowed
SystemCallFilter=@system-service
SystemCallFilter=~@reboot
SystemCallFilter=~@swap

SystemCallErrorNumber=EPERM

# Capability restrictions
# Remove the ability to block system suspends
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND

# Remove the ability to establish leases on files
CapabilityBoundingSet=~CAP_LEASE

# Remove the ability to use system resource accounting
CapabilityBoundingSet=~CAP_SYS_PACCT

# Remove the ability to configure TTY devices
CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG

# Remov below capabilities:
# - CAP_SYS_ADMIN: Various system administration operations
# - CAP_SYS_PTRACE: Ability to trace processes
# - CAP_NET_ADMIN: Various network-related operations
CapabilityBoundingSet=~CAP_SYS_ADMIN ~CAP_SYS_PTRACE ~CAP_NET_ADMIN


# Address family restrictions
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# Filesystem Access

ReadWritePaths=/var/log/opensearch
ReadWritePaths=/var/lib/opensearch
ReadWritePaths=/dev/shm/
ReadWritePaths=-/etc/opensearch
ReadWritePaths=-/mnt/snapshots

## Allow read access to system files
ReadOnlyPaths=/etc/os-release /usr/lib/os-release /etc/system-release

## Allow read access to Linux IO stats
ReadOnlyPaths=/proc/self/mountinfo /proc/diskstats

## Allow read access to control group stats
ReadOnlyPaths=/proc/self/cgroup /sys/fs/cgroup/cpu /sys/fs/cgroup/cpu/-
ReadOnlyPaths=/sys/fs/cgroup/cpuacct /sys/fs/cgroup/cpuacct/- /sys/fs/cgroup/memory /sys/fs/cgroup/memory/-


RestrictNamespaces=true

NoNewPrivileges=true

# Memory and execution protection

# Allow only native system calls
SystemCallArchitectures=native
# Service does not share key material with other services
KeyringMode=private
# Prevent changing ABI personality
LockPersonality=true
# Prevent creating SUID/SGID files
RestrictSUIDSGID=true
# Prevent acquiring realtime scheduling
RestrictRealtime=true
# Prevent changes to system hostname
ProtectHostname=true
# Prevent reading/writing kernel logs
ProtectKernelLogs=true
# Prevent tampering with the system clock
ProtectClock=true

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

peterzhuamazon · 2025-04-29T06:16:21Z

Fix systemd configs #18134

peterzhuamazon · 2025-04-29T19:42:02Z

mergedeb.txt

peterzhuamazon · 2025-04-30T02:57:32Z

[3.0] Add syscalls for merge operation error #18148

peterzhuamazon · 2025-04-30T07:08:38Z

Resolved by #18148 therefore close the issue:
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/9648/pipeline

Thanks.

peterzhuamazon added bug Something isn't working untriaged labels Apr 25, 2025

github-actions bot added the Other label Apr 25, 2025

peterzhuamazon mentioned this issue Apr 25, 2025

[RELEASE] Release version 3.0.0 opensearch-project/opensearch-build#3747

Closed

66 tasks

RajatGupta02 mentioned this issue Apr 28, 2025

Add shared memory access #18114

Merged

3 tasks

peterzhuamazon assigned peterzhuamazon and kumargu Apr 28, 2025

This was referenced Apr 28, 2025

Disable SystemCallFilter test temporarily #18115

Closed

Remove testSystemCallFilter temporarily #18123

Merged

peterzhuamazon mentioned this issue Apr 29, 2025

[Backport main] Add shared memory access #18119

Closed

RajatGupta02 mentioned this issue Apr 29, 2025

Fix systemd configs #18134

Merged

3 tasks

RajatGupta02 mentioned this issue Apr 29, 2025

Fix systemd configs in main #18135

Merged

3 tasks

peterzhuamazon closed this as completed in #18135 Apr 29, 2025

peterzhuamazon reopened this Apr 29, 2025

peterzhuamazon mentioned this issue Apr 29, 2025

[3.0] Add syscalls for merge operation error #18148

Merged

3 tasks

peterzhuamazon closed this as completed Apr 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] DEB systemd test failed as swap syscall not being blocked #18083

[BUG] DEB systemd test failed as swap syscall not being blocked #18083

peterzhuamazon commented Apr 25, 2025

peterzhuamazon commented Apr 25, 2025

kumargu commented Apr 27, 2025 •

edited

Loading

peterzhuamazon commented Apr 28, 2025

peterzhuamazon commented Apr 28, 2025

peterzhuamazon commented Apr 28, 2025 •

edited

Loading

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 30, 2025

peterzhuamazon commented Apr 30, 2025

[BUG] DEB systemd test failed as swap syscall not being blocked #18083

[BUG] DEB systemd test failed as swap syscall not being blocked #18083

Comments

peterzhuamazon commented Apr 25, 2025

Describe the bug

Related component

To Reproduce

Expected behavior

Additional Details

peterzhuamazon commented Apr 25, 2025

kumargu commented Apr 27, 2025 • edited Loading

peterzhuamazon commented Apr 28, 2025

peterzhuamazon commented Apr 28, 2025

peterzhuamazon commented Apr 28, 2025 • edited Loading

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 29, 2025

peterzhuamazon commented Apr 30, 2025

peterzhuamazon commented Apr 30, 2025

kumargu commented Apr 27, 2025 •

edited

Loading

peterzhuamazon commented Apr 28, 2025 •

edited

Loading