Using PrivateNetwork seems to break on systemd 253.5

[aragilar]

On the latest systemd in Debian, the netns@.service unit stopped working until I removed https://github.com/Jamesits/systemd-named-netns/blob/master/services/netns%40.service#L15. I suspect there's some changes to what security settings are used which are implied by PrivateNetwork.

The most obvious change (and the way I spotted this) was the associated file in /run/netns stopped being read-only, and instead had no permissions.

[Jamesits]

Yes, and I almost forgot that I had the answer somehow...

See systemd/systemd#2741 (comment)

[stevo1977]

First of all, this namespace setup has been working flawlessly for me for years, so thank you!

I recently upgraded to systemd 254 and netns-nat@ovpn failed on a dependency job. I tried @aragilar 's fix above (changing the PrivateNetwork to no), which got rid of the mount/dependency problem, but the service is still failing to start. I've attached the contents of journalctl -xeu netns-nat@ovpn.service. Any idea what is still causing an issue?
ovpn_fail.txt

[Jamesits]

@stevo1977

> ip route add default via 100.64.0.1 dev veth1 onlink
Error: Nexthop has invalid gateway.

This line failed; I'm not sure why, maybe 100.64.0.1 is used elsewhere on your system?

[stevo1977]

@Jamesits , thank you for having a look.

It looks like that assignment comes from lines 42-45 in netnsinit, as a part of autoconfigure_tunnel_up_inside():

	# setup default route
	if [ ! -z "${GATEWAY}" ]; then
		ip route add default via ${GATEWAY%%/*} dev ${DEVNAME_INSIDE} onlink
	fi

but I'm not sure where the variable GATEWAY comes from, or what the function of %%/* is in bash.

[Jamesits]

@stevo1977

%%/* means remove everything after (incl.) the character /. GATEWAY comes from the configuration file (/etc/default/...).

[stevo1977]

@Jamesits , ah, very cool.

I tried a few different IP addresses in /etc/default/netns-nat but they all returned the same invalid gateway error. Is there some other conflict there with the new systemd? So weird.

[CarloWood]

I ran into this too... I'll write a fix when I come back later today.

[CarloWood]

I've been looking into this for four hours now.
It seems that the way it used to work is as follows:

1. PrivateNetwork=yes puts the current process in a nameless(?) network namespace.
   This can be verified by adding ExecStart=ip address which then prints (to journal) only an lo loopback device.
2. Then some hack was performed using an umount and a bind mount to poke around in the implementation of the network namespace in a way that the result looks like it had some name that we want it to have.
   This should never have worked and certainly isn't the intended usage, so it was to be expected that it would break one day. With how network namespaces currently work, the directory that it tries to umount isn't mounted at all as far as I can tell.

The correct way to bring the current process into a named network namespace would be:
ip netns attach %I $pid, where $pid is the PID of a process in the current (nameless) namespace. However, that can't be done because all ExecStarts are already inside a network namespace (as a result of the PrivateNetwork=yes and it is not possible to create a new namespace while already inside one.

Moreover, the network namespace created by PrivateNetwork=yes only contains the current process (systemctl - and any long running process that would be executed from an ExecStart), but this unit is oneshot: it is supposed to exit immediately. The network namespace is probably created for just this process and immediately deleted, so that the namespace will be deleted automatically as soon as the process ceases to exist, which is pretty quick.

My conclusion is that (indeed) we don't need or want the PrivateNetwork=yes. It serves no purpose and only makes it impossible to create a named namespace that we want to keep alive until this .service is stopped.

Hence, the solution should be:

[Service]
Type=oneshot
# We change the state of the system: we create a network namespace called %I.
RemainAfterExit=yes
# Set up netns
ExecStart=/usr/bin/env ip netns add %I

At this point we have a new network namespace, but the current process is NOT running inside of it. I do not understand why that would be needed: this process is going to exit anyway immediately no?

[Jamesits]

@CarloWood Try use ExecStartPre=+/path/to/executable to get a hook script to run as root with full privileges. (See systemd.service(5)). There is still a challenge to get the PID though...

[CarloWood]

@Jamesits ExecStartPre= runs in the same network namespace as ExecStart; they are just run first. If I do:

PrivateNetwork=yes
ExecStartPre=ip address

it still prints only a lo loopback device.

[Jamesits]

@CarloWood Note the + character.

[CarloWood]

@Jamesits That is not working either - still inside the netns.

Nevertheless, I have something working.
If I add a file /etc/default/netns-raw with the contents:

echo "Running /etc/default/netns-raw; PIDs in netns $NSNAME:"
ip netns pids $NSNAME

echo "Network namespace of current process: $(ip netns identify $$)"

And then run sudo systemctl start netns@ns210.service,
I get the following output:

Apr 21 17:03:43 daniel env[933405]: + source /etc/default/netns
Apr 21 17:03:43 daniel env[933405]: + source /etc/default/netns-raw
Apr 21 17:03:43 daniel env[933405]: ++ echo 'Running /etc/default/netns-raw; PIDs in netns ns210:'
Apr 21 17:03:43 daniel env[933405]: Running /etc/default/netns-raw; PIDs in netns ns210:
Apr 21 17:03:43 daniel env[933405]: ++ ip netns pids ns210
Apr 21 17:03:43 daniel env[933406]: 933405
Apr 21 17:03:43 daniel env[933406]: 933406
Apr 21 17:03:43 daniel env[933407]: +++ ip netns identify 933405
Apr 21 17:03:43 daniel env[933405]: ++ echo 'Network namespace of current process: ns210'
Apr 21 17:03:43 daniel env[933405]: Network namespace of current process: ns210

Aka - the process netnsinit is running inside the (created) netns ns210.

Also, subsequently running sudo systemctl stop netns@ns210.service gives

Apr 21 17:07:13 daniel env[933595]: + source /etc/default/netns
Apr 21 17:07:13 daniel env[933595]: + source /etc/default/netns-raw
Apr 21 17:07:13 daniel env[933595]: ++ echo 'Running /etc/default/netns-raw; PIDs in netns ns210:'
Apr 21 17:07:13 daniel env[933595]: Running /etc/default/netns-raw; PIDs in netns ns210:
Apr 21 17:07:13 daniel env[933595]: ++ ip netns pids ns210
Apr 21 17:07:13 daniel env[933596]: 933595
Apr 21 17:07:13 daniel env[933596]: 933596
Apr 21 17:07:13 daniel env[933597]: +++ ip netns identify 933595
Apr 21 17:07:13 daniel env[933595]: ++ echo 'Network namespace of current process: ns210'
Apr 21 17:07:13 daniel env[933595]: Network namespace of current process: ns210

Afterwards the netns is deleted because nobody is using it anymore.

I believe this is where we were after.