Shared network for vfkit driver using vmnet-helper #20501

nirs · 2025-03-07T23:41:38Z

Add new network option for vfkit "vment-shared", connecting vfkit to the
vmnet shared network. Clusters using this network can access other
clusters in the same network, similar to socket_vment with QEMU driver.

If network is not specified, we default to the "nat" network, keeping
the previous behavior. If network is "vment-shared", the vfkit driver
manages 2 processes: vfkit and vmnet-helper.

Like vfkit, vmnet-helper is started in the background, in a new process
group, so it not terminated if the minikube process group is terminate.

Since vment-helper requires root to start the vmnet interface, we start
it with sudo, creating 2 child processes. vment-helper drops privileges
immediately after starting the vment interface, and run as the user and
group running minikube.

Stopping the cluster will stop sudo, which will stop the vmnet-helper
process. Deleting the cluster kill both sudo and vment-helper by killing
the process group.

This change is not complete, but it is good enough to play with the new
shared network.

Example usage:

Install vmnet-helper:
https://github.com/nirs/vmnet-helper?tab=readme-ov-file#installation
Setup vment-helper sudoers rule:
https://github.com/nirs/vmnet-helper?tab=readme-ov-file#granting-permission-to-run-vmnet-helper
Start 2 clusters with vmnet-shared network:

% minikube start -p c1 --driver vfkit --network vment-shared
...

% minikube start -p c2 --driver vfkit --network vmnet-shared
...

% minikube ip -p c1
192.168.105.18

% minikube ip -p c2
192.168.105.19

Both cluster can access the other cluster:

% minikube -p c1 ssh -- ping -c 3 192.168.105.19
PING 192.168.105.19 (192.168.105.19): 56 data bytes
64 bytes from 192.168.105.19: seq=0 ttl=64 time=0.621 ms
64 bytes from 192.168.105.19: seq=1 ttl=64 time=0.989 ms
64 bytes from 192.168.105.19: seq=2 ttl=64 time=0.490 ms

--- 192.168.105.19 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.490/0.700/0.989 ms

% minikube -p c2 ssh -- ping -c 3 192.168.105.18
PING 192.168.105.18 (192.168.105.18): 56 data bytes
64 bytes from 192.168.105.18: seq=0 ttl=64 time=0.289 ms
64 bytes from 192.168.105.18: seq=1 ttl=64 time=0.798 ms
64 bytes from 192.168.105.18: seq=2 ttl=64 time=0.993 ms

--- 192.168.105.18 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.289/0.693/0.993 ms

To complete this work we need:

Install vmnet-helper on the CI macOS hosts
Add test using --network vmnet-shared

Fixes #20557
Fixes #20558

k8s-ci-robot · 2025-03-07T23:41:46Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nirs
Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-03-07T23:41:47Z

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2025-03-07T23:42:02Z

Can one of the admins verify this patch?

nirs · 2025-03-07T23:49:23Z

@afbjorklund can you review this?

pkg/drivers/vmnet/vmnet.go

nirs · 2025-03-09T22:51:56Z

Example machine config when vmnet-shared is used:

{
    "ConfigVersion": 3,
    "Driver": {
        "IPAddress": "192.168.105.21",
        "MachineName": "minikube",
        "SSHUser": "docker",
        "SSHPort": 22,
        "SSHKeyPath": "",
        "StorePath": "/Users/nir/.minikube",
        "SwarmMaster": false,
        "SwarmHost": "",
        "SwarmDiscovery": "",
        "Boot2DockerURL": "file:///Users/nir/.minikube/cache/iso/arm64/minikube-v1.35.0-arm64.iso",
        "DiskSize": 20000,
        "CPU": 2,
        "Memory": 6000,
        "Cmdline": "",
        "ExtraDisks": 0,
        "Network": "vmnet-shared",
        "MACAddress": "de:a2:9c:71:3c:f7",
        "VmnetHelper": {
            "MachineDir": "/Users/nir/.minikube/machines/minikube",
            "InterfaceID": "a0c43efb-2dcc-4abb-a310-568782c5dc7a"
        }
    },
    "DriverName": "vfkit",
    "HostOptions": {
        "Driver": "",
        "Memory": 0,
        "Disk": 0,
        "EngineOptions": {
            "ArbitraryFlags": null,
            "Dns": null,
            "GraphDir": "",
            "Env": null,
            "Ipv6": false,
            "InsecureRegistry": [
                "10.96.0.0/12"
            ],
            "Labels": null,
            "LogLevel": "",
            "StorageDriver": "",
            "SelinuxEnabled": false,
            "TlsVerify": false,
            "RegistryMirror": [],
            "InstallURL": "https://get.docker.com"
        },
        "SwarmOptions": {
            "IsSwarm": false,
            "Address": "",
            "Discovery": "",
            "Agent": false,
            "Master": false,
            "Host": "",
            "Image": "",
            "Strategy": "",
            "Heartbeat": 0,
            "Overcommit": 0,
            "ArbitraryFlags": null,
            "ArbitraryJoinFlags": null,
            "Env": null,
            "IsExperimental": false
        },
        "AuthOptions": {
            "CertDir": "/Users/nir/.minikube",
            "CaCertPath": "/Users/nir/.minikube/certs/ca.pem",
            "CaPrivateKeyPath": "/Users/nir/.minikube/certs/ca-key.pem",
            "CaCertRemotePath": "",
            "ServerCertPath": "/Users/nir/.minikube/machines/server.pem",
            "ServerKeyPath": "/Users/nir/.minikube/machines/server-key.pem",
            "ClientKeyPath": "/Users/nir/.minikube/certs/key.pem",
            "ServerCertRemotePath": "",
            "ServerKeyRemotePath": "",
            "ClientCertPath": "/Users/nir/.minikube/certs/cert.pem",
            "ServerCertSANs": null,
            "StorePath": "/Users/nir/.minikube"
        }
    },
    "Name": "minikube"
}

nirs · 2025-03-09T23:18:04Z

Example multi-node cluster

% minikube start --network vmnet-shared --nodes 2 --cni auto
😄  minikube v1.35.0 on Darwin 15.3.1 (arm64)
✨  Using the vfkit (experimental) driver based on user configuration
❗  --network flag is only valid with the docker/podman, KVM and Qemu drivers, it will be ignored
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=4050MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.32.2 on containerd 1.7.23 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner

👍  Starting "minikube-m02" worker node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=4050MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.105.22
📦  Preparing Kubernetes v1.32.2 on containerd 1.7.23 ...
    ▪ env NO_PROXY=192.168.105.22
    > kubelet.sha256:  64 B / 64 B [-------------------------] 100.00% ? p/s 0s
    > kubectl.sha256:  64 B / 64 B [-------------------------] 100.00% ? p/s 0s
    > kubeadm.sha256:  64 B / 64 B [-------------------------] 100.00% ? p/s 0s
    > kubectl:  53.25 MiB / 53.25 MiB [------------] 100.00% 37.13 MiB p/s 1.6s
    > kubelet:  71.75 MiB / 71.75 MiB [------------] 100.00% 12.43 MiB p/s 6.0s
    > kubeadm:  66.81 MiB / 66.81 MiB [------------] 100.00% 10.11 MiB p/s 6.8s
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

% kubectl get node                                                           
NAME           STATUS   ROLES           AGE     VERSION
minikube       Ready    control-plane   5m1s    v1.32.2
minikube-m02   Ready    <none>          4m40s   v1.32.2

% kubectl get node -o jsonpath='{.items[*].status.addresses[0].address}{"\n"}'
192.168.105.22 192.168.105.23

Each node get its own vment-helper process:

% ps au | grep vmnet-helper | grep -v grep 
nir  60260   0.0  0.0 410743984   3792 s020  S     1:07AM   0:00.71 /opt/vmnet-helper/bin/vmnet-helper --fd 21 --interface-id c9ae713b-5937-42de-9018-bbb95a425506
root 60259   0.0  0.0 410737168   6448 s020  S     1:07AM   0:00.01 sudo --non-interactive --close-from 22 /opt/vmnet-helper/bin/vmnet-helper --fd 21 --interface-id c9ae713b-5937-42de-9018-bbb95a425506
nir  60254   0.0  0.0 410735792   3728 s020  S     1:07AM   0:00.73 /opt/vmnet-helper/bin/vmnet-helper --fd 13 --interface-id cf62fb28-2533-4c25-8d35-1eef1736f707
root 60253   0.0  0.0 410754576   6992 s020  S     1:07AM   0:00.01 sudo --non-interactive --close-from 14 /opt/vmnet-helper/bin/vmnet-helper --fd 13 --interface-id cf62fb28-2533-4c25-8d35-1eef1736f707

afbjorklund · 2025-03-10T06:50:38Z

@afbjorklund can you review this?

I could take a look at it later perhaps, but one of the minikube maintainers will still need to "take over" the vfkit driver.

Probably should have an issue, as well.

cfergeau

I also noticed several vment typos in commit logs

pkg/drivers/vfkit/vfkit.go

pkg/drivers/pidfile.go

System state changes should be more visible to make debugging easier.

nirs · 2025-04-02T20:00:10Z

@medyagh I rebased and tested again, it should ready now.

medyagh · 2025-04-08T17:09:19Z

/ok-to-test

medyagh

thank you @nirs for the PR I left some comments, PTAL,
additionally, can you plz paste the output of following scenario

1- start minikube with vnmnet-helper
2- delete minikube home folder rm -rf ~/.minikube
3- minikube profile list
4- minikube delete --all
ideally there should be no left over process but if there is, we should create a follow up issue to clean up the abandoned proceses or left overs

cmd/minikube/cmd/start_flags.go

pkg/drivers/vmnet/vmnet.go

nirs · 2025-04-08T18:42:14Z

thank you @nirs for the PR I left some comments, PTAL, additionally, can you plz paste the output of following scenario

1- start minikube with vnmnet-helper
2- delete minikube home folder rm -rf ~/.minikube
3- minikube profile list
4- minikube delete --all
ideally there should be no left over process but if there is, we should create a follow up issue to clean up the abandoned proceses or left overs

We will have abandoned vfkit and if using --network vmnet-shared also abandoned vment-helper process. I don't think we have a good way to clean up since we lost the pidfiles that we use to manage these processes.

If we want to be able to clean up we can keep the pidfiles in another location like ~/Library/Application Support/minikube/ which is less likely to be deleted. But if the user deletes the directory we cannot clean up.

In the libvirt case we have libvirt daemon, running as root, so we can find the related vms and delete them. In hyperkit, qemu, or vfkit we don't have any daemon managing the vms, so no way to manage them if the user deleted our data.

We can implement this using open files. Add #20605 to track this work.

pkg/drivers/vmnet/vmnet.go

nirs · 2025-04-08T22:09:36Z

can you plz paste the output of following scenario

% minikube start --driver vfkit --network vmnet-shared
😄  minikube v1.35.0 on Darwin 15.3.2 (arm64)
✨  Using the vfkit (experimental) driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.32.2 on containerd 1.7.23 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default


% rm -rf ~/.minikube 


% minikube profile list

🤹  Exiting due to MK_USAGE_NO_PROFILE: No minikube profile was found.
💡  Suggestion: 

    You can create one using 'minikube start'.
    

% minikube delete --all
🔥  Successfully deleted all profiles


% ps
  PID TTY           TIME CMD
20208 ttys006    0:00.49 /opt/vmnet-helper/bin/vmnet-helper --fd 12 --interface-id 46628bef-da23-4300-b484-55ca102d3cbc
20209 ttys006    0:00.02 vfkit --memory 6000 --cpus 2 --restful-uri unix:///Users/nir/.minikube/machines/minikube/vfkit.sock --device vir
...

Tracked in #20605

The package manages the vmnet-helper[1] child process, providing connection to the vmnet network without running the guest as root. We will use vmnet-helper for the vfkit driver, which does not have a way to use shared network, when guests can access other guest in the network. We can use it later with the qemu driver as alternative to socket_vmnet. [1] https://github.com/nirs/vmnet-helper

Add new network option for vfkit "vmnet-shared", connecting vfkit to the vmnet shared network. Clusters using this network can access other clusters in the same network, similar to socket_vmnet with QEMU driver. If network is not specified, we default to the "nat" network, keeping the previous behavior. If network is "vmnet-shared", the vfkit driver manages 2 processes: vfkit and vmnet-helper. Like vfkit, vmnet-helper is started in the background, in a new process group, so it not terminated if the minikube process group is terminate. Since vmnet-helper requires root to start the vmnet interface, we start it with sudo, creating 2 child processes. vmnet-helper drops privileges immediately after starting the vmnet interface, and run as the user and group running minikube. Stopping the cluster will stop sudo, which will stop the vmnet-helper process. Deleting the cluster kill both sudo and vmnet-helper by killing the process group. This change is not complete, but it is good enough to play with the new shared network. Example usage: 1. Install vmnet-helper: https://github.com/nirs/vmnet-helper?tab=readme-ov-file#installation 2. Setup vmnet-helper sudoers rule: https://github.com/nirs/vmnet-helper?tab=readme-ov-file#granting-permission-to-run-vmnet-helper 3. Start 2 clusters with vmnet-shared network: % minikube start -p c1 --driver vfkit --network vmnet-shared ... % minikube start -p c2 --driver vfkit --network vmnet-shared ... % minikube ip -p c1 192.168.105.18 % minikube ip -p c2 192.168.105.19 4. Both cluster can access the other cluster: % minikube -p c1 ssh -- ping -c 3 192.168.105.19 PING 192.168.105.19 (192.168.105.19): 56 data bytes 64 bytes from 192.168.105.19: seq=0 ttl=64 time=0.621 ms 64 bytes from 192.168.105.19: seq=1 ttl=64 time=0.989 ms 64 bytes from 192.168.105.19: seq=2 ttl=64 time=0.490 ms --- 192.168.105.19 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.490/0.700/0.989 ms % minikube -p c2 ssh -- ping -c 3 192.168.105.18 PING 192.168.105.18 (192.168.105.18): 56 data bytes 64 bytes from 192.168.105.18: seq=0 ttl=64 time=0.289 ms 64 bytes from 192.168.105.18: seq=1 ttl=64 time=0.798 ms 64 bytes from 192.168.105.18: seq=2 ttl=64 time=0.993 ms --- 192.168.105.18 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.289/0.693/0.993 ms

Trailing whitespace is removed by some editors or displayed as a warning. Clean up to make it easy to make maintain this file.

The vfkit driver supports now `nat` and `vmnet-shared` network options. The `nat` option provides the best performance and is always available, so it is the default network option. The `vmnet-shared` option provides access between machines with lower performance compared to `nat`. If `vment-shared` option is selected, we verify that vmnet-helper is available. The check ensure that vmnet-helper is installed and sudoers configuration allows the current user to run vment-helper without a password. If validating vment-helper failed, we return a new NotFoundVmnetHelper reason pointing to vment-helper installation docs or recommending to use `nat`. This is based on how we treat missing socket_vmnet for QEMU driver.

nirs · 2025-04-08T22:30:20Z

@medyagh thanks for reviewing!

Changes in current version:

Fix typo in start_flags.go
Update comments for buffer sizes based on recent changes in vmnet-helper

https://github.com/kubernetes/minikube/compare/7125441c47b515bb30e4e5ff1f8323672c7d158f..4595c49781c9e25c283632264448e235cf0fce36

minikube-pr-bot · 2025-04-08T23:17:21Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20501) |
+----------------+----------+---------------------+
| minikube start | 52.7s    | 53.3s               |
| enable ingress | 17.0s    | 19.2s               |
+----------------+----------+---------------------+

Times for minikube start: 51.6s 49.9s 53.1s 54.7s 54.3s
Times for minikube (PR 20501) start: 52.5s 52.8s 52.3s 54.1s 54.6s

Times for minikube ingress: 15.6s 20.1s 18.6s 15.1s 15.6s
Times for minikube (PR 20501) ingress: 21.6s 19.1s 19.7s 16.6s 19.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20501) |
+----------------+----------+---------------------+
| minikube start | 24.1s    | 23.4s               |
| enable ingress | 12.7s    | 12.8s               |
+----------------+----------+---------------------+

Times for minikube start: 23.9s 24.3s 21.8s 25.1s 25.4s
Times for minikube (PR 20501) start: 22.0s 24.5s 22.5s 23.7s 24.2s

Times for minikube ingress: 12.9s 12.4s 13.4s 12.4s 12.4s
Times for minikube (PR 20501) ingress: 12.4s 12.4s 13.8s 12.4s 12.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20501) |
+----------------+----------+---------------------+
| minikube start | 23.8s    | 21.6s               |
| enable ingress | 35.7s    | 35.7s               |
+----------------+----------+---------------------+

Times for minikube start: 21.9s 24.0s 25.0s 24.1s 24.2s
Times for minikube (PR 20501) start: 20.3s 24.0s 21.0s 21.4s 21.2s

Times for minikube ingress: 38.9s 22.9s 38.9s 38.9s 38.9s
Times for minikube (PR 20501) ingress: 38.9s 38.9s 38.9s 38.9s 22.9s

minikube-pr-bot · 2025-04-09T16:25:17Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
KVM_Linux_crio (10 failed)	TestFunctional/serial/KubectlGetPods(gopogh)	0.00% (chart)
KVM_Linux_crio (10 failed)	TestFunctional/serial/MinikubeKubectlCmd(gopogh)	0.00% (chart)
KVM_Linux_crio (10 failed)	TestFunctional/serial/MinikubeKubectlCmdDirectly(gopogh)	0.00% (chart)
KVM_Linux_crio (10 failed)	TestFunctional/serial/SoftStart(gopogh)	6.52% (chart)
KVM_Linux_crio (10 failed)	TestFunctional/serial/ExtraConfig(gopogh)	6.98% (chart)
KVM_Linux_containerd (1 failed)	TestAddons/parallel/Registry(gopogh)	3.92% (chart)

Besides the following environments also have failed tests:

Hyper-V_Windows: 17 failed (gopogh)
Docker_Linux_crio_arm64: 1 failed (gopogh)
Docker_Cloud_Shell: 3 failed (gopogh)
Docker_Linux_crio: 1 failed (gopogh)

To see the flake rates of all tests by environment, click here.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 7, 2025

k8s-ci-robot requested review from ComradeProgrammer and medyagh March 7, 2025 23:41

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 7, 2025

nirs force-pushed the vmnet-helper branch from 20767bb to c5693e5 Compare March 8, 2025 06:35

nirs mentioned this pull request Mar 9, 2025

vfkit: More robust state management #20506

Merged