[kube-prometheus-stack] node-exporter crashes at startup #467

davgia · 2020-12-09T09:51:37Z

Describe the bug
Node exporter pod underlying container fails to startup. This is the error reported by the deamon:

Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount

This error is related to #325 and it appears to be caused by this pull request. I've create this issue as suggested by @monotek in #80 (comment) (it appears that the original reporter of this issue has not created a new issue).

Moreover, I've tried to change kube-prometheus-stack chart manually to revert the commit above and I am able to deploy the chart without any error.

I am using docker-desktop (v2.5.0.1) on Windows 10 1909 and I have enabled Kubernetes cluster from it.

I'm not sure if this error is caused by docker-desktop, in any case, if it does not happen in other type of clusters, I would suggest to make the mount rootfs mountPropagation configurable via the chart values (and maybe enable it by default if you think it should not be optional) to prevent this error.

Another (better) solution would be to mount / as shared, but I've not found a way to do it.

Version of Helm and Kubernetes:

Helm Version:

version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.11"}

Kubernetes Version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Which chart: kube-prometheus-stack

Which version of the chart: 12.7.0

What happened: node-exporter pod fails to startup

What you expected to happen: node-exporter should not fail

How to reproduce it:

No additional change in values.yaml file to reproduce.

The helm command that you execute and failing/misfunctioning:

helm install prom-stack prometheus-community/kube-prometheus-stack --version v12.7.0 -n monitoring

The text was updated successfully, but these errors were encountered:

radhus · 2020-12-12T16:24:27Z

I have the same issue, running a k3s v1.18 cluster on Linux (Alpine) if that helps rule anything out.

Gowlar · 2020-12-15T10:27:58Z

I have the this issue also on Docker Desktop for Mac 3.0.0

mvaldes14 · 2020-12-18T19:24:21Z

Same issue on Docker for Windows.

Client: Docker Engine - Community
 Version:           20.10.0

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)

helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

kubectl.exe version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

jonyjalfon94 · 2020-12-19T14:28:07Z

Same issue in docker for windows ( Using Hyper-V backend ) with Kubernetes v1.19.3 and Helm v3.4.1.

msergei · 2020-12-23T12:25:00Z

The same issue with Docker Desktop for Mac v.3.

iemkamran · 2021-01-04T19:56:03Z

Same issue using Docker Desktop for windows 10 on WSL2

dshah2012 · 2021-01-07T05:08:03Z

Facing the same issue using Docker Desktop for windows 10.

DockerEngine Version: 20.10.2
Helm version: v3.3.1
Kubernetes version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

3wolf · 2021-01-10T16:32:35Z

+1

aagmangour · 2021-01-12T14:01:47Z

+1
Same issue on docker desktop for Mac. v 2.5.0.1
Kubernete version 1.19.3

ktzsolt · 2021-01-17T14:56:01Z

+1 for Docker Desktop for Windows 10 (Pro, version 2004) on WSL2

nunoguerreirorosa · 2021-01-17T14:57:16Z

+1

moatorres · 2021-01-31T23:08:02Z

+1

v1lev · 2021-02-02T08:25:08Z

+1

kingpeti · 2021-02-05T04:19:31Z

+1

golosegor · 2021-02-15T11:02:27Z

+1

prasetiyohadi · 2021-02-18T02:24:27Z

+1

monotek · 2021-02-18T09:51:09Z

Please stop the "+1".
It just creates noise.

Alternatively you just could add a thumbs up via the smiley function.

kristijorgji · 2021-02-18T21:58:36Z

Same issue for me, this + another one
#687
In above issue are all my versions of toolings and steps to reproduce

Prometheus is exporting nothing, kube-state-metrics if i port forward has some metrics, but querying prometheus is totally empty, node exporter broken

anktsrkr · 2021-03-06T12:34:01Z

@monotek what is the alternate to make it work?

Not always possible to mount / from host, see issue prometheus-community#467

Not always possible to mount / from host, see issue prometheus-community#467 Signed-off-by: William Johansson <[email protected]>

ortizjavier · 2021-03-09T10:28:59Z

I had the same issue and as a workaround, I removed the mountPropagation to use the default one.
It seems that everything is better but I'm not sure to understand what this change means.

kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

* [prometheus] make path.rootfs condition Not always possible to mount / from host, see issue #467 Signed-off-by: William Johansson <[email protected]> * [prometheus] bump minor version Signed-off-by: William Johansson <[email protected]> * [prometheus] bump minor version Signed-off-by: William Johansson <[email protected]> Co-authored-by: André Bauer <[email protected]>

davgia · 2021-03-11T12:17:13Z

@monotek I think this issue is not solved. It was opened for kube-prometheus-stack. The pull request by @radhus
changed only the prometheus chart that it's not a dependency of the kube-prometheus-stack chart.

The changes made to the prometheus chart must be replicated in the prometheus-node-exporter chart too.

Let me know if I'm correct, I will create a pull request myself if needed.

monotek · 2021-03-11T12:21:50Z

I think this was automatically closed because the pr #737 was merged and this issue was mentioned...
Reopening...

ozbillwang · 2021-06-25T13:18:54Z

you don't have to copy the whole values.yaml

with below way, you can fix the problem easily

$ cat values.yaml

prometheus-node-exporter:
  hostRootFsMount: false
  
$ helm list
$ helm upgrade --install my-prometheus prometheus-community/kube-prometheus-stack -f values.yaml

my-prometheus is my prometheus chart name, you can replace with yours

bluetechy · 2021-07-25T05:45:12Z

I had to do the following to get it to work in my values.yaml file because I am running kube-prometheus-stackas a dependency in my helm chart:

kube-prometheus-stack:
    prometheus-node-exporter:
        hostRootFsMount: false

OlamideOl1 · 2021-10-30T14:03:27Z

had to do the following to get it to work with my helm install command for Docker-Desktop for Mac:

--set nodeExporter.hostRootfs=false

jamesbright · 2021-11-02T04:23:59Z

I had the same issue and as a workaround, I removed the mountPropagation to use the default one. It seems that everything is better but I'm not sure to understand what this change means.
kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

Thanks a lot, this worked.

…rter Refer to prometheus-community/helm-charts@201e9e8, https://github.com/prometheus-community/helm-charts/blob/prometheus-node-exporter-1.17.0/charts/kube-prometheus-stack/Chart.yaml 14.7.1 is the lowerest version that supports hostRootFsMount. Without the option to set hostRootFsMount as false, the node-exporter would crash in some cases, refer to https://github.com/prometheus-community/helm-charts/issues/467\#issuecomment-802642666.

…rter Refer to prometheus-community/helm-charts@201e9e8, https://github.com/prometheus-community/helm-charts/blob/prometheus-node-exporter-1.17.0/charts/kube-prometheus-stack/Chart.yaml 14.7.1 is the lowerest version that supports hostRootFsMount. Without the option to set hostRootFsMount as false, the node-exporter would crash in some cases, refer to prometheus-community/helm-charts#467.

Refer to prometheus-community/helm-charts@201e9e8, https://github.com/prometheus-community/helm-charts/blob/prometheus-node-exporter-1.17.0/charts/kube-prometheus-stack/Chart.yaml 14.7.1 is the lowerest version that supports hostRootFsMount. Without the option to set hostRootFsMount as false, the node-exporter would crash in some cases, refer to prometheus-community/helm-charts#467.

vg185118 · 2022-03-04T18:14:25Z

had to do the following to get it to work with my helm install command for Docker-Desktop for Mac:

--set nodeExporter.hostRootfs=false

helm install {Name} prometheus-community/kube-prometheus-stack --set prometheus-node-exporter.hostRootFsMount.enabled=false

doublespaces · 2022-03-05T02:15:04Z

For those using the kube-prometheus-stack where prometheus-node-exporter is a dependency, you need to add this change to the prometheus-node-exporter values file and not the kube-prometheus-stack values file. Change the enabled value to false:

# Mount the node's root file system (/) at /host/root in the container
hostRootFsMount:
  enabled: false
  # Defines how new mounts in existing mounts on the node or in the container
  # are propagated to the container or node, respectively. Possible values are
  # None, HostToContainer, and Bidirectional. If this field is omitted, then
  # None is used. More information on:
  # https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
  mountPropagation: HostToContainer

filwu8 · 2022-05-07T11:41:40Z

PS C:\Users\do> kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'
Error from server (NotFound): daemonsets.apps "monitoring-prometheus-node-exporter" not found

puitiza · 2022-09-09T02:26:25Z

For those using the kube-prometheus-stack where prometheus-node-exporter is a dependency, you need to add this change to the prometheus-node-exporter values file and not the kube-prometheus-stack values file. Change the enabled value to false:
# Mount the node's root file system (/) at /host/root in the container
hostRootFsMount:
  enabled: false
  # Defines how new mounts in existing mounts on the node or in the container
  # are propagated to the container or node, respectively. Possible values are
  # None, HostToContainer, and Bidirectional. If this field is omitted, then
  # None is used. More information on:
  # https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
  mountPropagation: HostToContainer

@doublespaces thanks your comment I could add those values.
I'm more visual personal so If someone ask where is it ? here , look below.

mahesh2121 · 2022-09-11T03:48:32Z

@puitiza thanks for solution it works for me just updating the values.

helbing · 2022-12-11T14:03:29Z

The prometheus-node-exporter is version 4 now, you can use the values.yaml as follows to make it work.

prometheus-node-exporter:
  hostRootFsMount:
    enabled: false

GitJaehyeonLee · 2023-07-25T07:53:52Z

I found another way.

Because My error was another folder, But the solution was the same.

if your error is
Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount

OS [Shell Command]:
mount --make-rshared /

else if your error is
Error: failed to start container "node-exporter": Error response from daemon: path /sys is mounted on /sys but it is not a shared or slave mount

OS [Shell Command]:
mount --make-rshared /sys

But I haven't been able to confirm that it works even after rebooting.
If that doesn't work, you can consider fstab.

yybht155 · 2023-07-27T10:27:38Z

我找到了另一种方法。

因为我的错误是另一个文件夹，但解决方案是相同的。

如果你的错误是 Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount

操作系统[外壳命令]： mount --make-rshared /

否则如果你的错误是 Error: failed to start container "node-exporter": Error response from daemon: path /sys is mounted on /sys but it is not a shared or slave mount

操作系统[外壳命令]： mount --make-rshared /sys

但我无法确认它即使在重新启动后也能工作。如果这不起作用，您可以考虑 fstab。

In a k3s cluster on wls2 this method solved the problem for me!

patrykpanter · 2023-08-30T14:36:06Z

OS [Shell Command]:
mount --make-rshared /

Worked for me in a proxmox container (Ubuntu 18.04.6).
Thanks.

AxelTahmid · 2023-12-14T16:56:11Z

For those using the kube-prometheus-stack where prometheus-node-exporter is a dependency, you need to add this change to the prometheus-node-exporter values file and not the kube-prometheus-stack values file. Change the enabled value to false:
# Mount the node's root file system (/) at /host/root in the container
hostRootFsMount:
  enabled: false
  # Defines how new mounts in existing mounts on the node or in the container
  # are propagated to the container or node, respectively. Possible values are
  # None, HostToContainer, and Bidirectional. If this field is omitted, then
  # None is used. More information on:
  # https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
  mountPropagation: HostToContainer

Thanks. this worked for me, added it my command

helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
    --namespace monitoring --create-namespace \
    --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
    --set prometheus-node-exporter.hostRootFsMount.enabled=false \
    --set prometheus-node-exporter.hostRootFsMount.mountPropagation='HostToContainer' \

carlosschuenck-bjss · 2024-07-24T15:14:44Z

I solved using:

helm install prometheus prometheus-community/kube-prometheus-stack --set prometheus-node-exporter.hostRootFsMount.enabled=false

- Main examples: - https://github.com/oijkn/Docker-Raspberry-PI-Monitoring/blob/main/docker-compose.yml#L46-L65 - https://github.com/prometheus/node_exporter/tree/f35a592dd65d6b3488426a789b328c8ca5b2bc22?tab=readme-ov-file#docker - https://theawesomegarage.com/blog/monitor-your-raspberry-pi-with-prometheus-and-grafana - I attempted to use `network_mode: host` as in the node_exporter README example, but then I needed to add `extra_hosts: ["host.docker.internal:host-gateway"]` to the prometheus service and `- targets: ['host.docker.internal:9100']` to prometheus.yml as recommended by https://stackoverflow.com/a/66689508 but the connection was always refused - Additionally, I think this would have exposed the :9100/metrics endpoint publicly - The downside of not using `network_mode: host` is that the network metrics only include network traffic that goes through the docker container, and not the node itself. This should be fine though, since theoretically all network traffic that isn't admin-related (i.e. me) should go through the docker containers - I needed to remove rslave from `/:/host:ro,rslave` or else `dc up` failed on mac. - https://docs.docker.com/engine/storage/bind-mounts/#configure-bind-propagation - prometheus/node_exporter#2002 - prometheus-community/helm-charts#467 - prometheus/node_exporter#610 (comment)

davgia added the bug Something isn't working label Dec 9, 2020

radhus added a commit to radhus/prometheus-helm-charts that referenced this issue Mar 7, 2021

[prometheus] make path.rootfs condition

45e7a3c

Not always possible to mount / from host, see issue prometheus-community#467

radhus mentioned this issue Mar 7, 2021

[prometheus] make path.rootfs conditional #737

Merged

3 tasks

radhus added a commit to radhus/prometheus-helm-charts that referenced this issue Mar 7, 2021

[prometheus] make path.rootfs condition

c1dbc3a

Not always possible to mount / from host, see issue prometheus-community#467 Signed-off-by: William Johansson <[email protected]>

monotek closed this as completed in #737 Mar 10, 2021

monotek reopened this Mar 11, 2021

davgia mentioned this issue Mar 11, 2021

[prometheus-node-exporter] Allow to opt-out node exporter rootfs mount #756

Closed

3 tasks

poblin-orange mentioned this issue May 18, 2021

Bosh stemcell default / mount option not compatible with k8s (node_exporter / longhorn) orange-cloudfoundry/k3s-boshrelease#41

Closed

weibeld mentioned this issue Dec 9, 2021

[prometheus-node-exporter] Parameterise host root FS mount propagation #1583

Merged

3 tasks

Snehil03 mentioned this issue Jan 19, 2022

[prometheus-kube-stack] : Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount #1726

Closed

aaronwng mentioned this issue Jan 28, 2022

Upgrade Kube-Prometheus-Stack to support hostRootFsMount SumoLogic/sumologic-kubernetes-collection#2053

Closed

3 tasks

ktsakalozos mentioned this issue Feb 21, 2022

WSL2: Prometheus Daemon Set "node-exporter" failed to run canonical/microk8s#2933

Closed

manumura pushed a commit to manumura/argocd-demo that referenced this issue Apr 1, 2022

fix prometheus node-exporter prometheus-community/helm-charts#467

8000982

MPThLee mentioned this issue May 27, 2022

An option to set hostRootFsMount? timescale/tobs#391

Closed

chrkl mentioned this issue Jun 7, 2022

Node-exporter failed to start kyma-project/kyma#14505

Closed

lhotari mentioned this issue Apr 11, 2024

Fix prometheus node-exporter crashloop apache/pulsar-helm-chart#488

Merged

1 task

QkiZMR mentioned this issue Jan 27, 2025

Observability plugin: node-exporter cannot start in strict mode because it fails to mount the root directory canonical/microk8s#4783

Open

diablinux added a commit to diablinux/flux-minikube that referenced this issue Mar 25, 2025

https://github.com/prometheus-community/helm-charts/issues/467

da3da80

[kube-prometheus-stack] node-exporter crashes at startup #467

[kube-prometheus-stack] node-exporter crashes at startup #467

Comments

davgia commented Dec 9, 2020 • edited Loading

radhus commented Dec 12, 2020

Gowlar commented Dec 15, 2020

mvaldes14 commented Dec 18, 2020 • edited Loading

jonyjalfon94 commented Dec 19, 2020

msergei commented Dec 23, 2020

iemkamran commented Jan 4, 2021

dshah2012 commented Jan 7, 2021

3wolf commented Jan 10, 2021

aagmangour commented Jan 12, 2021

ktzsolt commented Jan 17, 2021 • edited Loading

nunoguerreirorosa commented Jan 17, 2021

moatorres commented Jan 31, 2021

v1lev commented Feb 2, 2021

kingpeti commented Feb 5, 2021

golosegor commented Feb 15, 2021

prasetiyohadi commented Feb 18, 2021

monotek commented Feb 18, 2021 • edited Loading

kristijorgji commented Feb 18, 2021 • edited Loading

anktsrkr commented Mar 6, 2021

ortizjavier commented Mar 9, 2021

davgia commented Mar 11, 2021

monotek commented Mar 11, 2021 • edited Loading

ozbillwang commented Jun 25, 2021

bluetechy commented Jul 25, 2021

OlamideOl1 commented Oct 30, 2021 • edited Loading

jamesbright commented Nov 2, 2021

vg185118 commented Mar 4, 2022 • edited Loading

doublespaces commented Mar 5, 2022 • edited Loading

filwu8 commented May 7, 2022

puitiza commented Sep 9, 2022

mahesh2121 commented Sep 11, 2022

helbing commented Dec 11, 2022

GitJaehyeonLee commented Jul 25, 2023 • edited Loading

yybht155 commented Jul 27, 2023

patrykpanter commented Aug 30, 2023 • edited Loading

AxelTahmid commented Dec 14, 2023

carlosschuenck-bjss commented Jul 24, 2024

davgia commented Dec 9, 2020 •

edited

Loading

mvaldes14 commented Dec 18, 2020 •

edited

Loading

ktzsolt commented Jan 17, 2021 •

edited

Loading

monotek commented Feb 18, 2021 •

edited

Loading

kristijorgji commented Feb 18, 2021 •

edited

Loading

monotek commented Mar 11, 2021 •

edited

Loading

OlamideOl1 commented Oct 30, 2021 •

edited

Loading

vg185118 commented Mar 4, 2022 •

edited

Loading

doublespaces commented Mar 5, 2022 •

edited

Loading

GitJaehyeonLee commented Jul 25, 2023 •

edited

Loading

patrykpanter commented Aug 30, 2023 •

edited

Loading