Bind mount of the top binary directory cause hardlinking to fail

[flx42]

NVIDIA/nvidia-docker#562

Easiest repro:

$ docker run --runtime=runc -ti nvidia/cuda:9.0-base ln /usr/bin/find /usr/myfind
$ docker run --runtime=nvidia -ti nvidia/cuda:9.0-base ln /usr/bin/find /usr/myfind
ln: failed to create hard link '/usr/myfind' => '/usr/bin/find': Invalid cross-device link

Comes from this code:

libnvidia-container/src/nvc_mount.c

Lines 50 to 52 in 3c1e925

	/* Bind mount the top directory and every files under it with read-only permissions. */
	if (xmount(err, path, path, NULL, MS_BIND, NULL) < 0)
	goto fail;

I feel this call should be better documented, I know there was a good reason for doing that, but I don't remember it.

[3XX0]

I did that for mount slave propagation initially and it's also useful for error teardown and driver unbinding from the container. Agreed it's badly documented, so as the rest of the code :)

I guess the error comes from overlay trying to hardlink across layers without a copy up. I remember some changes along those lines in recent kernels, it might be worth testing this as well as other storage drivers (e.g. AUFS).

[flx42]

I guess the error comes from overlay trying to hardlink across layers without a copy up.

That explains the error with yum install -y git, but the ln /usr/bin/find /usr/myfind will fail on all storage drivers, including vfs. That's because it indeed becomes a cross-device link with the code we have :)

I remember some changes along those lines in recent kernels, it might be worth testing this as well as other storage drivers (e.g. AUFS).

FWIW, AUFS on 4.10 (xenial) fails with the yum install -y git.

[3XX0]

Yeah cross-mount rename and link are not permitted:
https://github.com/torvalds/linux/blob/master/fs/namei.c#L4272-L4274
https://github.com/torvalds/linux/blob/master/fs/namei.c#L4525-L4527

We might get away with most of those if we overlay the entire partition but I'm not particularly fond of this idea.

[3XX0]

And oddly enough, this seems to be deliberate

http://lkml.iu.edu/hypermail/linux/kernel/0305.0/1222.html

[flx42]

Right, so looks like we don't have a choice so we have to bite the bullet and remove this top-level bind mount? At the cost of harder teardown.