os.link(..., follow_symlinks=True) broken on Linux

[jo-he]

BPO	37612
Nosy	@larryhastings, @takluyver, @serhiy-storchaka, @eryksun, @jo-he
PRs	bpo-37612: always call linkat() from os.link(), if available #14843 gh-81793: always call linkat() from os.link(), if available #24997

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-07-17.23:04:55.849>
labels = ['extension-modules', 'type-bug', '3.8', '3.9', '3.10']
title = 'os.link(..., follow_symlinks=True) broken on Linux'
updated_at = <Date 2021-03-23.17:43:01.277>
user = 'https://github.com/jo-he'

bugs.python.org fields:

activity = <Date 2021-03-23.17:43:01.277>
actor = 'takluyver'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2019-07-17.23:04:55.849>
creator = 'jo-he'
dependencies = []
files = []
hgrepos = []
issue_num = 37612
keywords = ['patch']
message_count = 4.0
messages = ['348086', '348103', '348110', '348111']
nosy_count = 5.0
nosy_names = ['larry', 'takluyver', 'serhiy.storchaka', 'eryksun', 'jo-he']
pr_nums = ['14843', '24997']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37612'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-81793: Always call linkat() from os.link(), if available #132517
• gh-81793: Skip tests for os.link(follow_links=) on Android #133386
• gh-81793: Skip tests for os.link() to symlink on Android #133388
• gh-81793: Replace skip decorators with specific helpers #133493

[jo-he]

Regarding link() POSIX states (https://pubs.opengroup.org/onlinepubs/9699919799/functions/link.html):

"If path1 names a symbolic link, it is implementation-defined whether link() follows the symbolic link, or creates a new link to the symbolic link itself."

In Linux, link() does _not_ follow symlinks (http://man7.org/linux/man-pages/man2/link.2.html):

"By default, linkat(), does not dereference oldpath if it is a symbolic link (like link())."

But Python 3 assumes the opposite to be always the case:

cpython/Modules/posixmodule.c

Line 3517 in 8cb65d1

(!follow_symlinks))

...which suits e.g. NetBSD (https://netbsd.gw.com/cgi-bin/man-cgi?link+2):

"When operating on a symlink, link() resolves the symlink and creates a hard link on the target."

Therefore, I recommend to always call linkat(), if the platform provides it. That's the modern superset of link() with clearly defined behavior.

Here are some commands to reproduce the issue on Linux (should hard link 'file' -> 'link', but tries '/tmp/symlink' -> 'link'):

~$ : >file
~$ ln -s "$PWD/file" /tmp/symlink
~$ strace -e link,linkat python -c 'import os; os.link("/tmp/symlink", "link", follow_symlinks=True)'
link("/tmp/symlink", "link")            = -1 EXDEV (Cross-device link)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OSError: [Errno 18] Cross-device link: '/tmp/symlink' -> 'link'
+++ exited with 1 +++

For comparison, calling linkat() without AT_SYMLINK_FOLLOW results in the same error:

~$ strace -e link,linkat python -c 'import os; os.link("/tmp/symlink", "link", follow_symlinks=False)'
linkat(AT_FDCWD, "/tmp/symlink", AT_FDCWD, "link", 0) = -1 EXDEV (Cross-device link)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OSError: [Errno 18] Cross-device link: '/tmp/symlink' -> 'link'
+++ exited with 1 +++

Currently, the only way to call linkat() with AT_SYMLINK_FOLLOW from Python, is to provide a directory file descriptor != AT_FDCWD:

~$ strace -e link,linkat python -c 'import os; d=os.open(".", 0); os.link("/tmp/symlink", "link", dst_dir_fd=d, follow_symlinks=True)'
linkat(AT_FDCWD, "/tmp/symlink", 3, "link", AT_SYMLINK_FOLLOW) = 0
+++ exited with 0 +++

[serhiy-storchaka]

Not always linkat() can be used instead of link(). But on Windows (where there is no linkat()) os.link() creates a new link to the symbolic link itself. This is yet one argument for making follow_symlinks=False by default and changing the default behavior of os.link() on NetBSD.

[eryksun]

on Windows (where there is no linkat()) os.link() creates a new link
to the symbolic link itself.

Yes, CreateHardLinkW opens the source file by calling NtOpenFile with the option FILE_OPEN_REPARSE_POINT. So the behavior is follow_symlinks=False.

Note, however, that Windows has distinct file and directory reparse points, so we can't hardlink to a directory symlink, or any other type of directory reparse point such as a junction mountpoint. In Unix, follow_symlinks=False (if implemented) allows creating a hardlink to a symlink that targets a directory.

Also, I noticed that I can pass follow_symlinks=False in Windows, but this should raise NotImplementedError. It's supposed to be checked via follow_symlinks_specified().

[jo-he]

The problem that POSIX does not define the behavior of link() regarding symlinks (and that Unix implementations differ indeed), is independent from Python's os.link() defaults.

Since it makes no sense to call link(), when linkat() is available, I propose this change:

--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -3512,15 +3512,11 @@ os_link_impl(PyObject *module, path_t *src, path_t *dst, int src_dir_fd,
 #else
     Py_BEGIN_ALLOW_THREADS
 #ifdef HAVE_LINKAT
-    if ((src_dir_fd != DEFAULT_DIR_FD) ||
-        (dst_dir_fd != DEFAULT_DIR_FD) ||
-        (!follow_symlinks))
-        result = linkat(src_dir_fd, src->narrow,
-            dst_dir_fd, dst->narrow,
-            follow_symlinks ? AT_SYMLINK_FOLLOW : 0);
-    else
+    result = linkat(src_dir_fd, src->narrow, dst_dir_fd, dst->narrow,
+                    follow_symlinks ? AT_SYMLINK_FOLLOW : 0);
+#else
+    result = link(src->narrow, dst->narrow);
 #endif /* HAVE_LINKAT */
-        result = link(src->narrow, dst->narrow);
     Py_END_ALLOW_THREADS

 if (result)

This fix also simplifies the code, and should be safe for back-porting.

Whether Python's defaults should be changed regarding Windows and those (mostly obsolete) Unix platforms that do not provide linkat(), is another question.

[serhiy-storchaka]

I created an alternate PR #132517 based on work by @jo-he and @takluyver. The code is now clearer and all special cases are handled.

• On Linux, os.link() without optional arguments now follows symlinks. This is a breaking change, but I think that this is unavoidable.
• On Linux, os.link() with follow_symlinks=True now follows symlinks, and with follow_symlinks=False does not follow.
• On OpenIndiana (and perhaps on Solaris), the same changes as on Linux.
• On Windows, os.link() with follow_symlinks=True now raises an error.
• On macOS, os.link() with follow_symlinks=False now raises an error if linkat() is not available at runtime.
• On other platforms, os.link() with follow_symlinks now raises an error if link() does not have the specified behavior.

[serhiy-storchaka]

Interesting, on OpenIndiana the link() system function follows symlinks, but the ln command does not and have no option to follow. This is the only platform with such difference, and I wonder why. If ln was implemented via link(), it would follow symplink. If ln was implemented via linkat(), it would have options to follow on not follow symplink.