The event loops should handle the non-blocking behavior of files, fildes and sockets

[ysbaddaden]

While working on the io_uring event loop (see #15634), and the comments by @yxhuvud regarding O_NONBLOCK I investigated the non-blocking behavior related to the OS and the event loop implementations, especially the blocking argument to File and Socket and their relation to the polling (epoll, kqueue, libevent) and async event loops (iocp, uio_uring).

UNIX (poll, epoll, kqueue)

The problem on UNIX is that some file descriptors support O_NONBLOCK while others don't, and it all depends on the file type. We even stat the file descriptor to determine if we should set O_NONBLOCK or not.

Regular disk files

By default we don't set O_NONBLOCK on the underlying fd when creating File objects because disk files are always blocking (by POSIX definition). Since a disk file fd is always ready, trying to read or write, even with O_NONBLOCK, will never fail with EAGAIN.

On Linux, trying to poll or epoll a disk file fd fails with EPERM (this is an error), while on FreeBSD trying to poll or kevent a disk file fd reports it as ready (aka it's pointless to ask).

NOTE: there seems to be nothing wrong to always set O_NONBLOCK on a regular file fd. Maybe there used to be an issue with libevent? Nope, the fd would never enter libevent (read and write would have to fail with EAGAIN).

FIFO and character devices

Trying to open a fifo can block the current thread unless the O_NONBLOCK flag has been passed, in which case opening with O_RDONLY will "succeed" (the fd is created) even without a writer, while trying to open for write will fail with ENXIO until reader has opened the fifo in read mode.

We can open("fifo", O_RDONLY|O_NONBLOCK) then read fails with EGAIN and poll or epoll or kevent eventually report when a writer also opened the fifo and there is something to read.

We can open("fifo", O_WRONLY|O_NONBLOCK) but it fails with ENXIO so we don't have a fd that we could poll, epoll or kevent on 😞

We can open("fifo", O_RDWR|O_NONBLOCK) and it always succeeds because there is a reader and a writer, a fd is created, a blocking read or write will fail with EAGAIN and we can poll for both read or write.

NOTE: shall we always open as read under-the-hood? 🙈

An issue is that we'd write to the fd buffer until it gets filled, at which point it will fail with EAGAIN (and we can poll), or it fit into the buffer and we leave to do something else, without knowing if an actual reader will ever happen, or close and lose the message forever.

Alternatively, we can try to open with O_NONBLOCK then on ENXIO start a thread that will open without O_NONBLOCK, then we'd merely suspend the current fiber until the thread terminates (or call pthread_cancel on timeout). We'd only block a fiber at the expense of a thread. The other cases for ENXIO are invalid cases (file is an UNIX socket or no such device), so we'd start a thread just to get the same error, but those sound exceptional.

Sockets

By default we always set O_NONBLOCK on sockets.

I don't think the blocking args make much sense. I'm even wondering if Socket#blocking[=] methods are very useful sense. Why would you change a socket to blocking? You can't use them anymore with the polling event loops.

If you have real use-cases (not hypothetical) please report 'em 🙇

LINUX (io_uring)

The rings don't make a difference between regular disk file, fifo, character devices or sockets. Operations are always run async. Newer kernels don't make a difference between O_NONBLOCK or not, while older kernels may fail with EAGAIN when O_NONBLOCK has been set on the fd, so we should avoid setting O_NONBLOCK on file descriptors and sockets to avoid issues with older kernels.

We can use IORING_OP_OPENAT to always open the file async and never block the current thread, be they regular disk files, fifo or character devices.

The blocking args are irrelevant. Even if we need to poll, we don't need O_NONBLOCK to poll or epoll the fd (or IORING_OP_POLL), we only need it for read and write to not block.

The #blocking[=] methods would only bring some potential bugs with older kernels by setting O_NONBLOCK.

Windows (IOCP)

We can only set the OVERLAPPED flag when creating the file handle. We can't change to blocking or non blocking afterwards because we can't enable or disable OVERLAPPED.

But File always sets blocking: true and then we don't set OVERLAPPED for files, and there are special checks to call ReadFile and WriteFile directly (no calls to the event loop). We thus never use the event loop on Windows to read/write files by default, only when we specify blocking: false. That's unexpected.

@HertzDevil any technical reason for blocking read/write files on Windows? less context switches?

PROPOSAL(s)

1. Move open to the event loop implementations, and have the event loop set the non-blocking flag (or not, for example io_uring).
2. Change the blocking arg of File to false by default, or maybe nil to mean "meh, we handle it".
3. Consider deprecating the blocking args?

Nothing wrong should happen to always open with O_NONBLOCK for the polling event loops, whatever the actual file type (regular disk file, fifo, socket, ...). io_uring shouldn't set O_NONBLOCK, and IOCP would like File to be async by default, too.

Pull Requests

• Add io_uring event loop #15634
• Fix async append to file in IOCP #15681
• Fix: reopen async File passed to Process.exec and .run (win32) #15703
• Add Crystal::EventLoop::FileDescriptor#open #15750
• Ask system to decide non-blocking IO::FileDescriptor (win32) #15753
• Crystal::EventLoop::FileDescriptor#open now sets the non/blocking flag #15754
• Open special files in another thread #15768
• Refactor Socket blocking mode #15804
• Refactor IO.pipe blocking mode #15823

[yxhuvud]

If you have real use-cases (not hypothetical) please report 'em

I've seen someone choose blocking sockets exactly once. That was a single fiber telnet client, and while I do think it may have simplified the code of the author a bit, I'm also fairly certain it was possible to implement using the normal nonblocking way. But I don't remember the details of why they chose blocking sockets there.

Edit: It was on the discord. Perhaps someone else remember who it was and can point them here? It is not impossible that there may be some missing method or something that could help things out for some corner case.

[ysbaddaden]

Yes, as long as Socket#read and Socket#write act in a transparent blocking way, the protocol will work just the same (the whole point of fibers in crystal). Unless you don't want extra things to interfere concurrently?

[ysbaddaden]

I did more digging.

IO::FileDescriptor

The blocking arg of #initialize is nil by default, and we query the file type to determine whether it should actually be true or false. We do this on UNIX to distinguish regular disk files (blocking) from stream types (non-blocking).

On Windows the file type can be either of FILE_TYPE_CHAR, FILE_TYPE_DISK or FILE_TYPE_PIPE that can all be used async as per
https://learn.microsoft.com/en-us/windows/win32/sync/synchronization-and-overlapped-input-and-output and don't need that check. We can always set the FILE_FLAG_OVERLAPPED on Windows.

In practice, I believe we usually call IO::FileDescriptor#initialize where blocking is a Bool, so we never actually run the check, except for a few calls in the specs.

IO.pipe

We can pass read_blocking and write_blocking to IO.pipe, and we do need these right in the stdlib: for example when redirecting IO when spawning child processes, because the fildes passed to the spawned process as stdio must be blocking.

That means the read_blocking and write_blocking of IO.pipe are important, and even a requirement on Win32 that needs to know these before creating the pipe. Now, we don't need the blocking arg to IO::FileDescriptor#initialize because they're redundant on Win32, and the UNIX version could just set O_NONBLOCK (or not) by itself AND IO::FileDescriptor never creates the system HANDLE or fd, it only wraps a previously created one.

[ysbaddaden]

I started hacking, and I realize we do use the blocking arg everywhere 🙈

I'm changing it to default to nil everywhere (meaning unspecified), and I'm introducing a Crystal::EventLoop.default_blocking to ask the current event loop for the actual value before calling to the Crystal::System types that will apply the value ASAP, for example LibC.socket(family, type | SOCK_CLOEXEC | SOCK_O_NONBLOCK, protocol) to spare a couple fcntl afterwards.

Fun fact: until io_uring the default_blocking value will always be false because the event loops want O_NONBLOCK or FILE_FLAG_OVERLAPPED.

[ysbaddaden]

Damn, I should have experimented instead of assuming: while we can open a FIFO file with O_RDONLY and O_NONBLOCK, further attempts to read will return 0 bytes read (aka EOF) on Linux and *BSD, with the exception of FreeBSD that fails with EAGAIN.

LibC.mkfifo("fifo", File::DEFAULT_CREATE_PERMISSIONS)

p! fd = LibC.open("fifo", LibC::O_RDONLY | LibC::O_NONBLOCK)

if fd == -1
  p! Errno.value if fd == -1
else
  buf = uninitialized UInt8[1024]
  p! ret = LibC.read(fd, buf.to_unsafe, buf.size)
  p! Errno.value if ret == -1
end

LibC.unlink("fifo")

And I should have read the POSIX specification, because returning 0 is the expected behavior:

When attempting to read from an empty pipe or FIFO: If no process has the pipe open for writing, read() shall return 0 to indicate end-of-file.

https://pubs.opengroup.org/onlinepubs/9799919799/functions/read.html#tag_17_476

[ysbaddaden]

We can open a file for WRITE with O_NONBLOCK: it always succeeds for regular disk files and fifo/pipes with a reader, and otherwise fail with ENXIO. We can then retry without O_NONBLOCK in another thread, which allows to suspend the current fiber until the thread terminates (good) instead of blocking all the fibers in the current thread (bad). Eventually we set O_NONBLOCK on the fd.

This has an impact on performance, but that only impacts opening special files. Regular files will just be opened immediately without overhead.

We can't open a file for READ with O_NONBLOCK because unlike WRITE fifo/pipes won't fail with ENXIO and read won't fail with EAGAIN but report EOF (damn). We can't use the same fallback as for WRITE, and obviously won't always start a thread to open a file (talk about inefficient).

We could always call stat to know the file type, and in case the file isn't a regular disk file we'd try to open in a thread directly. What's the cost of always calling stat?

A quick benchmark on my desktop (pcie4 nvme):

	regular disk file	fifo
stat	289ns	290ns
open+close	575ns	791ns
stat+open+close	873ns	1090ns

It introduces a noticeable overhead on a fast local disk (+30% to +50%). I can still open/close one million files in less than a second, but a slower disk or a remote mounted disk, might be more impacted, and that would be for every file open, not just as a fallback for special files 😢

Note: this only matters for the polling event loops, io_uring (and I assume IOCP, to be verified) just don't care.

Note: while we could always open as READ + WRITE that's a bad idea: we could inadvertently read or write and mess with the IO (bad) instead of raising an error. We would also have no mean to know about EPIPE or EOF since there would always be a writer and a reader connected.

[ysbaddaden]

CLARIFICATION: the above issue only applies to open.

Setting O_NONBLOCK after open is working correctly: they fail with EAGAIN for special files when the kernel buffer is respectively empty or full for special file types (fifos, pipes, character devices); regular disk files won't trigger EAGAIN.

The polling event loops can thus always set O_NONBLOCK. We just can't open directly as non-blocking and it might block the current thread for a while for fifo and character devices that need to wait for another reader/writer to open or until the device is connected.

We can refactor how we implement the blocking arg to File.new and File.open. The documentation merely states that it's true by default, must be set to false on UNIX when opening special files, and can be set to nil to be determined automatically; the following change would still do that:

• epoll, kqueue, libevent:

  • blocking: true: always open as blocking, then set O_NONBLOCK;
  • blocking: false: always open as blocking in a Thread, then set O_NONBLOCK;
  • blocking: nil: call stat to determine true or false, then do either of the above.

• iocp, io_uring:

  • the arg has no effect (it's always true).

This would have zero impact on opening regular disk files, unless you always pass blocking: nil (developer decision) or blocking: false because you expect a special file and don't want to block the current scheduler.

Opening a special file will have an impact (stat and/or Thread.new) and still require specific developer attention which ain't nice to set the blocking arg, but it should be better than the current state (block a thread/scheduler).

[ysbaddaden]

The current progress on the experimentation is at master...ysbaddaden:crystal:refactor/blocking-behavior-3.

It helped uncover a limitation on Windows: #15681, and now it's down to a couple failures. The actual exception outlines another limitation:

Unhandled exception: Non-blocking streams are not supported in `Process.exec` (IO::Error)
  from src\crystal\system\win32\process.cr:387 in 'reopen_io'

[ysbaddaden]

We want to run a command and either want to close the output (actually redirect to File::NULL) or want to redirect to a user specified File object.

In both cases we have a File that is now always opened with FILE_FLAG_OVERLAPPED and makes it impossible to use for redirecting stdio 😭

Handle does not support synchronous operations. The parameters to the FileStream constructor may need to be changed to indicate that the handle was opened asynchronously (that is, it was opened explicitly for overlapped I/O).

The NUL issue can be fixed with a system method to always open File::NULL as blocking.

I'm trying to find a way to reopen the HANDLE without FILE_FLAG_OVERLAPPED, for example with ReOpenFile but so far it doesn't work 😞

[ysbaddaden]

Aka: I just rediscovered #6957 (comment) 🤦

The arguments of input, output and error of Process.run must be handles without FILE_FLAG_OVERLAPPED.

[ysbaddaden]

Stupid idea, but it works: we keep the filename as File#path, so we can just open the file again with LibC.CreateFileW(src_io.path, ...).

It doesn't work for the redirect close, which means that it ain't a silver bullet (or there's something wrong with the chosen disposition / attributes). Nope, it doesn't call System::File.reopen_io, and the issue has nothing to with closing the output. It reproduces with a mere:

p Process.run("cmd.exe", {"/c", "dir"})
# => Process:Status[1]

It should be Process::Status[0] 🤨

[ysbaddaden]

By fixing #reopen_io I fixed the problem for Process.exec but now the problem is with Process.run that calls #handle_from_io, and Process.run closes all stdio by default 🙃

Process.run("cmd.exe", {"/c", "dir"}, error: Process::Redirect::Inherit)
# The system cannot write to the specified device.

So, #handle_from_io must check for nonblocking and if so reopen the file before calling DuplicateHandle which probably can't be told to not copy FILE_FLAG_OVERLAPPED.

[ysbaddaden]

The "just open File#path again" is working nicely. We can even spare the call to DuplicateHandle by opening with the inheritable flag.