Direct I/O file system flag not supported

[nazar-pc]

I was testing my direct/unbuffered/unchached I/O abstractions with isolation disabled and it turned out that the file file opening flag for direct I/O is not supported:

error: unsupported operation: unsupported flags 0x4000
    --> /home/nazar-pc/.rustup/toolchains/nightly-2025-07-10-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/fs/unix.rs:1185:36
     |
1185 |         let fd = cvt_r(|| unsafe { open64(path.as_ptr(), flags, opts.mode as c_int) })?;
     |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ unsupported operation occurred here

The usage was:

#[cfg(target_os = "linux")]
{
    use std::os::unix::fs::OpenOptionsExt;

    options.custom_flags(libc::O_DIRECT);
}
#[cfg(windows)]
{
    use std::os::windows::fs::OpenOptionsExt;

    options.custom_flags(
        windows::Win32::Storage::FileSystem::FILE_FLAG_WRITE_THROUGH.0
            | windows::Win32::Storage::FileSystem::FILE_FLAG_NO_BUFFERING.0,
    );
}
let file = options.open(path)?;

#[cfg(target_os = "macos")]
{
    use std::os::unix::io::AsRawFd;

    if unsafe { libc::fcntl(file.as_raw_fd(), libc::F_NOCACHE, 1) } != 0 {
        return Err(io::Error::last_os_error());
    }
}

It'd be very nice for all of these to be supported.

[saethlin]

What are you expecting this flag to change about the API of the I/O calls?

[nazar-pc]

Linux and Windows flags impose memory alignment requirements, macOS can probably just be no-op:
https://learn.microsoft.com/en-us/windows/win32/fileio/file-buffering#alignment-and-file-access-requirements
https://man7.org/linux/man-pages/man2/open.2.html

[bjorn3]

Has anything on the Linux side changed since Linus wrote https://yarchive.net/comp/linux/o_direct.html to make O_DIRECT actually safe to use?

[RalfJung]

Thanks for the report!

The docs for Linux say

   The O_DIRECT flag may impose alignment restrictions on the length
   and address of user-space buffers and the file offset of I/Os.  In
   Linux alignment restrictions vary by filesystem and kernel version
   and might be absent entirely.  The handling of misaligned O_DIRECT
   I/Os also varies; they can either fail with EINVAL or fall back to
   buffered I/O.

so it seems we'd be entirely correct if we just completely ignored the flag... I just don't know if that would be useful.

[nazar-pc]

It'll be useful in a sense that application compiles and runs unmodified, absolutely. But I'd be extra nice if it also checked the alignment, it is something basically everyone using the API will most likely hit in real world. Usually it means alignment needs to be 512 bytes, but increasingly often it is actually 4096 bytes. Anything else is possible, but extremely unlikely, so 4096 would be my preference.

[saethlin]

Usually it means alignment needs to be 512 bytes, but increasingly often it is actually 4096 bytes.

What is this statement based on?

[RalfJung]

Even violation of the alignment requirement can still fall back to buffered I/O so even that is, in general, not observable.

[nazar-pc]

What is this statement based on?

Lots of experimentation and user support with a software that does rely on direct I/O for acceptable performance characteristics (especially on Windows, where it is essentially unusable otherwise).

Software works with SSDs, which typically present themselves to have 512 or 4096 byte "LBA" size (though internally they are never really 512). So that 512 or 4096 is what OS expects user data to be aligned to. 512 bytes is still the majority in consumer-grade SSDs (even when physical page size is 16 kiB 🤷‍♂️).

Even violation of the alignment requirement can still fall back to buffered I/O so even that is, in general, not observable.

I guess it is hypothetically possible, but I only ever got EINVAL out of it.

[saethlin]

So... Under exactly what conditions should Miri report EINVAL? It seems like Linux isn't even sure how this works.

[nazar-pc]

I'd say that anything that isn't 512-byte aligned should immediately return EINVAL. If you want to be more modern then bump it to 4096. But not returning EINVAL for unaligned reads/writes I'd consider at least "unexpected" or "suspicious".

[RalfJung]

Okay, thanks. We'd probably accept a PR implementing something like this (on Linux; we'd need similar research for the other platforms), but it's not a priority from my side.

You're not going to get acceptable performance in Miri anyway however. ;)
So just disabling the flag when running in Miri seems reasonable to me.