Missing File::Info data.

[didactic-drunk]

Crystal::System::FileInfo is missing public accessors. I'd like to create a PR to expose some or all of the information below.

Most of the data is cross platform with some exceptions. The list is not exhaustive.

Name	Platforms	Notes
atime	POSIX, Windows
ctime	POSIX, Windows
birth_time	Dragonfly, FreeBSD, Linux, MacOS, Windows
ino	POSIX, Windows*
dev	POSIX, ?	Is the volume serial number on Windows equivalent?
nlink	POSIX, Windows
blocks	POSIX	Not technically required in the spec but implemented almost universally.
blksize	POSIX	Not technically required in the spec but implemented almost universally.
flags	Linux, *BSD, MacOS, ?

Suggested names:

• access_time or last_access_time.
• creation_time or birth_time.
• change_time or metadata_change_time.
• link_count.
• io_block_size.
• flags is used by something else. I don't know what to call it.

Currently I need ctime, birth_time, ino, dev, nlink, flags. I assume others may want the full stat structure when more non web applications are written.

[ysbaddaden]

What are your use cases?

[didactic-drunk]

Detecting hard links needs[ino, dev, nlink] and birth_time ideally. ctime as a fallback but I also need ctime when displaying file info.

Also getting or setting flags like APPEND.

[RX14]

We specifically didn't expose ino and dev because we have same_file? which compares ino and dev, but doesn't expose these platform-specific details.

Exposing the number of links would be fine - with a usecase.

Exposing more times is interesting: many filesystems don't support the birth time, and there's discrepancies between windows and linux on ctime and atime, iirc. Go supports only modification time for this same reason.

Stat flags are already exposed.

The blocks and block size are useless on modern-day filesystems.

[j8r]

There is also st_size missing, to get the disk size.

[ysbaddaden]

File::Info#same_file? is nice, but maybe hard to find? There is the higher level File.same?(a, b) but I had to dig the docs to find it, and the documentation is lacking details. Just "compares the device and inode on UNIX to detect hard links" could be a nice addition.

I recently had a use case for atime: a disk cache with an automated cleanup of files not accessed since N days —mounted with relatime on Linux for the 24h granularity.

ctime is useful to detect that metadata changed, for example the file owner or permissions changed, but there was no writes so mtime didn't change; a backup application could use it.

Note that Go has a non-portable direct access to the stat struct through FileInfo Sys(). See https://stackoverflow.com/a/55303743/199791 for example. It's a nice solution. It only supports high level portable data, but allows applications to have very specific use cases for hardly portable data (e.g. atime on a relatime mounted device on a Linux target) that can be legitimate, but too specific to bother with platform agnostic methods.

[j8r]

A platform specific API should exist for this kind of non-portable, low-level operations, like Rust std::os::unix, and Go unix package.
It may be kind of already present with src/crystal/system/unix and src/crystal/system/win32, but this files aren't meant to be used as-is.

[didactic-drunk]

Stat flags are already exposed.

No they aren't. Crystal split mode in to permissions and suid/sugid/sticky calling it flags.

I need what BSD systems refer to as flags or Linux as attributes (not to be confused with extended attributes). On BSD it's part of the stat structure in st_flags. On Linux it's available through one of the stat interfaces, but I don't remember how exactly.

https://en.wikipedia.org/wiki/Chattr

[didactic-drunk]

We specifically didn't expose ino and dev because we have same_file? which compares ino and dev, but doesn't expose these platform-specific details.

My laptop has 3 million files. How do I find the hard links? I can't keep 3 million files open. Comparing every file against every file is unacceptably slow so that won't work either. I also can't compare incrementally or between program runs by saving state. So no this doesn't work for my use cases at all. I've mentioned what I for porting one ruby program.

What is the solution? Either I make my own shard which monkey patches or duplicates the stat structure already available in src/crystal/system/unix and src/crystal/system/win32 or crystal exposes thing like atime/ctime/birth_time which may vary in their use between platforms.

mtime can vary. I've encountered NFS systems that return epoch mtime/ctime/atime for every file. Every reported time in the structure including mtime is not just platform specific but file system specific. Especially so when using network file systems.

I can handle the differences and expect to. On systems supporting birth time I use birth time (which includes Windows). Otherwise fall back to POSIX ctime and nlinks. Both solutions need dev and ino.

Additional use cases that I need dev and ino for:

• Detecting when crossing a mount point and running user defined triggers.
• Comparing [dev, ino] with [birth_time, ctime, nlink] and between program runs.

Additional use cases:

• Getting and setting flags similar to chflags/chattr.

Most of the flags could be handled by an enum. The ones I need are mostly portable like immutable and append. Linux version flag is an outlier with additional data. I have no need for it.

So how do I make this work in crystal considering when everything except birth_time is available in ruby and working for ~12 years?

[RX14]

Note that Go has a non-portable direct access to the stat struct through FileInfo Sys().

Yeah, this is the solution I prefer too - just expose a stat struct with a info.platform_specific.foo.

No they aren't. Crystal split mode in to permissions and suid/sugid/sticky calling it flags.

My bad. Looks like this needs statx though. Just like birth_time. I don't think we should bind statx, or use statx by default on linux. A statx binding should be a shard, since the difference is only visible on the platfom-specific members.

My laptop has 3 million files. How do I find the hard links?

Thanks for the usecase! I agree, indexing hardlinks is impossible with same_file?. We should expose the platform-specific members.

[didactic-drunk]

Yeah, this is the solution I prefer too - just expose a stat struct with a info.platform_specific.foo.

So make info.stat public and document it's platform specific? That would solve almost every use case.

[ysbaddaden]

What about File::Info#raw? It would return @stat on UNIX and @file_attributes on Windows.

[RX14]

I prefer platform_specific - but I'd rather ensure everyone agrees with this approach (:+1: / :-1: this comment) before bikeshedding on that.

[asterite]

No platform specific data, please. A Crystal program should compile and run exactly the same on all platforms. Or, said another way, the API should be exactly the same for all platforms. But it's fine if a method raises on one platform but works on another, given that it's documented to only work on certain platforms.

[didactic-drunk]

No platform specific data, please. A Crystal program should compile and run exactly the same on all platforms. Or, said another way, the API should be exactly the same for all platforms. But it's fine if a method raises on one platform but works on another, given that it's documented to only work on certain platforms.

Instead of handling it at compile time:

if stat.responds_to?(:birth_time)
  ...
else
  # ctime handler
end

I have to use exception handling at runtime:

begin
  stat.birth_time
rescue
  # ctime handler
end

@asterite How much overhead does that add when traversing file systems with > 10 million files? 100 million? They won't use spinning rust so seek times are not as much of a concern.

But it's not one exception handler. It's several. One for flags, another for acl's, another for resource forks, another for extended attributes plus anything I missed.

I'd much rather use feature checks than exception handling. To me the code is clearer. I know it's a platform feature check and only runs on specific platforms. With the exception handler am I handling an os error or the platform unsupported? It's even less clear when trying to understand someone else's code. What did they intend?

[asterite]

How much overhead does that add when traversing file systems with > 10 million files? 100 million?

Exception handling doesn't add overhead unless an exception is raised (as far as I know).

Additionally, if a method doesn't raise (the compiler knows which methods raise), and methods you want to use in an OS and they are available won't raise, then the compiler will skip the entire exception handler (or LLVM will do this). So zero overhead, really.

I'd much rather use feature checks than exception handling.

The problem is that if someone forgets to check for a feature flag in a library, nobody can use that library in some OS, even if the library never calls that code (for example if they call it conditionally at runtime). This was discussed in the past.