Migrate file format to protobuf #8995
Conversation
|
Web viewer built successfully. If applicable, you should also test it:
Note: This comment is updated whenever you push a commit. |
jprochazk
added
the
🪵 Log & send APIs
jprochazk
force-pushed
the
jan/migrate-file-protobuf
branch
from
b1ebf6e to
cb6f4f8
Compare
jprochazk
force-pushed
the
jan/migrate-file-protobuf
branch
from
cb6f4f8 to
ea90dd8
Compare
| // NOTE: This is `size_of` on a `repr(Rust)` struct, | ||
| // which is fine because we control the layout | ||
| // in the definition above, and tests would quickly | ||
| // catch any sort of regression. |
There was a problem hiding this comment.
Why are we not switching to the C ABI if we need control over layout though?
There was a problem hiding this comment.
I thought about this a bit more, and maybe this should just be a literal constant. We need control over the encoding, not the layout of the struct.
Alternatively we can control the layout of the struct, and use something like bytemuck to cast to/from a slice of bytes. But that's an extra dependency that may not be worth it for something so simple
| // NOTE: We use little-endian encoding, because we live in | ||
| // the 21st century. |
There was a problem hiding this comment.
Reading this just after reviewing an arrow PR that encodes stuff as BE to be spec compliant lol
There was a problem hiding this comment.
This part is our format, and it doesn't go over the network so nobody can claim it should be network-endian! 😄
...well it technically does if you download it over http. But we must resist the CPU cycles wasted on swapping bytes.
Related
rrdto protobuf #8993MsgPackserializer #8994What
protobufencoding for our file format.msgpackcode from the repository.Tested by opening running the
clockexample and attempting to load the data in both the native and web viewers.There are still some pieces left in the file format that have now been made redundant. One example is the per-recording
compressionfield. We don't use that anymore, compression is now specified perArrowMsg. There are a few other things like that, but I believe that refactoring is better left to the time when we decide to get rid ofLogMsg, which would be another breaking change for the file format.