RUST implementation?

[jeandet]

@HadrienG2 @pwnorbitals I'm considering re-implementing this lib in RUST. This is basically a structured binary files codec.
I feel like RUST could helps on some aspects like loading several variables in parallel and fore sure ensure 0 memory hazards as long as I can write this in "safe mode". What is not clear to me is if such piece of code can be written in RUST without unsafe sections? Are there any patterns to map structures on memory or to load such binary files?

[jeandet]

https://docs.rs/binrw/latest/binrw/
https://github.com/jam1garner/binrw/

[HadrienG2]

I've not done binary file parsing in Rust yet, but I've done some quite complex text parsing before (C++ entity names) with a crate/library that claims to support binary input too, and from my experience with text I would strongly recommend that crate.

The crate is nom. It works by using a parser combinator architecture: you design your parser as a tree of greedy parsing functions, where each function is tasked with a simple goal: assert if the input matches a certain pattern, if so return parsed output and unparsed input, otherwise signal error to caller. The routine can easily defer to lower-level parsing subroutines to do its job, and where the input grammar is ambiguous, it is easy to have a branching design where multiple parsers are tried successively and the first that matches is kept.

Anecdotically, it is also the most frequently downloaded general-purpose parsing crate, which shows that it's quite a popular choice and it shouldn't be too hard to find help on websites like users.rust-lang.org if you get stuck.

You may also want to skim through the crates.io search linked above as although overly common keywords like "parsing" produce a fair amount of noise, it's still possible to find gems that way in a reasonable amount of time. Recent downloads is my favorite metric personally as its meaning is transparent (unlike "relevance") and it's a fairly good indicator of popularity in other projects, which itself is not bad to keep in mind for the reason given above. Another field to watch is last update date, which may help notice abandoned projects (no update for 1 year may mean a project is very mature, but often it just means there is no maintainer anymore).

[HadrienG2]

Another popular choice if you want to serialize data as well, which I get the impression you do, is serde. This is a general-purpose serialization/deserialization framework that makes it very easy to convert Rust data structures to various popular text and binary representations and back. You just write #[derive(Serialize, Deserialize)] on your structs' declarations and boom, you can easily export values to formats like CBOR, MessagePack, BSON... and import them back.

[HadrienG2]

serde does not export general-purpose parsing utilities, however, so you will still need to write your own binary output/binary parser in order to add support for custom binary data formats, and that's where something like nom will shine.

[jeandet]

@HadrienG2 thank you for all this helpful information, CDF files are like tree of binary records with unspecified order (except for the two firsts records). This could make a streaming parser difficult to write, on the other hand, following pointers from record to records makes a lot of jumps inside the underlying file/memory repr. CDF format is really close FAT maybe reading crates like this one https://crates.io/crates/fat32 could give me some inspiration too.

[pwnorbitals]

For binary zerocopy encoding/decoding, you can definitely do it safely. I also suggest looking at the zerocopy crates even though it might be less fit for your use. Also this blog post series could be useful : https://manishearth.github.io/blog/2022/08/03/zero-copy-1-not-a-yoking-matter/

[CarrollMccullough8y]

The author is considering re-implementing a structured binary files codec library in Rust due to its potential benefits, such as parallel loading and memory safety. They are unsure whether it can be done without using unsafe sections and are seeking advice on patterns for mapping structures on memory and loading binary files.

[HadrienG2]

Well, the general pattern of reinterpreting bytes as non-byte data is type-unsafe by definition (the information that the pile of bytes you're reinterpreting as a struct S is indeed a struct S is not available at compile time, so the compiler cannot check that the conversion is sensible), and it can be memory unsafe too if you are not careful with things like padding bytes (reading/writing one of these is instant UB at the LLVM layer, so UB that applies whether you do it in Rust or in C/++).

But a subset of these bytes -> structured data conversions can be done safely, through a combination of imposing compile-time constraints on the target struct S (e.g. no padding bytes, no invalid value like null values for non-nullable pointers/refs) and performing runtime checks which assert that the non-allowed conversions do not happen (e.g. input alignment is right and invalid values, if allowed, are not present). This is done by unsafe code which checks the required safety preconditions at compile time if possible and at runtime if necessary.

So, the answer to the original question is, this cannot be done without using unsafe code at all, but if you stick to well-known patterns, it can be done by delegating the safety checks to someone else's unsafe code. See @pwnorbitals' comment for some tracks on the latter option.