Help with guaranteeing CoW going from Erlang to Rust

[josevalim]

Hi lovely folks,

We have bindings for the tokenizers library.

At some point, we need to create an EncodeInput. You can see the EncodeInput accepts an InputSequence. The InputSequence finally accepts a CoW smart pointer.

Therefore, I want to go from a Term to Cow on this function: https://github.com/elixir-nx/tokenizers/blob/396141719722348603eaa3856cf8be190c71dfd2/native/ex_tokenizers/src/tokenizer.rs#L416

In the current code, we use let Ok(seq) = term.decode::<String>(). The first question is: does this allocate a new string or it simply points to the binary? If so, we are all good.

However, in case it does make a copy, how could we go from Term to Cow? I was hoping I could go to Binary and then to a slice, but my Rust skills are failing me. I also tried to use the new as_str() in Binary (but my Rust skills also failed me :D).

I would appreciate any help you could give, thank you :)

[hansihe]

The first question is: does this allocate a new string or it simply points to the binary?

This indeed allocates a new string, as the rust String type manages its own memory.

However, in case it does make a copy, how could we go from Term to Cow?

If we decode the term to a &str, that does not allocate and instead points to the binary allocated by the BEAM.

We can then use the &str to construct a Cow, which should not allocate.

The slightly tricky part comes from managing the lifetimes, we need to introduce an explicit lifetime annotation to carry the lifetime of the input term into the output type.

fn term_to_encode_input<'a, 'b>(term: &'a Term<'b>) -> Result<EncodeInput<'b>, ExTokenizersError> {
    if let Ok(seq) = term.decode::<&'b str>() {
        Ok(EncodeInput::Single(seq.into()))
    } else if let Ok((seq1, seq2)) = term.decode::<(&'b str, &'b str)>() {
        Ok(EncodeInput::Dual(seq1.into(), seq2.into()))
    } else {
        Err(ExTokenizersError::Other(String::from(
            "input must be either a string or a tuple",
        )))
    }
}

Here we end up with the EncodeInput having the same 'b lifetime as the input Term, which means the EncodeInput will have a lifetime bounded to the current NIF invocation.

[josevalim]

Thank you so much, this worked as is! Would you link to send a PR or is it ok if I copy and paste it there? :)

[filmor]

Just a tiny note: Decoding as &str will still run the UTF-8 validation, so it's not "just pointing to the binary".

What was the issue with binary's as_slice? decode() is effectively str::from_utf8(binary.as_slice()) and I don't think we have an as_str (though that might be a worthwhile addition).

[josevalim]

I saw as_str here, but it is under the DeRef section: https://docs.rs/rustler/latest/rustler/types/binary/struct.Binary.html#method.as_str

[josevalim]

I am a bit torn on the checking for UTF-8 bit. In Elixir we assume it is checked at the edge, should we forward this assumption to the Rust side?

[hansihe]

Unless tokenizers can also accept a raw binary as input, I don't think there is much we can do to elide the UTF-8 checks.

Rust requires validation of UTF-8 at string construction time, and algorithms that operate on can and do assume valid UTF-8. It is therefore not sound to create a str/String that does not contain valid UTF-8 data. Not following this requirement can definitely cause soundness issues.

https://doc.rust-lang.org/std/primitive.str.html#invariant

Invariant

Rust libraries may assume that string slices are always valid UTF-8.

Constructing a non-UTF-8 string slice is not immediate undefined behavior, but any function called on a string slice may assume that it is valid UTF-8, which means that a non-UTF-8 string slice can lead to undefined behavior down the road.

[josevalim]

I see. So if we want to expose it, perhaps we should have it called "unsafe_decode" or similar. That's because we are passing data from Explorer (where data is kept as Rust strings) into Tokenizers (also in Rust) but through Elixir.

[josevalim]

Just for completeness, I believe this is what the unsafe_encode version would look like:

      if let Ok(binary) = term.decode::<Binary>() {
        let slice: &'b [u8] = binary.as_slice();
        let string = unsafe { std::str::from_utf8_unchecked(slice) };
        Ok(EncodeInput::Single(string.into()))

Final question: given Cow is part of Rust, do you think it would make sense to implement the From trait for Binary that goes to Cow<'s, [&'s str]>? Or maybe even Term itself could decode into that? How do you draw the line which traits you implement for?

Thank you so much for the replies and the PR! ❤️