Should RoPE be included in embeddings?

[lockwo]

In the docs RoPE is only shown in the attention tab, however, since it is an embedding method (granted, probably almost entirely used with attention based methods) and is in _embedding.py I was wondering if it should at least be cross listed in the docs.

[patrick-kidger]

I'd be happy to have a cross-link if think it's worth it! :)

(Whilst we're here, I'm also conscious that our current rope implementation is kind of annoying to use, with the need for a helper function between it and the attention layer. Not sure if we could have done something better there.)

[lockwo]

I'd be happy to have a cross-link if think it's worth it! :)

sure I will open one

(Whilst we're here, I'm also conscious that our current rope implementation is kind of annoying to use, with the need for a helper function between it and the attention layer. Not sure if we could have done something better there.)

I would certainly have strong opinions here, but I haven't ever really used RoPE yet (so I will save those opinions for another issue in the future).