Use Rotary Embeddings in SRU++

[bratao]

@taolei87 I'm way too excited with SRUpp. I already adopted in all my projects with impressive results.

Recently the great people at Eleuther.ai found a silver-bullet in transformers. The Rotary Embeddings.

The RoPE is a relative position encoding method with promise theoretical properties. The main idea is to multiply the context embeddings (q,k in the Transformer) by rotation matrices depending on the absolute position. One can prove that the inner product of the context embeddings will become only depending on the relative position.

It gave a performance increase in virtually all tasks tested.
There is more information here:
https://blog.eleuther.ai/rotary-embeddings/

From the little of I understood of SRU++ builds in self-attention, it would be possible to use in SRUppAttention to q, k. Do you think that worth the test?

[taoleicn]

Thank you @bratao. Let me take a look and get back to you.

[bratao]

Thank you @taoleicn . I did a very possibly wrong implementation here
bratao@1c614c3
However, I'm running some tests and apparently it is converging faster

[taoleicn]

Thanks for sharing @bratao. I read the paper and it is indeed very interesting.
Do you have any preliminary results? If you see consistent or significant improvement using RoE, we should find a way to incorporate it as an optional module.

[bratao]

@taoleicn I only tested on toy examples and them converged faster. Unfortunately, I'm too overwhelmed with work to test on something like enwik8

Did you had the opportunity to check if I placed on the right place on my implementation? I have a big impression that I placed on the wrong place.