Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add RoPE positional encoding #714

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Add RoPE positional encoding #714

wants to merge 12 commits into from

Commits on Jul 28, 2024

  1. Add RoPE - support on cmdline

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    242566b View commit details
    Browse the repository at this point in the history

  2. Add RoPE init kernel

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    a36bc62 View commit details
    Browse the repository at this point in the history

  3. Use float buffer for rope freqs; tested against python ref

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    c08540d View commit details
    Browse the repository at this point in the history

  4. Do not use WPE when RoPE enabled

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    61a0376 View commit details
    Browse the repository at this point in the history

  5. Add initial RoPE kernel

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    46babfe View commit details
    Browse the repository at this point in the history

  6. Reduce freqs table 2x

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    e90062c View commit details
    Browse the repository at this point in the history

  7. Use x128 loads for RoPE fwd kernel

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    8516330 View commit details
    Browse the repository at this point in the history

  8. Implement rope bwd kernel

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    0f27b28 View commit details
    Browse the repository at this point in the history

  9. Change default rope value

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    96222c6 View commit details
    Browse the repository at this point in the history

  10. Minor refactor + fix fwd enc bug

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    841e229 View commit details
    Browse the repository at this point in the history

  11. Remove wpe grad communication

    @gordicaleksa
    gordicaleksa committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    3fda17b View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. Bug fix: missing /2 in freq table in the kernel

    @gordicaleksa
    gordicaleksa committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    7e0c497 View commit details
    Browse the repository at this point in the history