State Space Models 101

Resources for my presentation on State Space Models at the Singapore Machine Learning Meetup.

Presentation

Presentation available here

Learning Resources

The following is an incomplete list of resources to learn more about SSMs and Mamba:

• HazyResearch AI-Sys Building Blocks
• The Annotated S4
• Awesome State Space Models
• Albert Gu's thesis on SSMs
• Maarten Grootendorst's blog
• Youtube Videos:
  • Yannick Kilcher
  • Do we need Attention?
  • Albert Gu's presentations:
    • S4
    • S4 at CMU
    • S4 at MLSys
  • Umar Jamil
• GPU Programming:
  • Jeremy Howard: Getting started with CUDA
  • Jeremy Howard: Going Further with CUDA for Python Programmers
  • Parallel Prefix sum with CUDA
• Other models
  • SSM Based
    • DSS [Gupta, 2022]
    • GSS [Mehta et al., 2022]
    • S4D [Gu et al., 2022]
    • BiGS [Wang et al. 2022]
    • SGConv [Li et al., 2022]
  • RNN Based
    • QRNN [McCann et al., 2017]
    • LRU [Orvieto et al., 2023]
    • RWKV [Peng et al., 2023]
  • Unknown
    • Gemini 1.5

Implementations

• SSM Official Implementations
• Mamba: The Hard Way, Sasha Rush: Arguably the best resource to understand Mamba via code.
• mamba
• mamba-minimal

Contact

I'm always looking to connect with like-minded individuals and discuss and work on ideas. Do reach out via LinkedIn or via Email