You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Arguably, attn_metadata is the most complicated part in the forward computation logic. And it becomes even more complicated when we consider:
continuous batching, where we batch data from different sequences together
heterogeneous models, where we can have different attention metadata for different layers (e.g. Gemma 2)
optimized torch.compile logic, where we want to hide the complexity of attention layer from the compiler
Therefore, I'm considering to hide the complexity of continuous batching through forward context. The idea is to have a global forward context, which can be set by the model runner during every forward pass. The forward context can be used to store the attention metadata, and the model can access the attention metadata through the forward context.
Proposed Change.
The changes are:
the model runner will set the forward context before running the model, and the forward context will be used to store the attention metadata and kvcache.
For the sake of generality, the forward context should contain a list of attention metadata and kvcache, where each element in the list corresponds to the attention metadata and kvcache for a layer. In the common case where all the layers share the same attention metadata, the model runner is responsible for duplicating the attention metadata.
all the files in vllm/model_executor/models will know nothing about attention metadata and kvcache. They will only know about the input tensors and the output tensors, as if they are just doing token-wise computation. Every attention layer will have a new self.layer_index attribute, which will be used to index the attention metadata and kvcache in the forward context.
all the attention implementation will be wrapped into a PyTorch custom op so that it is easy to compile. The custom op will only take input tensors, and retrieve the attention metadata and kvcache from the forward context. This way, the complexity of attention metadata and kvcache will be hidden from the compiler.
Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
The text was updated successfully, but these errors were encountered:
the model runner will set the forward context before running the model, and the forward context will be used to store the attention metadata and kvcache.
there's one alternative: the top-level model owns and sets the forward context. In the llama case, LlamaForCausalLM sets the forward context.
this approach works better for encoder-decoder models, or multi-modality models.
Motivation.
take a look at the current llama forward computation logic:
if we don't consider
attn_metadata
andkv_caches
, it can be simplified as:Arguably,
attn_metadata
is the most complicated part in the forward computation logic. And it becomes even more complicated when we consider:torch.compile
logic, where we want to hide the complexity of attention layer from the compilerTherefore, I'm considering to hide the complexity of continuous batching through forward context. The idea is to have a global forward context, which can be set by the model runner during every forward pass. The forward context can be used to store the attention metadata, and the model can access the attention metadata through the forward context.
Proposed Change.
The changes are:
vllm/model_executor/models
will know nothing about attention metadata and kvcache. They will only know about the input tensors and the output tensors, as if they are just doing token-wise computation. Every attention layer will have a newself.layer_index
attribute, which will be used to index the attention metadata and kvcache in the forward context.see #9029 and #9097 for initial steps.
Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...
The text was updated successfully, but these errors were encountered: