-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BugFix][Core] Fix BlockManagerV2 when Encoder Input is None #9103
Conversation
Pull from head
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
@comaniac / @heheda12345 PTAL when you get a chance |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
LGTM too. |
Head branch was pushed to by a user without write access
This PR addresses an issue encountered with certain encoder-decoder models, e.g. Llama-3.2-11B-Vision-Instruct, where encoderInput is set to None. This situation arises in the following test cases:
tests/models/encoder_decoder/vision_language/test_mllama.py::test_models[5-128-bfloat16-sizes0-meta-llama/Llama-3.2-11B-Vision-Instruct]
tests/models/encoder_decoder/vision_language/test_mllama.py::test_models[5-128-bfloat16-sizes4-meta-llama/Llama-3.2-11B-Vision-Instruct]
Currently, these tests function correctly with BlockManagerV1 because, when encoder_input is None during the allocation of cross_block_table, an empty BlockTable is created without any blocks. However, this behavior does not hold for BlockManagerV2, where the addition of a BlockTable and attempts to allocate blocks for an empty or None encoder_input lead to assertion failures.
We make the following changes in this PR:
FIX #9099
FIX #9084