Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GPU] Fix weightless caching with low precision models #28641

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

[GPU] Fix weightless caching with low precision models #28641

wants to merge 9 commits into from

Conversation

tkrupa-intel
Copy link
Contributor

@tkrupa-intel tkrupa-intel commented Jan 23, 2025 •
edited
Loading

Details:

  • The existing mechanism for recreating precision conversion in weightless caching was not sufficient to work with low precision models (found out in Phi-3-mini-4k-instruct-int4-ov model).
  • This change:
    • adds copying of weightless cache attr in transformations relevant only to low precision subgraphs (for weight, scale and zero point constants).
    • replaces recreation of dtype change in propagate_constants to recreation of any kind of reorder from there.

Verified with Phi-3-mini-4k-instruct-int4-ov model: cache reduced from 1.9 GB to 12 MB.

Tickets:

@tkrupa-intel tkrupa-intel requested review from a team as code owners January 23, 2025 12:48
@tkrupa-intel tkrupa-intel requested review from itikhono and removed request for a team January 23, 2025 12:48
@github-actions github-actions bot added category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations labels Jan 23, 2025
@tkrupa-intel tkrupa-intel changed the title [GPU] Fix weightless caching with low precision models [DO NOT MERGE][GPU] Fix weightless caching with low precision models Jan 23, 2025
@tkrupa-intel tkrupa-intel changed the title [DO NOT MERGE][GPU] Fix weightless caching with low precision models [DO NOT MERGE][WIP][GPU] Fix weightless caching with low precision models Jan 23, 2025
@tkrupa-intel tkrupa-intel changed the title [DO NOT MERGE][WIP][GPU] Fix weightless caching with low precision models [GPU] Fix weightless caching with low precision models Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@t-jankowski t-jankowski t-jankowski left review comments

@itikhono itikhono Awaiting requested review from itikhono itikhono is a code owner automatically assigned from openvinotoolkit/openvino-ie-transformations-maintainers

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tkrupa-intel @t-jankowski