Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] use cutlass for 24 #33

Draft
wants to merge 53 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
5d51361
Add cutlass 2:4 infrastructure
Faraz9877 Oct 22, 2024
17f5b96
Update with test code
Faraz9877 Oct 28, 2024
471a03c
Clean up a bit; both fp8 and int8 working
Faraz9877 Oct 30, 2024
0b332fb
Add fp16 and bf16 support to sparse cutlass mm
Faraz9877 Oct 30, 2024
da31648
semi_structured for fp16 and bf16 and int8
ilmarkov Oct 1, 2024
e655f94
Fix A100 int8 tests
ilmarkov Oct 2, 2024
5fc3c1c
Add fp8 cusparseLt
ilmarkov Oct 9, 2024
9cf36d6
wip
ilmarkov Oct 9, 2024
ad09e79
Fix signatures
ilmarkov Oct 9, 2024
e75eabc
Fix compilation and tests
ilmarkov Oct 13, 2024
0306390
Update for older platforms
ilmarkov Oct 15, 2024
1021acb
Add benchmarks
ilmarkov Oct 16, 2024
19ce358
Fix typo
ilmarkov Oct 23, 2024
959408c
Added scaled_mm for fp8.
ilmarkov Oct 24, 2024
117b87b
Add docstrings
ilmarkov Oct 28, 2024
2c7e68e
Update for torch 2.5
ilmarkov Oct 30, 2024
922f4f8
Add handling contiguous dense input for int8 and fp8
ilmarkov Oct 30, 2024
beca038
Add fp8 cusparseLt
ilmarkov Oct 9, 2024
5d9cd25
Fix compilation and tests
ilmarkov Oct 13, 2024
39ad9d4
Add caching of cusparseLT meta
ilmarkov Oct 23, 2024
520eb62
Cached cusparseLt
ilmarkov Oct 25, 2024
20956e6
Fix destroy function
ilmarkov Oct 25, 2024
87c8088
Prepare for reproduce
ilmarkov Oct 25, 2024
4ea58b1
Fix cusparseLt caching
ilmarkov Oct 30, 2024
f0551ef
Make cached version default function
ilmarkov Nov 5, 2024
d7476e8
Fixes and polishing after rebase
ilmarkov Nov 6, 2024
681ea5e
add sparse 2:4 weight loading suport
dsikka Oct 23, 2024
ecf878f
Some more changes!
rahul-tuli Oct 29, 2024
80952dc
Cleanup
rahul-tuli Oct 31, 2024
8462c9d
get uncompressed to work; update gemm to use contiguous; use alex's u…
dsikka Nov 1, 2024
0a3e506
patch
dsikka Nov 4, 2024
2e28972
use our decompressor
dsikka Nov 4, 2024
28f0abb
Some more work
rahul-tuli Nov 6, 2024
c7a97a8
Use new scaled_T function
rahul-tuli Nov 7, 2024
ccadad0
Add multiprocessing for kernel sweep benchmarking
Faraz9877 Nov 8, 2024
807737c
Add multi-GPU
Faraz9877 Nov 8, 2024
04c19a5
Add cutlass_scaled_sparse_mm op
Faraz9877 Nov 14, 2024
2a85c5a
Clean up
Faraz9877 Nov 14, 2024
1b381c9
Update code
Faraz9877 Nov 14, 2024
4e31076
Update code
Faraz9877 Nov 14, 2024
13fccf4
Clean up the benchmarking
Faraz9877 Nov 14, 2024
b345cc8
Clean up the cutlass benchmarking
Faraz9877 Nov 14, 2024
2d03e1d
Fix cmake errors
Faraz9877 Nov 14, 2024
e9439cc
Fix the cmake TAG
Faraz9877 Nov 14, 2024
4ba7c0f
Merge branch 'buildable' into rahul-quant-merged-rs
robertgshaw2-redhat Nov 15, 2024
f74ef37
update
robertgshaw2-redhat Nov 15, 2024
f5bc9eb
fixed
robertgshaw2-redhat Nov 15, 2024
1316076
updated
robertgshaw2-redhat Nov 15, 2024
4d2b12c
updated
robertgshaw2-redhat Nov 15, 2024
fe30b53
updated, calling things properly
robertgshaw2-redhat Nov 15, 2024
4c61b19
running end to end but not passing
robertgshaw2-redhat Nov 15, 2024
86716f8
updated
robertgshaw2-redhat Nov 15, 2024
c796ac8
Some cleanup
rahul-tuli Nov 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
updated
  • Loading branch information
robertgshaw2-redhat committed Nov 15, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 13160765c0d5c0924a6bf5c4543f442d344cbc42
18 changes: 0 additions & 18 deletions csrc/torch_bindings.cpp
Original file line number Diff line number Diff line change
@@ -323,24 +323,6 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
"int pad_slot_id) -> ()");
ops.impl("causal_conv1d_fwd", torch::kCUDA, &causal_conv1d_fwd);

ops.def("cslt_compress_fp8_semi_structured(Tensor! input) -> Tensor");
ops.impl("cslt_compress_fp8_semi_structured", torch::kCUDA,
&cslt_compress_fp8_semi_structured);

ops.def(
"cslt_mm_semi_structured(Tensor! compressed_A, Tensor! denseB,"
"float!? scale, Tensor!? bias) -> Tensor");
ops.impl("cslt_mm_semi_structured", torch::kCUDA, &cslt_mm_semi_structured);

ops.def(
"cslt_mm_fp8_semi_structured2(Tensor! compressed_A, Tensor! denseB,"
"float!? scale, Tensor!? bias) -> Tensor");
ops.impl("cslt_mm_fp8_semi_structured2", torch::kCUDA,
&cslt_mm_fp8_semi_structured2);

ops.def("cslt_clear_cache() -> ()");
ops.impl("cslt_clear_cache", &cslt_clear_cache);

#endif

// Quantized GEMM for GPTQ.
Original file line number Diff line number Diff line change
@@ -113,7 +113,6 @@ def apply_weights(self,
bias=bias
)


# if not self.quantized:
# return semi_structured_dense_sparse_T_gemm(
# a_dense=x,