BH MMul Hang Parent Case

[cmaryanTT]

Using this to organize the different manifestations of the MMul hang in BH

MMul in isolation (main case): #16439
MMul in Conv2D, in Resnet: #18065
MMul in SDPA, in LLama: #16673 (working with semaphore LLK fix)

• SPDA MMul LLK investigation subcase 18094

[ncvetkovicTT]

On https://github.com/tenstorrent/tt-metal/tree/ncvetkovic/bh_hang instruciton gathering is partially disabled, please see the commit message to disable it completely. This fixes the hang for single core hang #16439, maybe with ResNet50, #18065, but doesn't help with multiple cores, including SDPA #16673 / #18094.

[cmaryanTT]

Most recent experiment is to disable instruction gathering and branch prediction. Some improvement, but some test cases still fail. Request @mywoodstock to test with Resnet and @ncvetkovicTT 's latest branch.

[ncvetkovicTT]

@mywoodstock branch soon to come, after I do the rebase again.

[ncvetkovicTT]

After I rebased SDPA started failing with PCC errors, but no hangs, so I gave up on it. Instead, I'll continue working on this branch ncvetkovic/bh_hang which is a bit apart from latest main.

@mywoodstock please introduce the changes from the branch tt-llk/ncvetkovic/bh_hang to whichever tt-metal branch has the ResNet50 test and tell us if it hangs.

[ncvetkovicTT]

@cmaryanTT @ttmtrajkovic @cglagovichTT @vmilicevicTT @mywoodstock @pgkeller Take a look at the following table for the overview of the workarounds that were attempted and their results. You can add suggestions and comments either somehow in the sheet or here/DM.

https://docs.google.com/spreadsheets/d/1i0bfPhjZcUI1ce_CTE44a-VMnvquQjY-kRPAb5oMXow/edit?gid=0#gid=0

[cmaryanTT]

Note issue #18250 - there may be a new MMul hang cause if you're using a recent build.

[ncvetkovicTT]

I had to rerun SDPA tests to confirm the behavior. Multi core variant doesn't hang when CSR writes are protected, single core variant behaves like single core test_benchmark.py and C++ repros. For now it seems that the multi core variant of test_benchmark.py hangs with all atempted workarounds. I cannot reproduce ResNet50 hang on yyzo-bh-15.

[ncvetkovicTT]

All issues except #16439 original variant can be solved with the fix #10 from the table. In order to understand why disabling cache at the start of trisc.cc doesn't help with multi core SDPA, I am trying to introduce the hang on minimal core grid possible for simulation to be reasonably long. My current findings show that reducing the core grid below 16 removes the hang in test_sdpa_decode_paged_attention.

I propose that workaround attempt no.10 becomes a workaround, or a solution for now, for all other issues. The issues are caused by a HW bug which requires that CSR writes be guarded by either nops, fences, or, even better, branch prediction disable/enable.

[cmaryanTT]

@ncvetkovicTT @cglagovichTT - have you confirmed if the hang is the same in SDPA after the no 10 workaround?

I agree about committing the workaround for now to allow other work to proceed.

[ncvetkovicTT]

@cmaryanTT Fix no10 works on SDPA, fixes 11 - 16 don't and I am trying to understand why.

[ncvetkovicTT]

@cmaryanTT @pgkeller @mywoodstock @nathan-TT @cglagovichTT Let me know if you guys need a separate branch for the fix. Essentially the workaround is to change disable_gathering in ckernel.h to be:

template <bool add_nops = true>
inline void disable_gathering() {
    // Disable gathering: set bit 18
    asm(R"ASM(
        .option push
        li   t1, 0x2
        csrrs zero, 0x7c0, t1
        li   t1, 0x1
        slli t1, t1, 18
        fence
        csrrs zero, 0x7c0, t1
        li   t1, 0x2
        csrrc zero, 0x7c0, t1
        fence
        .option pop
         )ASM"
    :::"t1");
     
     //Gathering is done early in the pipeline, so we need to make sure
     //the above csrrw gets processed before the load-replay instructions
     if constexpr (add_nops) {
         TTI_NOP;
         TTI_NOP;
         TTI_NOP;
    }
}

and to add the SFPI compiler flag -fno-rvtt-sfpu-replay.

[ncvetkovicTT]

@ejouretTT @vmilicevicTT @cmaryanTT @mywoodstock @pgkeller @cglagovichTT PR #18359 should resolve all issues except #16439 multi core case, which will be addressed as soon as this PR is merged. Please rebase to the latest main once this PR is merged, I need to see why EventSync is failing first, any input is welcome.

[ncvetkovicTT]

To clarify - this workaround should unblock everyone on all related issues from the description. I suggest to keep #16439 open until we understand why full-grid test fails. Also, keep #18094 open as a tracker until we realize the effect of clock gating + instruction cache on this test.

We can close this issue.