There seems to be a bug in HBMCtrl

[fotuodunwu]

I tested HBM2Stack() and found that some addresses were inaccessible, and the following error occurred:

src/mem/mem_ctrl.cc:633: panic: panic condition !mem_intr->getAddrRange().contains(pkt->getAddr()) occurred: Can't handle address range for packet ReadSharedReq [1e3dc0:1e3dff]

By debugging, I found that HBMCtrl is a single memory controller to the membus, and it seems that it does not correctly transmit the requested address to the corresponding internal pseudo channel, regardless of whether _interleave_addresses is used.

When using _interleave_addresses, the test code is as follows:

# Create address ranges
start = 0
end = 4*1024
mem_range = AddrRange(str(start) + 'MiB', str(end) + 'MiB')
system.mem_ranges.append(mem_range)

# Instantiate HBM2Stack and complete the correct configuration
 hbm_memory = HBM2Stack(size="4GiB")
 system.hbm_memory = hbm_memory
 system.hbm_memory._create_mem_interfaces_controller()
 system.hbm_memory.set_memory_range([system.mem_ranges[0]])
 system.hbm_memory._interleave_addresses()
 for ctrl in system.hbm_memory.mem_ctrl:
      ctrl.port = system.membus.mem_side_ports

When not using _interleave_addresses, the test code is as follows:

DRAM_START_ADDR = 0x0000_0000_0000_0000
CHANNELS = 16 

mem_ranges = []
mem_ctrls  = [] # DRAM <---> system.membus
for i in range((int)(CHANNELS/2)):
    # 
    mem_ctrl_i        = HBMCtrl()
    mem_ctrl_i.dram   = HBM_2000_4H_1x64(device_size="256MiB") # 0.25 GiB/device * 1 devices/rank * 1 ranks/channel = 0.25 GiB/channel
    mem_ctrl_i.dram_2 = HBM_2000_4H_1x64(device_size="256MiB") # 0.25 GiB/device * 1 devices/rank * 1 ranks/channel = 0.25 GiB/channel
    dram_addr_is      = DRAM_START_ADDR + i * (256*1024*1024) * 2
    # 
    mem_ctrl_i.dram.range   = AddrRange(dram_addr_is, size="256MiB")
    # 
    mem_ctrl_i.dram_2.range = AddrRange(dram_addr_is + 256*1024*1024, size="256MiB")
    # 
    mem_ctrl_i.port = system.membus.mem_side_ports
    #
    mem_ranges.append(AddrRange(dram_addr_is, size="512MiB"))
    mem_ctrls.append(mem_ctrl_i)
system.mem_mode   = "timing"  # Use timing accesses
system.mem_ranges = mem_ranges
system.mem_ctrls  = mem_ctrls

[powerjg]

I don't fully understand the bug report. Can you show a complete example of the HBM controller not working so we can reproduce?

I believe the second example is incorrect. The ranges for the two pseudo channels are assumed to be interleaved with bit 6. See https://github.com/gem5/gem5/blob/stable/src/mem/hbm_ctrl.cc#L229.

Maybe we can provide a fatal if this is misconfigured.

[fotuodunwu]

Thank you very much for your reply. Since the domain specific accelerator (DSA) sometimes need to fully utilize the bandwidth of HBM, I do not use the method of interleaving addresses and instead assign specific PEs to fixed channels. I initially thought that HBMCtrl, like a general MemCtrl, could function without interleaving address.

Thanks for your reminder, I modified the code (which might not be entirely rigorous) and successfully passed the tests. The updated code is as follows:

// TODO: make the interleaving bit across pseudo channels a parameter
// if (bits(pkt->getAddr(), 6) == 0) {
//     is_pc0 = true;
// } else {
//     is_pc0 = false;
// }
if (pc0Int && pc0Int->getAddrRange().contains(pkt->getAddr())) {
    is_pc0 = true;
} else if (pc1Int && pc1Int->getAddrRange().contains(pkt->getAddr())) {
    is_pc0 = false;
} else {
    panic("Can't handle address range for packet %s\n",
          pkt->print());
}

The problem I encountered with interleave addresses is because my test environment was incorrect. I used DMA to perform consecutive accesses to a range of addresses with single-byte granularity. I guess that this conflicted with the interleaving method, causing certain address ranges to be split across two channels, which led to errors. The occurred error is shown below:

src/mem/xbar.cc:368: fatal: Unable to find destination for [0x10003bc:0x10003c4] on system.membus

Once again, I sincerely appreciate your patient and helpful response.