Question runnning ONNX Runtime on FPGA

[JAEWOOKe]

Dear all,

Hi, I have a question about CopyInputsAcrossDevices @ utils.cc

I have added an Execution Provider that uses my Xilinx FPGA.

By using Alloc of my device, it allocates data to PL area with 256MB memory limit.

However, I have noticed that ONNX Runtime copies all the tensor to the device before Execution.

By doing so, it exceeds the memory limit, and the kernel panicked.

Is there any device for this situation?

[ytaous]

@skottmckay / @snnn - feedback?

[skottmckay]

Not quite following. CopyInputsAcrossDevices will do a copy if the data is not on the device the execution provider (EP) the node was assigned to said the data needs it to be on. If there's a new version of the data created that should mean the OrtDevice info for the input did not match the OrtDevice info for the node. It is however copying it to where the EP said it needed it.

Firstly you can control the OrtDevice info returned by the EP's allocator (device member inside the OrtMemoryInfo value returned by IExecutionProvider->GetAllocator(...)->Info()).

You can also control the data copy logic to/from a given OrtDevice via what is returned by IExecutionProvider::GetDataTransfer.

And finally if the input data is created on with a matching OrtDevice value there would be no copy required.