Memory allocation in GPU plugin

Available memories in GPU plugin

GPU plugin supports 4 types of memory allocation as follows. Here "usm_" types refer to the allocation type using the Intel Unified Shared Memor (USM) extension for OpenCL. For more detailed explanation about the USM extension, refer to this page.

• cl_mem : Standard OpenCL cl_mem allocation
• usm_host : Allocated in host memory and accessible by all of them. Not migratable.
• usm_shared : Allocated in host and devices and accessible by all of them. The memories are automatically migrated on demand.
• usm_device : Allocated in device memory and accessible only by the device which owns the memory. Not migratable.

Memory allocation in GPU plugin

In GPU plugin, actual allocation for each allocation types is be done through engine::allocate_memory which calls the corresponding memory object wrapper for each allocation type: gpu_buffer, gpu_usm.

Also, the total allocated amount of memory for each allocation type are managed per engine, so that you can check the allocation history by setting environment variable OV_GPU_Verbose=1 for OpenVino built with ENABLE_DEBUG_CAPS=ON.

...
GPU_Debug: Allocate 58982400 bytes of usm_host allocation type (current=117969612; max=117969612)
GPU_Debug: Allocate 44621568 bytes of usm_device allocation type (current=44626380; max=44626380)
GPU_Debug: Allocate 44236800 bytes of usm_host allocation type (current=162206412; max=162206412)
GPU_Debug: Allocate 14873856 bytes of usm_device allocation type (current=59500236; max=59500236)
...

The major allocation done in GPU plugin can be categorized as follows:

• Constant memory allocation
• Output memory allocation
• Intermediate memory allocation

Memory reuse in GPU plugin