MKL: add comment about APIs with multiple USM inputs, all USM pointers should be of same type

[spencerpatty]

@mkrainiuk @petercad Does it make sense to add to our USM section a comment about APIs with multiple USM inputs needing to have all of them created with the same malloc_XXX api ?

[petercad]

Hey @spencerpatty -- Actually, I think we should allow mixed USM pointer types in the same call. On the implementation side, it shouldn't be too hard to support, right? For some BLAS APIs, in fact, it will be best to have some USM pointers reside on the host and some on the device, for optimal performance.

[spencerpatty]

@petercad Can you be more specific about what use cases this might be important for? I am thinking about sparse domain where we may have three arrays for each matrix rowPtr, colInd and values which should all exist in the same places... Although now that I am thinking about it, possibly the rowPtr could be only on host...

[petercad]

@spencerpatty The example I'm thinking of the BLAS group batch APIs, which take a number of arrays stored in pointers. In our current implementation some of these arrays are only accessed on the host, while others are only accessed on the device. So the configuration that provides best performance would have the first type as host USM allocations and the second as device USM allocations -- other configurations might require copying data.

[spencerpatty]

I see, it makes sense. Do you think we should explicitly mention that this disparate approach to USM allocations should be fully supported, maybe as a note to developers?

[zettai-reido]

@spencerpatty
May be, it would be better to state

• it is recommended to support disparate pointers in full
• specification for specific method or function should state current level of support individually, if it differs from recommended.
  Don't you think?

[spencerpatty]

Yes this is a very diplomatic way to say it :)

[spencerpatty]

@sknepper @dbollapr @mmeterel @aelizaro @zettai-reido Do you want to contribute to this discussion about USM pointers and whether they need to be all the same type or not... Right now we don't say anything about it in the spec, so in theory each USM pointer should individually be handled whether it is device, host or shared.

[zettai-reido]

I support Peter here.
VM was designed to work on every pointer and wrap/copy individually.

say

vm::fma(A, B, C, Y) -- they may be all different types of pointers advised by USM (say even host-accessible pointer from other context, which will be recognized just as "heap" pointer")

[spencerpatty]

would the host-accessible pointer from another context return as usm::alloc::unknown in the get_pointer_type(void *x, context ctxt) api ? since presumably the context being passed here is different than the context the pointer x was created with ?

[spencerpatty]

In this case, do you memcpy the data to the device yourself as needed ?

[zettai-reido]

Yes, this what happens.
It returns usm::unknown to the pointers not allocated to context.

[zettai-reido]

And NULL pointers are universal.
So no need to copy.
And one may be careful to see, if this operation is in-place or that output needs copy back if it was temporary.