1D chare array within a 3D chare array

[sshudler]

Hi all,

I have a question related to running Charm++-based data-analysis and visualization (DAV) routines within the Lulesh code, which also uses Charm++. The domain is decomposed into a 3D chare array in the beginning of the execution. What I’m trying to do is to add some in situ DAV as part of the execution. My DAV dataflow also uses a chare array, a 1D chare array.

Basically, I’m in a situation where I need to create this 1D chare array from within one of the chares in the 3D chare array. To maintain decoupling I don’t want to create the 1D array for DAV in the main chare. Is it possible at all? If it is, how one can communicate the proxy for this 1D array to other chares of the 3D array? Because each chare of the 3D chare array will need to pass local data to the appropriate 1D chare.

Thanks in advance!

-Sergei

[epmikida]

Hi Sergei, it is possible to create a chare array from within another. Once created, you can simply broadcast the proxy of the 1D array to the chares of the 3D array. There is however one small caveat: currently arrays can only be created on PE 0. So if you make sure that the 3D chare array creates the 1D array from a chare on PE 0, then you can just do it the same way you'd do it as if you were creating the new array from the main chare.

However, if you want it to be more flexible/general, you can use the API for asynchronous array creation, which allows you to create an array from any PE. That API is described in the manual in section 2.3.4.2.7 (https://charm.readthedocs.io/en/latest/charm++/manual.html#more-chare-array-features). When using this, you would need to pass a callback to array creation that would get called once the array is created, and you could use a callback which broadcasts the array ID for the new array to the 3D chare array, for example.

If you have any further questions let me know.

[sshudler]

Thanks, Eric! This helped me understand this issue better.
I tried adapting the "SimpleArrayHello" example to try that. Basically, to broadcast the newly created proxy I need another entry function (SendProxy) and then call the original entry function (SayHi) once the proxy is initialized in other chares:

void CreateAnotherChareArr() 
{
  if (CkMyPe() == 0)   // How to make sure only one single chare does this?
  {
    mBelloProxy = CProxy_Bello::ckNew(nElements);  // mBelloProxy is a member of class Hello
    thisProxy.SendProxy(mBelloProxy);
  }
}

void SayHi(int hiNo)
{
  // Send message to the other chare array
  mBelloProxy[thisIndex].SayB(hiNo);
  // ...
}

void SendProxy(CProxy_Bello prxb)
{
  mBelloProxy = prxb;
  thisProxy[thisIndex].SayHi(thisIndex);
}

Is it possible to make it shorter? Create the new chares, call SendProxy, wait for the broadcast and then continue with the execution. Sort of doing it all within SayHi? Another question is whether I can add entry function in a separate .ci file - add the SendProxy function in a separate file?

[epmikida]

I can't really think of a good way to make it much shorter while keeping it general. Normally ways to avoid some of these steps would be to use a readonly, or have the proxy be sent in the constructor of the other array, but that would involve creating the new array inside of the main chare, which you want to avoid. The only other way to "shorten it" would be to combine the SendProxy entry method above with other application specific work, but that would depend on your application.

There is no way to add an entry method to an existing chare in a separate .ci file at the moment. All entry methods need to be declared inside of the chare declaration.

In terms of your code example, I would probably suggest using the asynchronous array creation API. As your comment points out, if (CkMyPe() == 0) does not guarantee that only a single chare will create the array. The way to guarantee that would be something like if (CkMyPe() == 0 && thisIndex == x) where x is some index that you are certain would be on PE 0. But this would require you to know specifics about how the array is mapped, and thats not a very flexible solution.

Instead my suggestion would be something like: if (thisIndex == 0) CProxy_Bello::ckNew(nElements, CkCallback(CkIndex_Hello::SendProxy(NULL), thisProxy));. This way, once the array was created, it would automatically broadcast the information to the SendProxy method. Your particular use case in the above example doesn't rely on having mBelloProxy set immediately, so the asynchronous solution should work quite well.