REISA is a data analytics model for in-situ and in-transit workflows based Ray, a Python framework providing a distributed execution model.
REISA exposes a distributed task-based paradigm for online analytics coupled to MPI numerical simulations at scale. The coupling and communication between Ray and the simmulation is managed by PDI, a portable data interface that offers a declarative API to decouple application codes from I/O. Therefore, just adding some simple PDI command to the numerical simulation and writing a simple analysis kernel we can get an online distributed analysis at scale for the data stream generated by the simulation.
Authors: Xico Fernández Lozano, Bruno Raffin, Julien Bigot and Emilio J. Padrón
To launch the program ensure you have loaded pycall and mpi plugins from PDI (v1.6.0) and dependencies, you have installed ray['complete'] and ray['debug'] (v2.4.0), the python module netifaces and run the following script:
./Launcher.sh
This command will compile simulation.c with PDI and MPI, then a batch job will start Ray in every necessary node and run the previous simulation.
The simulation will use the simulation.yml in order set up Ray in each MPI process.
The user just needs to modify client.py file to manage the analytics tasks.
The execution will generate a timeline-client.json file that you can read with Google Chrome in chrome://tracing to see the execution stream.