Add memory-mapped support for Kinetics-skeleton data converter

[JVPC0D3R]

Description

This pull request introduces an updated data converter script for Kinetics-skeleton data. By default, the script continues to use the original in-memory approach (gendata). However, users can now enable memory-mapped numpy arrays (open_memmap) by passing the --use_mmap flag. This enhancement significantly reduces peak RAM usage for large datasets by processing data in configurable chunks (--chunk_size).

Key Changes

1. New Function: gendata_mmap

   • Uses open_memmap to write data in chunks directly to disk.
   • Lowers memory usage by avoiding loading the entire dataset into memory.

2. Command-line Flag: --use_mmap

   • Toggles between gendata (in-memory) and gendata_mmap (memory-mapped) approaches.

3. Chunk Size: --chunk_size

   • Configurable size of data chunks to process at a time when using open_memmap.
   • Defaults to 128.

Testing

• Verified that both modes (--use_mmap and without) produce equivalent .npy and .pkl results.
• Tested a training loop with the spatio_temporal_gcn_learner.py script and the ST-GCN model.

[passalis]

Hi, thank you for your contribution. Seems fine to me. My only concern would be if it is possible to add a short documentation on this utility (e.g., on the README.md of the corresponding tools).

[JVPC0D3R]

Hi, I've already added the documentation and explained the changes 🤠

[passalis]

Thank you.

[passalis]

It looks like there are some test failures related to PEP8 formatting issues.

https://github.com/opendr-eu/opendr/actions/runs/13030640596/job/36348929365?pr=516

Edit: Ignore the licence failures, I am working to fix them.

[passalis]

Thank you! Seems fine.