Deleting large data blob seems slow compared with put/get

[Caviarion]

Context: I'm thinking about the possibility of using vineyard as a centralized IPC service to transfer data blobs between multiple processes.

Our previous approach is socket, so I tried to do some benchmark as comparison:

import vineyard, time, humanize
import numpy as np

client = vineyard.connect('/var/run/vineyard.sock')

arr = np.ones(1920 * 1080 * 3, dtype=np.uint8)
t = time.time()
cnt = 0
while True:
    arr_id = client.put(arr)
    shared_arr = client.get(arr_id)
    client.delete(arr_id)
    cnt += 1
    if cnt == 100:
        print(f"Speed: {humanize.naturalsize(1920*1080*3*cnt/(time.time()-t))}/s")
        t = time.time()
        cnt = 0

The result is around 100MB/s on our machine.

Speed could reach > 1GB/s without delete, but ends with NotEnoughMemoryException soon.

Is this reasonable ? If so, are there any chances to improve the speed of delete?

[sighingnow]

Hi @Caviarion,

Thanks for letting us know that. In fact there's a bug in the implementation (there's uncessary sync with etcd). That is a unknown bug and has been fixed by pull request #207.

A new version of vineyard will be release tomorrow or Fri.

[sighingnow]

Hi @Caviarion,

We have just released v0.1.11 and have fixed the performance issue about client.delete.