mfs bias parallel writes policy - mfspar?

[WikiBox]

I have two multibay USB DAS. I sometimes see very impressive bandwidths, close to the max 10Gbps of USB 3.1 gen2, by reading and writing to several drives in parallel.

Using mfs and writing multiple files with the drives well balanced will also sometimes be extremely fast, since a lot of the writes will be in parallel and also maximize use of the internal drive caches.

But if the pool is not well balanced, all writes may be to one drive, being much slower than more parallel writes. Parallel writes to many drives in the pool can be several times faster then writes to just one single drive.

I suggest a distributed create policy with mfs bias that allows more parallel writing to multiple drives in a pool.

As usual, when writing a file, the file is written to the drive with most free space. But if the next file is written immediately after, then the next most free drive is used. Perhaps using a timeout or detect if the drive is busy. The idea is to first write to the drive with the most free storage, but if that drive is busy or recently written to, write to another drive instead. This would lead to multiple writes being spread out to more/all drives that could then work very efficiently in parallel.

When all drives with free storage are active, writes are done round robin to all drives, if possible. Or possibly in new order based on mfs again. Not sure if it is possible to direct larger writes to drives with more free storage, in order to increase the mfs bias?

Since the batches of writes always start on the mfs drives, over time it is likely that the storage will balance. But it is possible that you will need to balance more.

It might even be possible to most of the time use mfs just as normal and detect bulk file transfers, a certain number of files/bytes queued, and only then "turn on" the parallel distribution of writes.

[trapexit]

The topic has come up before and I'm not opposed to something like this in theory but in practice it is far more complicated than existing policy.

To be pedantic... the policies are for "create" not "write" so it can only spread that out so much and one has to keep in mind that you have no idea how large a file will be when created so you can't really use size in any equation.

You could keep track of number of open files but open files don't really tell you that much. You'd have to actively keep track of how much reading and writing is being done. But that couldn't account for out of band traffic or in the future the passthrough mode. mergerfs doesn't currently keep global track of what is open where or how much data is passing through. Naturally this can be done but it is nice not to keep track of that. What I'm getting at is "busy" is not trivial. Especially when every device can handle different amount of throughput, have different cache sizes, etc.

I'm not convinced adding all that will materially improve performance compared to already existing policies like "rand" and "pfrd" and it is non-trivial amount of work to implement the thing to test it in a real world scenario. Not that I can't look into it at some point but I have other priorities.