Implement multi-threading to fully utilize computing resources

[k4yt3x]

This ticket tracks the implementation of multi-threading.

Right now only the decoder and encoder are multi-threaded. The processors (Real-ESRGAN, RIFE, etc.) can also be multi-threaded to better utilize the available computing power and VRAM. This requires a major redesign of the processing pipeline. The structure will look something like:

flowchart LR
    A(Decoder Thread) -->|Decoded AVFrames| Q1(Queue)
    Q1 -->|Work stealing| T1(Processor Thread 1)
    Q1 -->|Work stealing| T2(Processor Thread 2)
    Q1 -->|Work stealing| T3(Processor Thread 3)
    T1 -->|Processed AVFrames| Q2(Queue)
    T2 -->|Processed AVFrames| Q2
    T3 -->|Processed AVFrames| Q2
    Q2 --> E(Encoder Thread)

Loading

[Pete4K]

That would be totally great. My Processors are all cold and don't do anything. Only my GPU is working. Btw.: Thanks for the upload. I will test it.

[Pete4K]

Would it be an idea to combine TensorRT and NCNN for efficient inference across many GPUs for still better speed, too? I don't know if TensorRT works with this.

[Pete4K]

It seems that TensorRT could possibly make Real ESRGAN x4 Plus faster: https://github.com/yuvraj108c/ComfyUI-Upscaler-Tensorrt

[k4yt3x]

My Processors are all cold and don't do anything. Only my GPU is working.

I don't think I'll do multi-GPU support just yet. The workload will still be on on GPU.

Would it be an idea to combine TensorRT and NCNN for efficient inference across many GPUs for still better speed, too?

TensorRT only works on NVIDIA GPUs. If we need to support it then we'll need to support multiple backends simultaneously and dynamically select which one to use during runtme. We'll also need to include multiple versions of models. I don't think that's ideal. This better belongs under #1231.

[Pete4K]

Sorry, I don't mean GPU-Multi Support. I meant only implementing multi-threading would be a great Idea.

[Pete4K]

Ok, when the Models are supported ist the best thing