Optimise performance of real-time face detection

[julizet]

Hi @vladmandic,

first of all: thank's for this massive work on humanJS. The developer experience was great and it was a joy to code with it since the documentation is clear and its usage very straight forward.

However, we are facing some FPS performance issues with our current face detection prototype. It is a web-app using TS/JS, a webcam and the humanJS to detect faces in real-time, analyse properties like e.g. age, emotion, gender, gaze direction and mouth opening in order to create a parameterized, artsy Dall-E interpretation of it's user.

As long as we use an M2 machine that comes with a a lot of GPU we can reach enough frames per second (FPS) so the user experience is smooth. However, if we use an Intel i3, an Intel i5 or even a gamin laptop with slightly lower GPU we face some performance issues and reach a maximum of 7 to 14 FPS which make the prototype feel very laggy.

We found this multithreading demo of you but this (seems) to only handle multiple models and not synchronous FPS.

Do you have any idea how we could improve the FPS of our application – based on the code?

Thanks in advance

[vladmandic]

you can adopt multithreading to run separate instance of human in each thread and use that as a thread pool.

[Jaszkowic]

Hi @vladmandic, I'm working on the same project as @julizet. During my multithreading investigation I stumbled across your live demo here: https://vladmandic.github.io/human/demo/index.html. In the Options menu, I enabled or disabled the Use web worker option, which in my understanding is enabling or disabling the multithreading approach you are referring to. I have tried both options on my machine (MacBook Air, M2 Chip, 2022) and the frame rate is dropping from ~18fps to ~11fps when I switch on the Use web worker option. I would like to understand why this is happening - under what circumstances should web workers not be used for optimization due to specific parameter combinations? Conversely, in what scenarios can web workers effectively enhance performance? Thanks for your help!

Log output when Use web worker is enabled -> FPS 11.6

video: hd pro webcam c920 (046d:082d) | facing: back | screen: 2560 x 1239 camera: 1920 x 1080 processing: 1920 x 1080
backend: webgpu | tensors: 1.485 in worker
performance: initBackend:2, loadModels:135, inputProcess:35, totalFrames:191, cachedFrames:189, cacheCheck:8, total:73, interpolate:0, draw:2ms FPS process:11.6 refresh:33.9

Log output when Use web worker is disabled -> FPS 18.3

video: hd pro webcam c920 (046d:082d) | facing: back | screen: 2560 x 1239 camera: 1920 x 1080 processing: 1920 x 1080
backend: webgpu | system: 70.955.268 bytes gpu: 0 bytes | tensors: 1.491
performance: initBackend:4, loadModels:173, inputProcess:12, totalFrames:270, cachedFrames:268, cacheCheck:10, total:63, interpolate:0, draw:2ms FPS process:18.3 refresh:34.7

[vladmandic]

it depends on two factors:

• which backend are you using?
  if using webgl, canvas context cannot be transferred to worker (thats a limitation of a standard, not library),
  so image needs to be read gpu->cpu before transfer and that causes latency
• which browser is used - how efficient it is to transfer image from main thread to worker thread.
  chrome has zero-copy transfer (change ownership of data) that human uses, but safari actually has to create a copy while transferring

in general, if you're looking for realtime performance, single thread is almost always faster.
what webworkers give you is fully responsive main thread, so if you have an app running, it will not stall while human is processing.

other use case is if you're performing batch processing - lets say 1000 images - you can start multiple threads and pass images to them for processing in a thread pool manner.

[Jaszkowic]

Thank you for your reply, that clarifies a lot!