Skip to content
This repository has been archived by the owner on Sep 6, 2024. It is now read-only.
/ movenet Public archive

MoveNet: Body Segmentation for TFJS and NodeJS

License

Notifications You must be signed in to change notification settings

vladmandic/movenet

Repository files navigation

MoveNet: Body Segmentation for TFJS and NodeJS

Models are included in /model-* folders
Models descriptors have been additionally modified for readability

Supports:

  • MoveNet-Lightning v3 and v4
  • MoveNet-Thunder v3 and v4
  • MoveNet-Multipose v1

Performance:

  • Lightning v3: 5.450ms
  • Lightning v4: 5.481ms
  • Thunder v3: 10.388ms
  • Thunder v4: 10.508ms
  • Multipose: 11.217ms

Implementation

Actual model parsing implementation in movenet.js and movenet-multipose.js does not follow original
and is implemented using native TFJS ops and optimized for JavaScript execution

Function processResults() takes model.execute output and returns array of 17 keypoints:

  • id
  • score: score as number
  • label: annotated body part as string
  • xRaw: x coordinate normalized to 0..1
  • yRaw: y coordinate normalized to 0..1
  • x: x coordinate normalized to input image size
  • y: y coordinate normalized to input image size




Example

Example Image

Test

node movenet.js inputs/model1.jpg
2021-05-29 16:32:48 INFO:  movenet version 0.0.1
2021-05-29 16:32:48 INFO:  User: vlado Platform: linux Arch: x64 Node: v16.2.0
2021-05-29 16:32:49 INFO:  Loaded model { modelPath: 'file://model-lightning/movenet-lightning.json' } tensors: 178 bytes: 9299876
2021-05-29 16:32:49 INFO:  Model Signature {
  inputs: { 'input:0': { name: 'input:0', dtype: 'DT_INT32', tensorShape: { dim: [ { size: '1' }, { size: '192' }, { size: '192' }, { size: '3' }, [length]: 4 ] } } },
  outputs: { 'Identity:0': { name: 'Identity:0', dtype: 'DT_FLOAT', tensorShape: { dim: [ { size: '1' }, { size: '1' }, { size: '17' }, { size: '3' }, [length]: 4 ] } } }
}
2021-05-29 16:32:49 INFO:  Loaded image: inputs/model1.jpg inputShape: [ 938, 638, 3, [length]: 3 ] modelShape: [ 1, 192, 192, 3, [length]: 4 ] decoded size: 1795332
This model execution did not contain any nodes with control flow or dynamic output shapes. You can use model.execute() instead.
2021-05-29 16:32:49 INFO:  Inference time: 145 ms
2021-05-29 16:32:49 INFO:  Tensor output [ 1, 1, 17, 3, [length]: 4 ]
2021-05-29 16:32:49 INFO:  Processing time: 4 ms
2021-05-29 16:32:49 DATA:  Results: [
  { id: 0, label: 'nose', score: 0.851518988609314, xRaw: 0.133351668715477, yRaw: 0.4853442311286926, x: 309, y: 125 },
  { id: 1, label: 'leftEye', score: 0.7733256816864014, xRaw: 0.11329909414052963, yRaw: 0.501908540725708, x: 320, y: 106 },
  { id: 2, label: 'rightEye', score: 0.7870607376098633, xRaw: 0.11650494486093521, yRaw: 0.46030735969543457, x: 293, y: 109 },
  { id: 3, label: 'leftEar', score: 0.5259113311767578, xRaw: 0.11611292511224747, yRaw: 0.5221166610717773, x: 333, y: 108 },
  { id: 4, label: 'rightEar', score: 0.7544838786125183, xRaw: 0.12976181507110596, yRaw: 0.42634811997413635, x: 272, y: 121 },
  { id: 5, label: 'leftShoulder', score: 0.5883469581604004, xRaw: 0.1994987577199936, yRaw: 0.6012834906578064, x: 383, y: 187 },
  { id: 6, label: 'rightShoulder', score: 0.8768258094787598, xRaw: 0.23354817926883698, yRaw: 0.3692959249019623, x: 235, y: 219 },
  { id: 7, label: 'leftElbow', score: 0.7374638319015503, xRaw: 0.3235592842102051, yRaw: 0.6565064191818237, x: 418, y: 303 },
  { id: 8, label: 'rightElbow', score: 0.8474915623664856, xRaw: 0.3795775771141052, yRaw: 0.32592085003852844, x: 207, y: 356 },
  { id: 9, label: 'leftWrist', score: 0.6389445662498474, xRaw: 0.46755996346473694, yRaw: 0.6036725640296936, x: 385, y: 438 },
  { id: 10, label: 'rightWrist', score: 0.6672286987304688, xRaw: 0.4508911073207855, yRaw: 0.3890339136123657, x: 248, y: 422 },
  { id: 11, label: 'leftHip', score: 0.6918626427650452, xRaw: 0.49237537384033203, yRaw: 0.5632658004760742, x: 359, y: 461 },
  { id: 12, label: 'rightHip', score: 0.824874758720398, xRaw: 0.48929092288017273, yRaw: 0.42534327507019043, x: 271, y: 458 },
  { id: 13, label: 'leftKnee', score: 0.46882709860801697, xRaw: 0.6760866045951843, yRaw: 0.5007705688476562, x: 319, y: 634 },
  { id: 14, label: 'rightKnee', score: 0.5292187333106995, xRaw: 0.6698553562164307, yRaw: 0.5213510990142822, x: 332, y: 628 },
  { id: 15, label: 'leftAnkle', score: 0.4727800190448761, xRaw: 0.8808274865150452, yRaw: 0.44615471363067627, x: 284, y: 826 },
  { id: 16, label: 'rightAnkle', score: 0.23934555053710938, xRaw: 0.8793723583221436, yRaw: 0.5921049118041992, x: 377, y: 824 },
  [length]: 17
]
2021-05-29 16:32:49 STATE: Created output image: outputs/model1.jpg size: [ 638, 938, [length]: 2 ]




Credits