Question: Obtaining Bones and Joints

[Kaszanas]

I would like to ask if it is possible to get the joints and bones as presented in the dataset that was introduced on the project page:

I think the related issue is #86

[geopavlakos]

For the HInt dataset, we only annotate the 2D keypoints of the hand.
If you want to get the 3D bone orientation or the 3D joints of the hand, you could run the HaMeR network on these images. This will not be ground truth, but it will give you a reasonable estimate for these parameters.

[Kaszanas]

For the HInt dataset, we only annotate the 2D keypoints of the hand. If you want to get the 3D bone orientation or the 3D joints of the hand, you could run the HaMeR network on these images. This will not be ground truth, but it will give you a reasonable estimate for these parameters.

So if I understand correctly, during the inference these 3D keypoints are inferred to later fit the 3D mesh of the hand?
And to get the joint positions I am supposed to look at the:

You can get the 3D coordinates of the hand keypoints in the camera frame by adding pred_cam_t_full (which is calculated here) to out['pred_keypoints_3d'] (which is calculated here). Then you can export the sum in a pkl file.

Is there any order to the inferred 3D keypoints that the network generates?

I understand that this is not ground truth but ANN output.

[geopavlakos]

Actually, the model infers the hand model parameters (pose and shape parameters from MANO). Given these parameters, we can use MANO to reconstruct the mesh and the 3D keypoints.

The description in the above issue is accurate for inferring the locations of the 3D keypoints. The keypoints are in the OpenPose order. And yes, HaMeR will only give a neural network estimate - this is not ground truth.

[Kaszanas]

Actually, the model infers the hand model parameters (pose and shape parameters from MANO). Given these parameters, we can use MANO to reconstruct the mesh and the 3D keypoints.

The description in the above issue is accurate for inferring the locations of the 3D keypoints. The keypoints are in the OpenPose order. And yes, HaMeR will only give a neural network estimate - this is not ground truth.

Would you be open to merging if I modify the demo.py and add a separate boolean flag argument?

[Kaszanas]

@geopavlakos I am working on this in #89 but I am hitting some issues, I will create a devcontainer and run this with a debugger. Hopefully I can come up with a solution that works for me locally ;)

[longshot-pting]

@geopavlakos I am working on this in #89 but I am hitting some issues, I will create a devcontainer and run this with a debugger. Hopefully I can come up with a solution that works for me locally ;)

hi,have you gotten the img with visual joint like you first mentioned? I have tried pred_3d and pred_2d, it can not correspond location with ori img

[Kaszanas]

See if you can get it working with #89

[longshot-pting]

See if you can get it working with #89

I can get 3d keypoints and 2d keypoints, but I can not get right img, like this:

[Kaszanas]

See if you can get it working with #89

I can get 3d keypoints and 2d keypoints, but I can not get right img, like this:
Uploading WechatIMG47.jpg…

I think the image is broken.

Anyway if the output from: #89 does not work, we will have to wait for @geopavlakos to answer what is the correct way to get these keypoints.

[longshot-pting]

See if you can get it working with #89
I can get 3d keypoints and 2d keypoints, but I can not get right img, like this:
Uploading WechatIMG47.jpg…

I think the image is broken.

Anyway if the output from: #89 does not work, we will have to wait for @geopavlakos to answer what is the correct way to get these keypoints.

I have get right img with keypoints, In a class named SkeletonRender, we can get pred_keypoints_3d_proj_img, when I addweight it with ori image, It fits. But I still don't know why the pred_keypoints_2d doesn't work, maybe question is the perspective_projection function. please tell me your idea if you have solution, thanks!

[JuanCarlosHR2003]

@longshot-pting Can you post how you got the pred_keypoints_3d_proj_img, please? I am trying to get the Skeleton Renderer to work, but to no avail.

[longshot-pting]

@longshot-pting Can you post how you got the pred_keypoints_3d_proj_img, please? I am trying to get the Skeleton Renderer to work, but to no avail.

of course! I checked it's function named skeleton_renderer and get five pictures with skeleton, in which the picture named ''pred_keypoints_3d_proj_img'' is right, and I use it's focal and cam_center in hamer forward_step like this:
pred_keypoints_2d = perspective_projection(pred_keypoints_3d, translation=pred_cam_t, rotation=rotation.repeat(batch_size, 1, 1).to(device), focal_length = torch.tensor([self.cfg.EXTRA.FOCAL_LENGTH, self.cfg.EXTRA.FOCAL_LENGTH]).reshape(1, 2).repeat(batch_size, 1), camera_center = (torch.tensor([self.cfg.MODEL.IMAGE_SIZE, self.cfg.MODEL.IMAGE_SIZE], dtype=torch.float).reshape(1, 2) / 2.).repeat(batch_size, 1) )
Hope it helpful for you.