Skip to content

Commit

Permalink
v0.1.7 - Hip rotation using the hip center as the pivot point.
Browse files Browse the repository at this point in the history
  • Loading branch information
jellydreams committed Sep 29, 2024
1 parent fce8bb7 commit e0d9972
Show file tree
Hide file tree
Showing 4 changed files with 42 additions and 27 deletions.
38 changes: 25 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,26 @@
# VTS FULLBODY TRACKING
_Version ALPHA 0.1.6_

🔗 website: [vts-fullbody-tracking.gitlab.io](https://vts-fullbody-tracking.gitlab.io/)

[discord-link]: https://discord.gg/9K9gejWQ3s

[![Last Release][github-release-badge]](https://github.com/jellydreams/VTS-Fullbody-Tracking/releases)
[![VTS FULLBODY TRACKING Discord][discord-badge]](https://discord.gg/9K9gejWQ3s)
[![Twitter Follow][twitter-badge]](https://twitter.com/_JellyDreams_)

[discord-badge]: https://img.shields.io/badge/Join_Discord-indigo?logo=discord&logoColor=white&color=7289da
[twitter-badge]: https://img.shields.io/twitter/follow/_JellyDreams_.svg?style=social
[github-release-badge]: https://img.shields.io/github/v/release/jellydreams/VTS-Fullbody-Tracking?label=ALPHA%20release

This plugin integrates full body tracking functionality using Mediapipe.
It allows users to use tracked body parameters as inputs to control your Live2D model in [VTube Studio](https://denchisoft.com/).
It allows users to use tracked body parameters as inputs to control Live2D model in [VTube Studio](https://denchisoft.com/).

![Demo Tracking Arms](readme_img/Demo_Tracking_Arms.png)

## THIS PLUGIN IS UNDER DEVELOPMENT
This plugin may contain bugs and lack certain features.<br>
Visit the [Wiki](https://github.com/jellydreams/VTS-Fullbody-Tracking/wiki) for more information on how the plugin works <br>
Join [Discord Server](https://discord.gg/9K9gejWQ3s) for discuss about the plugin.
Visit the [Wiki](https://github.com/jellydreams/VTS-Fullbody-Tracking/wiki) or [Website](https://vts-fullbody-tracking.gitlab.io/) for more information on how the plugin works. <br>
Join [Discord Server](https://discord.gg/9K9gejWQ3s) to share tests and feedbacks, ask questions or get help.

### How you can Help
- **Live2D Rigger**: Help understand how to effectively rig models for the body parts feature
Expand All @@ -25,12 +36,11 @@ Join [Discord Server](https://discord.gg/9K9gejWQ3s) for discuss about the plugi
**Requirements**: Window, VTube Studio, Camera

1. Download the executable from the [releases page](https://github.com/jellydreams/VTS-Fullbody-Tracking/releases)
2. Connect a Camera and Open VTube Studio
4. Double-click on the executable file `VTS_Fullbody_Tracking.exe` to launch the plugin
5. A settings window will appear. Select your camera and click on the 'Start Tracking' button
6. Allow the plugin in vtube Studio
7. Window displaying a preview of pose tracking will appear.
8. In your model's parameter settings, you can now choose body parts X, Y, Z coordinates, and visibility as inputs
2. **Connect a Camera** and **Open VTube Studio.**
4. **Double-click on the executable file** `VTS_Fullbody_Tracking.exe` to launch the plugin. A settings window will appear.
5. **Select your camera** and **click on the 'Start Tracking' button**
6. **Allow the plugin in vtube Studio**. Window displaying a preview of pose tracking will appear.
7. **Configure your model's parameter settings**, using plugin parameters as inputs. you can now choose body parts X, Y, Z coordinates, and visibility as inputs

📖 Wiki Section - [Run the plugin](https://github.com/jellydreams/VTS-Fullbody-Tracking/wiki/Run-the-plugin)

Expand All @@ -46,7 +56,7 @@ Displays the image captured by the camera
📖Wiki Section - [Settings Window](https://github.com/jellydreams/VTS-Fullbody-Tracking/wiki/Settings-Window)

## Custom Parameters
TThis plugin add new controls for various body parts in Vtube Studio. \
This plugin add new controls for various body parts in Vtube Studio. \
Each body part has parameters for controlling its position and visibility.

![List Bodyparts MediaPipe](readme_img/list_bodyparts.png)<br/>
Expand All @@ -61,11 +71,13 @@ Each body part has parameters for controlling its position and visibility.

## DEVELOPMENT

Instructions for developers who want to run the plugin from the source code.

### Requirements

- Python 3.11

Install dependencies
#### Install dependencies

```shell
pip install -r requirements.txt
Expand All @@ -86,7 +98,7 @@ python app.py
### Build executable

```shell
pyinstaller ./app.py -n VTS_Fullbody_Tracking-0.1.6 --add-data='models/*:models' --add-data='icon.png:.' -F -w
pyinstaller ./app.py -n VTS_Fullbody_Tracking-0.1.7 --add-data='models/*:models' --add-data='icon.png:.' -F -w
```

## Documentation
Expand Down
12 changes: 6 additions & 6 deletions app.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def set_result(result: PoseLandmarkerResult, output_image: mp.Image, timestamp_m
await vts.connect()

# Authenticate with VTube Studio API
print('authentification VtubeStudio..')
#print('authentification VtubeStudio..')
await vts.request_authenticate_token() # get token
await vts.request_authenticate() # use token

Expand All @@ -63,7 +63,7 @@ def set_result(result: PoseLandmarkerResult, output_image: mp.Image, timestamp_m

# ---- VTUBE STUDIO: INITIATE CUSTOM PARAMETERS ------

print('Create parameters in VTube Studio')
#print('Create parameters in VTube Studio')

# Prepare parameter names for each body part
parameter_names = get_parameters_names()
Expand All @@ -82,7 +82,7 @@ def set_result(result: PoseLandmarkerResult, output_image: mp.Image, timestamp_m

while running:

print('========== START LIVE TRACKING =========')
#print('========== START LIVE TRACKING =========')

cap = cv2.VideoCapture(settings['camera_id'])

Expand Down Expand Up @@ -122,7 +122,7 @@ def set_result(result: PoseLandmarkerResult, output_image: mp.Image, timestamp_m
values = list(data.values())
names = list(data.keys())

print('Update parameters in Vtube Studio')
# print('Update parameters in Vtube Studio')
# -- Update parameters in VTube Studio
send_request_parameter = vts.vts_request.requestSetMultiParameterValue(names, values, weight=1, face_found=True, mode='set')
await vts.request(send_request_parameter)
Expand Down Expand Up @@ -155,7 +155,7 @@ def render_image(image, preview_enabled=False):

if __name__ == '__main__':

print('=== VTS FULLBODY TRACKING ===')
#print('=== VTS FULLBODY TRACKING ===')

# --- OPEN USER WINDOW : CONFIGURATION TRACKING
root, settings = window_tracking_configuration()
Expand All @@ -167,5 +167,5 @@ def render_image(image, preview_enabled=False):
# ========= STOP PLUGIN ==========

root.destroy()
print('=== VTS FULLBODY TRACKING STOPPED ===')
#print('=== VTS FULLBODY TRACKING STOPPED ===')

2 changes: 1 addition & 1 deletion info.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
VERSION = 'v0.1.6'
VERSION = 'v0.1.7'
ICON_PATH = "icon.png"
17 changes: 10 additions & 7 deletions utils_mediapipe.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,14 +156,11 @@ def get_bodyparts_values(parameters):
# Find center for this part of the body
bodypart_center_name = BodyCenters[bodypart.name]

# Skip right hip as left hip calculates rotation for both
if bodypart.name != 'RIGHT_HIP':
# Skip hips as orignal values are used
if bodypart.name not in ['RIGHT_HIP', 'LEFT_HIP']:
bodypart_center = parameters_world[bodypart_center_name.value.value]
bodypart_values = parameters_world[bodypart.value]
bodypart_name = bodypart.name
# rename Hips Left to Hips Rotation
if bodypart_name == 'LEFT_HIP':
bodypart_name = 'HIPS_ROTATION'

# Calculate values from new center
data = calcul_data(bodypart_values, bodypart_center, bodypart_name)
Expand All @@ -176,6 +173,7 @@ def get_bodyparts_values(parameters):
values = calcul_body_position(values, parameters_img)
values = calcul_hips_position(values, parameters_img)
values = calcul_clavicles_position(values, parameters_world)
values = calcul_hips_rotation(values, parameters_world)

return values

Expand Down Expand Up @@ -229,12 +227,17 @@ def calcul_hips_position(values, parameters_img):
values['HIPS_POSITION_Y'] = (values['BODY_Y'] - body_center_y) * 10
values['HIPS_POSITION_Z'] = (values['BODY_Z'] - body_center_z) * 10

values['HIPS_ROTATION_Y'] *= 10

values['HIPS_POSITION_VISIBILITY'] = 1.0
return values


def calcul_hips_rotation(values, parameters_world):
values['HIPS_ROTATION_X'] = parameters_world[BodyParts.LEFT_HIP.value].x * 100
values['HIPS_ROTATION_Y'] = parameters_world[BodyParts.LEFT_HIP.value].y * 100
values['HIPS_ROTATION_Z'] = parameters_world[BodyParts.LEFT_HIP.value].z * 100
return values


def calcul_clavicles_position(values, parameters_world):
# Determine clavicle position using the hip at the center
values['CLAVICLES_X'] = parameters_world[BodyParts.RIGHT_SHOULDER.value].x * 10
Expand Down

0 comments on commit e0d9972

Please sign in to comment.