Filter OOB points when training #2060
Conversation
WalkthroughThis pull request introduces several modifications across multiple files in the SLEAP neural network data processing and training modules. The changes primarily focus on enhancing image processing, resizing, and visualization capabilities. Key updates include adding new parameters for image resizing in inference and training classes, improving instance handling in data providers, and refining visualization methods for different model trainers. The modifications aim to provide more flexibility in how input images are processed and how model performance is monitored during training. Changes
Sequence DiagramsequenceDiagram
participant DataProvider
participant Pipeline
participant Resizer
participant Transformer
participant Trainer
DataProvider->>Pipeline: Provide data
Pipeline->>Resizer: Apply image resizing
Resizer-->>Pipeline: Resized image
Pipeline->>Transformer: Process instances
Transformer-->>Pipeline: Processed data
Pipeline->>Trainer: Feed training data
Trainer->>Trainer: Visualize and train
Poem
Tip CodeRabbit's docstrings feature is now available as part of our Early Access Program! Simply use the command Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
docs/conf.py(1 hunks)sleap/nn/data/pipelines.py(2 hunks)sleap/nn/data/providers.py(2 hunks)sleap/nn/inference.py(15 hunks)sleap/nn/training.py(2 hunks)
✅ Files skipped from review due to trivial changes (1)
- docs/conf.py
🔇 Additional comments (12)
sleap/nn/training.py (2)
1322-1322: Set resize_input_image=False in FindInstancePeaks initialization
Setting resize_input_image=False ensures that the input images are not resized during peak finding, which maintains the original image scale and is appropriate for this context.
1763-1763: Set resize_input_image=False in FindInstancePeaks initialization
Setting resize_input_image=False ensures that the input images are not resized during peak finding, aligning with the expected behavior in this trainer.
sleap/nn/inference.py (7)
734-738: Add input_scale parameter to CentroidCropGroundTruth
The introduction of the input_scale parameter allows for resizing images and centroids during centroid cropping, providing flexibility to handle different input scales.
769-771: Scale images and centroids based on input_scale
By resizing images and scaling centroids according to input_scale, consistency between image sizes and centroid coordinates is maintained.
940-940: Introduce resize_img parameter in InferenceLayer.preprocess
Adding the resize_img parameter allows control over image resizing during preprocessing, enhancing flexibility in handling resized inputs.
Also applies to: 961-961
1968-1970: Add resize_input_image parameter to FindInstancePeaks
Including the resize_input_image parameter provides control over whether crops should be resized during peak finding, enhancing flexibility in image processing.
Also applies to: 2016-2016
2114-2114: Use resize_input_image in FindInstancePeaks.call method
Passing resize_img=self.resize_input_image to the preprocess method ensures consistent resizing behavior based on the initialization parameter.
3901-3904: Add resize_input_image parameter to TopDownMultiClassFindPeaks
Adding the resize_input_image parameter allows for control over image resizing during peak finding in multi-class top-down models, improving flexibility.
Also applies to: 3917-3917
4035-4035: Pass resize_input_image to preprocess in TopDownMultiClassFindPeaks.call
Using resize_img=self.resize_input_image in the preprocess call ensures that the resizing behavior aligns with the specified parameter.
sleap/nn/data/providers.py (1)
4-4: Import Instance class from sleap.instance
The import of Instance is necessary for creating new instances after filtering invalid points.
Also applies to: 9-9
sleap/nn/data/pipelines.py (2)
778-778: Add Resizer to visualization pipeline in TopdownConfmapsPipeline
Including the Resizer transformer ensures that images are appropriately resized during visualization, matching the preprocessing steps used during training.
1254-1254: Add Resizer to visualization pipeline in TopDownMultiClassPipeline
Adding the Resizer transformer ensures consistent image resizing during visualization, aligning with the training preprocessing steps.
| height, width = raw_image_size | ||
|
|
||
| instances = [] | ||
| for instance in lf.instances: | ||
| pts = instance.numpy() | ||
| # negative coords | ||
| pts[pts < 0] = np.NaN | ||
|
|
||
| # coordinates outside img frame | ||
| pts[:, 0][pts[:, 0] > height - 1] = np.NaN | ||
| pts[:, 1][pts[:, 1] > width - 1] = np.NaN | ||
|
|
||
| # remove all nans | ||
| pts = pts[~np.isnan(pts).any(axis=1), :] | ||
|
|
||
| instances.append(Instance.from_numpy(pts, lf.skeleton, lf.track)) | ||
| lf.instances = instances |
There was a problem hiding this comment.
Ensure instances have valid points before adding to the dataset
After filtering out invalid coordinates, instances may have no valid points remaining. Adding such instances could lead to issues downstream. Please check if pts contains at least one valid point before creating the Instance.
Apply this diff to verify that instances have valid points:
for instance in lf.instances:
pts = instance.numpy()
# negative coords
pts[pts < 0] = np.NaN
# coordinates outside img frame
pts[:, 0][pts[:, 0] > height - 1] = np.NaN
pts[:, 1][pts[:, 1] > width - 1] = np.NaN
# remove all nans
pts = pts[~np.isnan(pts).any(axis=1), :]
+ if len(pts) > 0:
instances.append(Instance.from_numpy(pts, lf.skeleton, lf.track))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| height, width = raw_image_size | |
| instances = [] | |
| for instance in lf.instances: | |
| pts = instance.numpy() | |
| # negative coords | |
| pts[pts < 0] = np.NaN | |
| # coordinates outside img frame | |
| pts[:, 0][pts[:, 0] > height - 1] = np.NaN | |
| pts[:, 1][pts[:, 1] > width - 1] = np.NaN | |
| # remove all nans | |
| pts = pts[~np.isnan(pts).any(axis=1), :] | |
| instances.append(Instance.from_numpy(pts, lf.skeleton, lf.track)) | |
| lf.instances = instances | |
| height, width = raw_image_size | |
| instances = [] | |
| for instance in lf.instances: | |
| pts = instance.numpy() | |
| # negative coords | |
| pts[pts < 0] = np.NaN | |
| # coordinates outside img frame | |
| pts[:, 0][pts[:, 0] > height - 1] = np.NaN | |
| pts[:, 1][pts[:, 1] > width - 1] = np.NaN | |
| # remove all nans | |
| pts = pts[~np.isnan(pts).any(axis=1), :] | |
| if len(pts) > 0: | |
| instances.append(Instance.from_numpy(pts, lf.skeleton, lf.track)) | |
| lf.instances = instances |
Description
This PR addresses #1901, to remove OOB/ negative points while creating the dataset for training.
Types of changes
Does this address any currently open issues?
#1901
Outside contributors checklist
Thank you for contributing to SLEAP!
❤️
Summary by CodeRabbit
New Features
Bug Fixes
Documentation