Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

issue with masks_to_marks mapping #18

Closed
1 of 2 tasks
dokooh opened this issue Dec 4, 2023 · 10 comments
Closed
1 of 2 tasks

issue with masks_to_marks mapping #18

dokooh opened this issue Dec 4, 2023 · 10 comments
Assignees
@SkalskiP
Labels
bug Something isn't working

Comments

@dokooh
Copy link

dokooh commented Dec 4, 2023

Search before asking

  • I have searched the Multimodal Maestro issues and found no similar bug report.

Bug

Hi,

First and foremost thanks for your nice work so far.

I was testing your code with your google collab tutorial, and the mark creation (SAM), visualization and refining goes smoothly. Also the prompt call with marks to gpt4 goes well without any issue and I get response back.

In the part that you try to extract and visualize relevant marks, the resultset of masks_to_marks throws the error shown below.

With the example I used I expect a large output (20-30 marks), if this helps.

Environment

0.1.0rc1
Google collab (T4 vm)

Minimal Reproducible Example

masks = maestro.extract_relevant_masks(text=response, detections=refined_marks)
masks = np.array([mask for mask in masks.values()])
detections = maestro.masks_to_marks(masks)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-61-a9e5dd9e84f7>](https://localhost:8080/#) in <cell line: 3>()
      1 masks = maestro.extract_relevant_masks(text=response, detections=marked_image)
      2 masks = np.array([mask for mask in masks.values()])
----> 3 detections = maestro.masks_to_marks(masks)

3 frames
[/usr/local/lib/python3.10/dist-packages/supervision/detection/core.py](https://localhost:8080/#) in _validate_mask(mask, n)
     27     )
     28     if not is_valid:
---> 29         raise ValueError("mask must be 3d np.ndarray with (n, H, W) shape")
     30 
     31 

ValueError: mask must be 3d np.ndarray with (n, H, W) shape

Additional

No response

Are you willing to submit a PR?

  • Yes I'd like to help by submitting a PR!
@dokooh dokooh added the bug Something isn't working label Dec 4, 2023
@SkalskiP
Copy link
Collaborator

SkalskiP commented Dec 4, 2023

Hi @dokooh! 👋🏻 So you used our official Colab with no code changes? You only uploaded your own image?

@SkalskiP SkalskiP self-assigned this Dec 4, 2023
@dokooh
Copy link
Author

dokooh commented Dec 4, 2023

Hi @dokooh! 👋🏻 So you used our official Colab with no code changes? You only uploaded your own image?

Correct.

@SkalskiP
Copy link
Collaborator

SkalskiP commented Dec 4, 2023

Okay, I just ran the code to confirm it works on our example. It works. Now, let's Try to debug yours.

  • Could you share your version of Google Colab? The one with your example image?
  • Alternatively, add print statement here:
masks = maestro.extract_relevant_masks(text=response, detections=refined_marks)
masks = np.array([mask for mask in masks.values()])
>>> print(masks)
detections = maestro.masks_to_marks(masks)

@dokooh
Copy link
Author

dokooh commented Dec 4, 2023 •
edited
Loading

Okay, I just ran the code to confirm it works on our example. It works. Now, let's Try to debug yours.

  • Could you share your version of Google Colab? The one with your example image?
  • Alternatively, add print statement here:
masks = maestro.extract_relevant_masks(text=response, detections=refined_marks)
masks = np.array([mask for mask in masks.values()])
>>> print(masks)
detections = maestro.masks_to_marks(masks)
Detections(xyxy=array([[448, 374, 504, 430],
       [339, 267, 364, 292],
       [195,   5, 224,  34],
       [424,  88, 453, 117],
       [496, 145, 521, 170],
       [249, 159, 273, 184],
       [ 31, 194,  60, 224],
       [392,  57, 421,  87],
       [245, 268, 274, 297],
       [539,  84, 564, 109],
       [506, 405, 591, 430],
       [507,  90, 531, 114],
       [453,  17, 499,  58],
       [379,   0, 404,  31],
       [  0, 141, 122, 319],
       [189,  84, 214, 110],
       [354, 173, 378, 199],
       [274, 117, 320, 157],
       [355,  69, 380, 114],
       [274, 212, 320, 252],
       [189, 179, 214, 221],
       [450, 111, 498, 130],
       [189, 179, 214, 204],
       [466,  75, 481,  90],
       [355,  69, 380,  94],
       [280,  45, 309,  74],
       [353, 173, 379, 209],
       [457,  56, 490,  67],
       [  0,   0, 194,  63],
       [530,   0, 555,  11],
       [287,  88, 302, 104],
       [274, 212, 320, 281],
       [416,  33, 426,  42],
       [581,  33, 591, 118],
       [  4, 196,  11, 223],
       [  0, 323,  50, 394],
       [584, 363, 591, 376],
       [ 42, 371,  50, 394],
       [  0, 322,  72, 394],
       [562,   0, 571,  24],
       [280,  45, 309, 116],
       [189,  84, 214, 138],
       [331, 229, 367, 240],
       [562,   0, 582,  24],
       [228, 109, 259, 117],
       [ 64, 103,  93, 118],
       [317, 275, 326, 293],
       [  0, 194,  60, 224],
       [409, 323, 502, 332],
       [238, 227, 249, 237],
       [244,  29, 368,  38],
       [287, 275, 302, 290],
       [516,  58, 564,  66],
       [500,   0, 559,  28],
       [332, 134, 368, 145],
       [278, 251, 311, 262],
       [558, 114, 567, 128],
       [  0,  91,  12, 115],
       [406, 134, 497, 142],
       [397, 221, 489, 229],
       [  0,   6,  91,  14],
       [229, 216, 255, 223],
       [452,   0, 500,   8],
       [504,  32, 540,  43],
       [340, 254, 433, 263],
       [320, 161, 329, 180],
       [517,  47, 561,  55],
       [  0, 139, 323, 319],
       [121,  70, 123, 119],
       [229, 121, 255, 128],
       [409, 308, 502, 319],
       [278, 156, 311, 167],
       [ 44, 348,  58, 357],
       [504,  33, 511,  42],
       [408, 307, 503, 332],
       [228, 206, 258, 213],
       [  0,  50,  66,  60],
       [340, 149, 433, 157],
       [259, 225, 268, 239],
       [  0,  90,  16, 135],
       [244, 388, 252, 403],
       [228, 109, 260, 129],
       [513,  33, 540,  43],
       [544, 173, 557, 182],
       [238, 132, 249, 142],
       [574,   0, 582,  24],
       [ 75, 343,  89, 357],
       [176, 245, 219, 253],
       [404, 254, 432, 262],
       [329, 105, 355, 113],
       [228, 205, 258, 223],
       [370,   0, 438,  42],
       [ 51, 124, 144, 134],
       [515,  46, 564,  66],
       [259, 130, 268, 145],
       [317, 267, 364, 293],
       [176, 245, 267, 253],
       [ 82, 323,  83, 329],
       [436,  47, 459,  85],
       [ 81, 323,  83, 329],
       [519,  59, 531,  66],
       [400, 101, 414, 110],
       [340, 254, 400, 262],
       [404,  13, 434,  30],
       [340, 149, 398, 157],
       [462,  94, 489, 108],
       [448, 374, 591, 430],
       [  0,   6,  61,  14],
       [193, 230, 198, 235],
       [244,  29, 319,  37],
       [192, 204, 197, 225],
       [244,  30, 254,  37],
       [239, 130, 268, 145],
       [192, 110, 197, 130],
       [450,  75, 498, 129]]), mask=array([[[False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        ...,
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False]],

       [[False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        ...,
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False]],

       [[False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        ...,
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False]],

       ...,

       [[False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        ...,
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False]],

       [[False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        ...,
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False]],

       [[False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        ...,
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False]]]), confidence=None, class_id=None, tracker_id=None)

@SkalskiP
Copy link
Collaborator

SkalskiP commented Dec 4, 2023 •
edited
Loading

@dokooh, I can't open it.

@dokooh
Copy link
Author

dokooh commented Dec 4, 2023 •
edited by SkalskiP
Loading

@dokooh, I can't open it.

added your email, let me know if you still can't open so I download the ipynb file and share it.

@SkalskiP
Copy link
Collaborator

SkalskiP commented Dec 4, 2023

I need your image as well. Could you upload it here?

@dokooh
Copy link
Author

dokooh commented Dec 4, 2023 •
edited
Loading

I need your image as well. Could you upload it here?

@SkalskiP
Copy link
Collaborator

SkalskiP commented Dec 4, 2023

@dokooh looks like after refinement, you lost all makes. I will add a fix for that edge case. As a workaround try skipping maestro.refine_marks step.

@SkalskiP
Copy link
Collaborator

The maestro project has pivoted in the direction of a VLM fine-tuning toolkit. As a result, I am closing legacy issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SkalskiP SkalskiP

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dokooh @SkalskiP