Blue screen error of Windows machine during SLEAP training

[skyshi0]

Hi, I’m encountering some issues of the SLEAP Usage of GPU. (1)I couldn't initialize a training with more frames labeled. (1209). I was able to train the model at the beginning, but later when I added more frames, there’s an error massage “An error occurred while training centroid. Your command line terminal may have more information "shown on the screen when I was training the multi-animal top down model and I couldn’t find the exact reason shown in the terminal. (2)My Windows computer will show this ‘Blue Screen Error’ if I try to initialize the training several times, and I observed that it may be related to the maximal usage of the GPU. I’ve decreased the batch size to 1 but it still happened. I installed SLEAP by using Mambaforge, and I attached my computer GPU/CUDA/CUDA Toolkit version here:
NVIDIA Driver version: 566.03
CUDA Version: 12.7
System installed CUDA Toolkit: v11.8(check nvcc –version in terminal)
SLEAP installed CUDA Toolkit: v11.3(nvcc –version in sleap environment)

Do you think the CUDA/CUDA Toolkit has any conflict in my computer, and do you have any suggestions to troubleshoot this problem?

[talmo]

Hi @skyshi0,

Do you have the console logs a few lines below your second screenshot? That's where we might see the error.

Thanks!

Talmo

[skyshi0]

Yes, I attached here. I didn't see any errors and it seemed that the centered instance model was continuing, so I hit the keyboard interrupt.

[skyshi0]

I attached another readout when I trained a single instance model, and the error indicated the resource exhausted

[eberrigan]

Hello @skyshi0,

This seems similar to the discussion #1889. You can try disabling the auto crop option for centered instance.

That tensor is very large. What is the resolution of your images?

Can you send us your training package by uploading it with your training images embedded to his link.

Best,

Elizabeth

[skyshi0]

Thank you! The video is 1440x1080px, and I've sent my training package with videos to the link.