Error frame towards end of video and then when using select frames to continue prediction, predictions off.

[dlu67]

I have been encountering issues where SLEAP would crash at 99% when training due to errors to load a specific frame at the end of videos. It happens with every video and previously I resolved it by trimming away from the end to the frame causing the error to remove it. This stopped working and continued to produce new error frames towards the end of the video, the issue is not resolved by re-encoding the video either.
I then proceeded to use the select function to select frames up to the frame before the error frame to run predictions on. This allowed for the video to finish training and for predictions to appear, however I noticed that the predictions in this video became very off, it seems like it is tracking the movements of the mouse in prior or future frames instead, resulting in predictions being way off. Looking for solutions or suggestions to this problem, thank you so much.

[roomrys]

Hi @dlu67,

In the past, re-encoding has worked for us and the errors you mention ("errors to load specific frame" and "predictions being way off") after trying re-encoding sound like the video was not properly re-encoded. So, I wonder if either the re-encoding failed or the Video.filename was not updated in SLEAP.

Do you mind walking through with me (and answering a few questions)?

1. Find the name of the video you want to re-encode (let's call it name_of_video.mp4)
2. Rename the video you want to re-encode to name_of_video_broken.mp4. (Renaming the video allows us to save the encoded video to the original name - which is what SLEAP's Video.filename will be referencing).
3. Open a terminal at the directory/folder that contains the video
4. Activate the sleap environment (which contains ffmpeg) with

   conda activate sleap
   

5. Run the command

   ffmpeg -y -i "name_of_video_broken.mp4" -c:v libx264 -pix_fmt yuv420p -preset superfast -crf 23 "name_of_video.mp4"
   

   . Note that if your terminal is not open in the directory/folder that contains name_of_video_broken.mp4, then the command will fail (but it can also fail for other reasons).
6. Copy and paste any terminal output for us here. It is possible that the re-encoding failed in which case we can help solve.
7. If the terminal messages indicate that the re-encoding was successful, then try running inference again to see if the problem was resolved.

Please let us know if you need any assistance or if this was helpful.

Thanks,
Liezl

[dlu67]

Hi Liezl,

Thank you for your suggestions, here is the output code. However, I believe it was re-encoded successfully last time too, it still crashes after re-encoding was the issue. That was why I instead tried to just select the frames before the crash occurred which led to the predictions being off, but I noticed that the predictions were actually just predicting frames faster than the video was displaying if that makes sense. As in, the predictions were accurate if they were to be compared to frames a couple hundred frames ahead of the current one. (This is not unique to one video either, it happens to all the videos in which I select a range of frames to predict on)
Anyways, here is the output code.

[roomrys]

Hi @dlu67,

I understand you have already attempted some of these steps - I just want to make sure we aren't overlooking something, so thanks for bearing with me!

• Q1. Also, before we continue, did inference behave strangely after re-attempting the re-encoding?

Ok, continuing... From your terminal output, it does seem like the re-encoding executed successfully, but it is a bit strange that ffmpeg entered interactive mode and is waiting for further input as seen by:

Enter command: <target>|all <time>|-1 <command>[ <argument>]

. Where you get an interactive prompt, I get a breakdown of the encoding statistics:

[libx264 @ 0x14c804a90] frame I:5     Avg QP: 7.93  size: 24321
[libx264 @ 0x14c804a90] frame P:476   Avg QP:12.22  size:  3106
[libx264 @ 0x14c804a90] frame B:620   Avg QP:12.00  size:   923
[libx264 @ 0x14c804a90] consecutive B-frames: 19.3% 13.6% 10.1% 57.0%
[libx264 @ 0x14c804a90] mb I  I16..4: 42.3% 10.4% 47.3%
[libx264 @ 0x14c804a90] mb P  I16..4:  1.9%  0.6%  2.3%  P16..4:  5.5%  0.0%  0.0%  0.0%  0.0%    skip:89.7%
[libx264 @ 0x14c804a90] mb B  I16..4:  0.1%  0.0%  0.2%  B16..8:  6.1%  0.0%  0.0%  direct: 0.2%  skip:93.4%  L0:49.6% L1:48.8% BI: 1.6%
[libx264 @ 0x14c804a90] 8x8 transform intra:11.5% inter:62.1%
[libx264 @ 0x14c804a90] coded y,uvDC,uvAC intra: 26.9% 0.1% 0.0% inter: 1.1% 0.0% 0.0%
[libx264 @ 0x14c804a90] i16 v,h,dc,p: 73% 21%  6%  0%
[libx264 @ 0x14c804a90] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 27% 21% 27% 10%  1%  1%  0%  1% 12%
[libx264 @ 0x14c804a90] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 48% 25% 14%  3%  3%  2%  2%  1%  3%
[libx264 @ 0x14c804a90] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0x14c804a90] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x14c804a90] kb/s:394.62

followed by a new prompt liezl:~$.

The next thing I would like to check is that the video we just re-encoded is definitely being used by our SLEAP project.

1. Open a terminal at the directory containing your slp file
2. Activate your sleap environment with

   conda activate sleap

3. Start an interactive python session

   ipython

4. Check the video filenames references in your project are as expected

   (sleap) liezl:~$ipython
   Python 3.9.20 | packaged by conda-forge | (main, Sep 30 2024, 17:48:00) 
   Type 'copyright', 'credits' or 'license' for more information
   IPython 8.18.1 -- An enhanced Interactive Python. Type '?' for help.
   
   In [1]: import sleap
   
   In [2]: ds = "courtship_labels.slp"
   
   In [3]: labels = sleap.load_file(ds)
   
   In [4]: labels.videos
   Out[4]: 
   [Video(backend=MediaVideo(filename='/Users/liezl/Projects/sleap-datasets/drosophila-melanogaster-courtship/20190128_113421.mp4', grayscale=True, bgr=True, dataset='', input_format='')),
    Video(backend=MediaVideo(filename='/Users/liezl/Projects/sleap-datasets/drosophila-melanogaster-courtship/dmc_720.mp4', grayscale=True, bgr=True, dataset='', input_format=''))]
   
   In [5]: exit()

• Q2. Do the video filenames received in step 4 match the filename of the re-encoded video?
• Q3. You mentioned the predictions were good, but off when you select a clip to predict on. Would you say that they are off by the amount of frames from start of video to start of selected clip?

Thanks,
Liezl

[dlu67]

Hi Liezl,

Thank you so much for your time and help. I have run the test with ipython and can confirm that the output names matches the videos I will include a screenshot.

I would say that they are off by the amount of frames from the end of the selected clip to the end of the video. The selected clip starts where the video starts but ends earlier than the actual video to avoid the error frame that it always crashes on.

[roomrys]

Hi @dlu67,

Sorry for the long troubleshooting.

Our group had a meeting and someone brought up that it might be worth while to try re-encoding to mp4 because avi can be "troublesome". To make this go a little faster, is it possible for me to get my hands on the error-prone video. We have a user upload form here and can remove your data after troubleshooting if you'd like.

Thanks,
Liezl

[dlu67]

Hi,

Thank you so much for helping, I have uploaded a re-encoded video for you guys to look at. I also have tried re-encoding to mp4 before but it also ended up with the same error.
To move on with my current project I started selecting frames to encode (up until 1 frame before the error frame) and this worked okay for a while, despite the frame-shift in predictions, I could still get the data I wanted. However, now I am encountering a new problem where even though I selected up to before the error frame, it would still crash and say it was unable to load the error frame. For example if the error frame was 84923, and I selected for the video to infer from 1-84900, it would still crash and would produce the KeyError of unable to load frame 84923.