Kalman filter calling for parameters not specifiable in GUI

[goodwinnastacia]

Bug description

When I try to run the Kalman filter (Predict > run inference> flow, enable filter after 10 frames, connect single track breaks), I get an error in my terminal saying that the Kalman filter requires target instance count. I do not see anywhere to specify the target instance count in the inference window. I am running this on videos with 9 bees.

Your personal set up

• OS:
  Windows 11 Pro

• Version(s):
  sleap-v1.3.4
  sleap_v1.4.1a2

• SLEAP installation method (listed here):
  Quick conda install, happening with versions 1.3.4 and 1.4.1a2

Environment packages

# Name Version Build Channel absl-py 2.1.0 pypi_0 pypi albumentations 1.3.1 pyhd8ed1ab_0 conda-forge astunparse 1.6.3 pypi_0 pypi attrs 21.4.0 pyhd8ed1ab_0 conda-forge backports-zoneinfo 0.2.1 pypi_0 pypi blosc 1.21.5 hdccc3a2_0 conda-forge brotli 1.0.9 hcfcfb64_9 conda-forge brotli-bin 1.0.9 hcfcfb64_9 conda-forge bzip2 1.0.8 h2466b09_7 conda-forge c-blosc2 2.12.0 h183a6f4_0 conda-forge ca-certificates 2024.8.30 h56e8100_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.3 pypi_0 pypi cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2024.8.30 pyhd8ed1ab_0 conda-forge cfitsio 4.0.0 hd67004f_0 conda-forge charls 2.3.4 h39d44d4_0 conda-forge charset-normalizer 3.3.2 pypi_0 pypi cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge cuda-nvcc 11.3.58 hb8d16a4_0 nvidia cudatoolkit 11.3.1 hf2f0253_13 conda-forge cudnn 8.2.1.32 h754d62a_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.12.0 py37hcc03f2d_0 conda-forge dask-core 2022.2.0 pyhd8ed1ab_0 conda-forge efficientnet 1.0.0 pypi_0 pypi flatbuffers 2.0.7 pypi_0 pypi fonttools 4.38.0 py37h51bd9d9_0 conda-forge freeglut 3.2.2 he0c23c2_3 conda-forge freetype 2.12.1 hdaf720e_2 conda-forge fsspec 2023.1.0 pyhd8ed1ab_0 conda-forge gast 0.4.0 pypi_0 pypi giflib 5.2.2 h64bf75a_0 conda-forge google-auth 2.29.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.62.1 pypi_0 pypi h5py 3.7.0 nompi_py37h24adfc3_101 conda-forge hdf5 1.12.2 nompi_h2a0e4a3_101 conda-forge hdmf 3.6.1 pypi_0 pypi icu 69.1 h0e60522_0 conda-forge idna 3.6 pypi_0 pypi image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2021.11.20 py37hfea8715_2 conda-forge imageio 2.35.1 pyh12aca89_0 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 4.2.0 pypi_0 pypi importlib_resources 6.0.0 pyhd8ed1ab_0 conda-forge intel-openmp 2024.2.1 h57928b3_1083 conda-forge jasper 2.0.33 hc2e4405_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e hcfcfb64_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge jxrlib 1.1 hcfcfb64_3 conda-forge keras 2.7.0 pypi_0 pypi keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.4.4 py37h8c56517_0 conda-forge krb5 1.20.1 h6609f42_0 conda-forge lcms2 2.14 h90d422f_0 conda-forge lerc 3.0 h0e60522_0 conda-forge libaec 1.1.3 h63175ca_0 conda-forge libblas 3.9.0 24_win64_mkl conda-forge libbrotlicommon 1.0.9 hcfcfb64_9 conda-forge libbrotlidec 1.0.9 hcfcfb64_9 conda-forge libbrotlienc 1.0.9 hcfcfb64_9 conda-forge libcblas 3.9.0 24_win64_mkl conda-forge libclang 18.1.1 pypi_0 pypi libcurl 8.1.2 h68f0423_0 conda-forge libdeflate 1.10 h8ffe710_0 conda-forge libhwloc 2.11.1 default_h8125262_1000 conda-forge libiconv 1.17 hcfcfb64_2 conda-forge liblapack 3.9.0 24_win64_mkl conda-forge liblapacke 3.9.0 24_win64_mkl conda-forge libopencv 4.5.1 py37ha0199f4_0 conda-forge libpng 1.6.43 h19919ed_0 conda-forge libprotobuf 3.21.8 h12be248_0 conda-forge libsodium 1.0.18 h8d14728_1 conda-forge libsqlite 3.46.1 h2466b09_0 conda-forge libssh2 1.10.0 h680486a_3 conda-forge libtiff 4.4.0 hc4061b1_0 conda-forge libwebp-base 1.4.0 hcfcfb64_0 conda-forge libxcb 1.13 hcd874cb_1004 conda-forge libxml2 2.12.7 h283a6d9_1 conda-forge libxslt 1.1.39 h3df6e99_0 conda-forge libzlib 1.2.13 h2466b09_6 conda-forge libzopfli 1.0.3 h0e60522_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4-c 1.9.3 h8ffe710_1 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.4.4 pypi_0 pypi markdown-it-py 2.2.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.5 pypi_0 pypi matplotlib-base 3.5.3 py37hbaab90a_2 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge mkl 2024.1.0 h66d3029_694 conda-forge msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge ndx-pose 0.1.1 pyhd8ed1ab_0 conda-forge networkx 2.7 pyhd8ed1ab_0 conda-forge nixio 1.5.3 pypi_0 pypi numpy 1.21.6 py37h2830a78_0 conda-forge oauthlib 3.2.2 pypi_0 pypi opencv 4.5.1 py37h03978a9_0 conda-forge openjpeg 2.5.0 hc9384bd_1 conda-forge openssl 1.1.1w hcfcfb64_0 conda-forge opt-einsum 3.3.0 pypi_0 pypi packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 1.3.5 py37h9386db6_0 conda-forge partd 1.4.1 pyhd8ed1ab_0 conda-forge patsy 0.5.6 pyhd8ed1ab_0 conda-forge pillow 9.2.0 py37h42a8222_2 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge protobuf 3.20.3 pypi_0 pypi psutil 5.9.3 py37h51bd9d9_0 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge py-opencv 4.5.1 py37heaed05f_0 conda-forge pyasn1 0.5.1 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi pygments 2.17.2 pyhd8ed1ab_0 conda-forge pykalman 0.9.7 pyhd8ed1ab_0 conda-forge pynwb 2.3.3 pypi_0 pypi pyparsing 3.1.4 pyhd8ed1ab_0 conda-forge pyrsistent 0.18.1 py37hcc03f2d_1 conda-forge pyside2 5.13.2 py37h760f651_8 conda-forge python 3.7.12 h7840368_100_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-rapidjson 1.9 py37h7f67f24_0 conda-forge python_abi 3.7 4_cp37m conda-forge pytz 2024.2 pyhd8ed1ab_0 conda-forge pywavelets 1.3.0 py37h3a130e4_1 conda-forge pyyaml 6.0 py37hcc03f2d_4 conda-forge pyzmq 24.0.1 py37h7347f05_0 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt 5.12.9 h556501e_6 conda-forge qtpy 2.4.1 pyhd8ed1ab_0 conda-forge qudida 0.0.4 pyhd8ed1ab_0 conda-forge requests 2.31.0 pypi_0 pypi requests-oauthlib 2.0.0 pypi_0 pypi rich 13.8.1 pyhd8ed1ab_0 conda-forge ruamel.yaml 0.17.21 py37hcc03f2d_1 conda-forge ruamel.yaml.clib 0.2.6 py37hcc03f2d_1 conda-forge scikit-image 0.19.2 py37h9386db6_0 conda-forge scikit-learn 1.0 py37ha78be43_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.7.3 py37hb6553fb_0 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge segmentation-models 1.0.1 pypi_0 pypi setuptools 59.8.0 py37h03978a9_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sleap 1.4.1a2 pypi_0 pypi snappy 1.1.10 hfb803bf_1 conda-forge sqlite 3.46.1 h2466b09_0 conda-forge statsmodels 0.13.2 py37h3a130e4_0 conda-forge tbb 2021.13.0 hc790b64_0 conda-forge tensorboard 2.11.2 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow 2.7.0 pypi_0 pypi tensorflow-estimator 2.7.0 pypi_0 pypi tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-io-gcs-filesystem 0.31.0 pypi_0 pypi termcolor 2.3.0 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tifffile 2021.11.2 pyhd8ed1ab_0 conda-forge tk 8.6.13 h5226925_1 conda-forge toolz 0.12.1 pyhd8ed1ab_0 conda-forge typing-extensions 4.7.1 hd8ed1ab_0 conda-forge typing_extensions 4.7.1 pyha770c72_0 conda-forge tzdata 2024.1 pypi_0 pypi tzlocal 5.1 pypi_0 pypi ucrt 10.0.22621.0 h57928b3_0 conda-forge unicodedata2 14.0.0 py37hcc03f2d_1 conda-forge urllib3 2.0.7 pypi_0 pypi vc 14.3 h8a93ad2_21 conda-forge vc14_runtime 14.40.33810 ha82c5b3_21 conda-forge vs2015_runtime 14.40.33810 h3bf8584_21 conda-forge werkzeug 2.2.3 pypi_0 pypi wheel 0.42.0 pyhd8ed1ab_0 conda-forge wrapt 1.16.0 pypi_0 pypi xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge zeromq 4.3.4 h0e60522_1 conda-forge zfp 0.5.5 h0e60522_8 conda-forge zipp 3.15.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h2466b09_6 conda-forge zlib-ng 2.0.7 hcfcfb64_0 conda-forge zstd 1.5.6 h0ea2cb4_0 conda-forge

Logs

Happy SLEAPing! :)
Using already trained model for centroid: //172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp\models\240830_162626.centroid.n=196\initial_config.json
Using already trained model for centered_instance: //172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp\models\240830_171136.centered_instance.n=196\initial_config.json
Command line call:
sleap-track //172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp/Basic_Aruco_Test.slp --video.index 0 --frames 0,-5242 -m //172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp\models\240830_162626.centroid.n=196\initial_config.json -m //172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp\models\240830_171136.centered_instance.n=196\initial_config.json --batch_size 4 --tracking.tracker flow --max_instances 9 --tracking.similarity instance --tracking.match greedy --tracking.track_window 10 --tracking.kf_init_frame_count 10 --tracking.kf_node_indices 1,8,9 --tracking.post_connect_single_breaks 1 --controller_port 9000 --publish_port 9001 -o //172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp\predictions\Basic_Aruco_Test.slp.241001_161416.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2024-10-01 16:14:21.456126
Args:
{
    'data_path': '//172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp/Basic_Aruco_Test.slp',
    'models': [
        '//172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp\\models\\240830_162626.centroid.n=196\\initial_config.json',
        '//172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp\\models\\240830_171136.centered_instance.n=196\\initial_config.json'
    ],
    'frames': '0,-5242',
    'only_labeled_frames': False,
    'only_suggested_frames': False,
    'output': '//172.25.226.40/ZYWangLab/SLEAP_Models_Shared/Aruco_11bp\\predictions\\Basic_Aruco_Test.slp.241001_161416.predictions.slp',
    'no_empty_frames': True,
    'verbosity': 'json',
    'video.dataset': None,
    'video.input_format': 'channels_last',
    'video.index': '0',
    'cpu': False,
    'first_gpu': False,
    'last_gpu': False,
    'gpu': 'auto',
    'max_edge_length_ratio': 0.25,
    'dist_penalty_weight': 1.0,
    'batch_size': 4,
    'open_in_gui': False,
    'peak_threshold': 0.2,
    'max_instances': 9,
    'tracking.tracker': 'flow',
    'tracking.max_tracking': None,
    'tracking.max_tracks': None,
    'tracking.target_instance_count': None,
    'tracking.pre_cull_to_target': None,
    'tracking.pre_cull_iou_threshold': None,
    'tracking.post_connect_single_breaks': 1,
    'tracking.clean_instance_count': None,
    'tracking.clean_iou_threshold': None,
    'tracking.similarity': 'instance',
    'tracking.match': 'greedy',
    'tracking.robust': None,
    'tracking.track_window': 10,
    'tracking.min_new_track_points': None,
    'tracking.min_match_points': None,
    'tracking.img_scale': None,
    'tracking.of_window_size': None,
    'tracking.of_max_levels': None,
    'tracking.save_shifted_instances': None,
    'tracking.kf_node_indices': [1, 8, 9],
    'tracking.kf_init_frame_count': 10
}

INFO:sleap.nn.inference:Auto-selected GPU 0 with 6929 MiB of free memory.
Traceback (most recent call last):
Versions:
  File "C:\Users\Nastacia\anaconda3\envs\sleap_v1.4.1a2\Scripts\sleap-track-script.py", line 33, in <module>
    sys.exit(load_entry_point('sleap==1.4.1a2', 'console_scripts', 'sleap-track')())
  File "C:\Users\Nastacia\anaconda3\envs\sleap_v1.4.1a2\lib\site-packages\sleap\nn\inference.py", line 5467, in main
    tracker = _make_tracker_from_cli(args)
  File "C:\Users\Nastacia\anaconda3\envs\sleap_v1.4.1a2\lib\site-packages\sleap\nn\inference.py", line 5401, in _make_tracker_from_cli
    tracker = Tracker.make_tracker_by_name(**policy_args["tracking"])
  File "C:\Users\Nastacia\anaconda3\envs\sleap_v1.4.1a2\lib\site-packages\sleap\nn\tracking.py", line 916, in make_tracker_by_name
    raise ValueError("Kalman filter requires target instance count.")
ValueError: Kalman filter requires target instance count.
SLEAP: 1.4.1a2
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0

System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
       Initialized: False
     Memory growth: True

Process return code: 1

Screenshots

How to reproduce

Open SLEAP labeling interface
Open project
Predict > run inference
Select parameters in screenshot
(multi-animal top down, max instances 9, batch size 4, tracker method flow, max number of tracks 9, similarity method instance, matching method greedy, elapsed frame window 10, robust quantile of similarity scores use max, enable filters after 10 initial frames, nodes to use for tracking 1,8,9, connect single track breaks. Predict on entire current video).

[github-actions]

Thank you for opening a new issue!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[eberrigan]

Hi @goodwinnastacia,

Thanks for bringing this to our attention. Working on the fix now and will get back to you soon!

Best,

Elizabeth

[eberrigan]

@goodwinnastacia I am working on this now and hope to have the fix in the GUI by the end of the week. In the mean time, you can use sleap-track from the command-line with the relevant flags. See #1583.

You will use

--tracking.kf_node_indices TRACKING.KF_NODE_INDICES
                        For Kalman filter: Indices of nodes to track. (default: )
--tracking.kf_init_frame_count TRACKING.KF_INIT_FRAME_COUNT
                        For Kalman filter: Number of frames to track with other tracker. 0 means no Kalman filters will be used. (default: 0)
--tracking.target_instance_count TRACKING.TARGET_INSTANCE_COUNT
                        Target number of instances to track per frame. (default: 0)
--tracking.post_connect_single_breaks TRACKING.POST_CONNECT_SINGLE_BREAKS
                        If non-zero and target_instance_count is also non-zero, then connect track breaks when exactly one track is lost and exactly
                        one track is spawned in frame. (default: 0)

and the following are optional

  --tracking.pre_cull_to_target TRACKING.PRE_CULL_TO_TARGET
                        If non-zero and target_instance_count is also non-zero, then cull instances over target count per frame *before* tracking.
                        (default: 0)
  --tracking.pre_cull_iou_threshold TRACKING.PRE_CULL_IOU_THRESHOLD
                        If non-zero and pre_cull_to_target also set, then use IOU threshold to remove overlapping instances over count *before*
                        tracking. (default: 0)

Please note that in general we have more options from the command-line than the GUI, so you might prefer to use the command-line.

[goodwinnastacia]

Thank you so much!

[eberrigan]

Hello @goodwinnastacia,

We have allowed the max tracks in the GUI to be used for the Kalman filter so that max tracking can be used instead of target_instance_count.

In general we recommend using optical flow instead of the Kalman filter. Optical flow uses continuously updated points, shifted by the Lucas-Kanade Optical Flow equation. Please see our tracking tutorial.

We have limited the scope of the use of the Kalman filter because in general we get much better results using optical flow for tracking. The Kalman filter now only works with simple tracking as the initializer and no longer works with the normalized_instance method for similarity. Note that the initial tracker is only used for the specified number of frames. Then the Kalman filter filters out the least probable tracks based on evolution from these initial frames.

Changes are in #1986.

Let us know if you have any more questions!

Elizabeth