Timestamps in data

[roaldarbol]

Hi again! :-)
I've tried to look through the documentation and the .slp file itself, but I can't find any timestamps. Does SLEAP not keep the timestamps from the video?

[roomrys]

Hi @roaldarbol,

SLEAP does not keep the timestamps. If the video you loaded in SLEAP has one of these extensions:

sleap/sleap/io/video.py

Line 354 in 28c34e2

EXTS = ("mp4", "avi", "mov", "mj2", "mkv")

then the Video object it was loaded into has a fps property

sleap/sleap/io/video.py

Lines 432 to 435 in 28c34e2

	@property
	def fps(self) -> float:
	"""Returns frames per second of video."""
	return self.__reader.get(cv2.CAP_PROP_FPS)

that returns the frames per second which you can use in conjunction with the frame index to calculate a relative timestamp (video starts at time 0).

(sleap) roomrys:~$ipython

In [1]: from sleap import Labels

In [2]: ds = "/Users/roomrys/Projects/sleap-datasets/drosophila-melanogaster-
   ...: courtship/courtship_labels copy.slp"

In [3]: labels = Labels.load_file(ds)

In [4]: labeled_frame = labels.labeled_frames[0]

In [5]: video = labeled_frame.video

In [6]: frame_idx = labeled_frame.frame_idx

In [7]: video.fps
Out[7]: 25.0

In [8]: frame_idx / video.fps
Out[8]: 0.4

Sorry about that,
Liezl

[roaldarbol]

Hi @roomrys!
Ah, I see. Yeah, I guess the video files don't encode the timestamps (which is a pity!). Since most cameras have a rolling shutter, the fps approach is quite a rough estimate (as fps fluctuate a fair amount throughout); I actually ended up outputting time stamps with Bonsai whilst recording to have the precise ones. But I completely see how it's out of the SLEAP remit, I'de just forgotten all about that. ;-)

[roaldarbol]

Just came across this which can be achieved with ffprobe: https://superuser.com/a/1418998.

[roaldarbol]

Looked into it a bit more, and it doesn't work - it simply uses the fps to "re-create" the playback timestamp, not the actual time of recording.

[talmo]

Hi @roaldarbol,

In case you haven't found your way over to this issue yet, there's a good discussion over here on the topic of video timestamps/accurate+fast seeking/encoding and etc: janclemenslab/napari-video#3

Hope that helps!

FWIW, our setup is to store the timestamps separately from Bonsai as well. We pretty much entirely disregard the video FPS and "timestamps" stored in the video container metadata.

Talmo

[roaldarbol]

Hi @talmo!

I indeed hadn't! Super interesting, and I learned a lot there!

The more I've immersed myself in all this video stuff, the more I feel like it would be great to bring together a team to (get funding for and) develop a super robust video acquisition software that can handle a bunch of different camera types, simultaneous recording from multiple cameras; as you mentioned, a lot of it comes down to lots of poor acquisition software, I've never had access to high-end cameras or proprietary video acquisition software. Bonsai's great, but the fact that it's only cross-platform. with a lot of hassle has turned me away from it in the past. Something simple, that works for everyone, video only.

Cheers, Mikkel

[talmo]

You might like: https://github.com/ksseverson57/campy

[roaldarbol]

Ideally I'd like to see something that works with webcams too, GoPros, Raspberry Pi cams, etc. (as well as industrial cameras). It's simply much more common in smaller, poorly funded labs. Something that could make it just as easy to gain the advantages on a laptop in the field. :-)

[roaldarbol]

I've brought up the idea to the FreeMoCap folks, as FreeMoCap also features a recording interface, and that work might be well worth building on.