Add LFS_F_WRUNCHECKED option to avoid validation and hence reading blocks when writing

[andyp1per]

In ArduPilot we predominantly stream logs to disk while flying. In our current block logger we do this without validation and this PR adds an option to skip validation while writing to streamline the writing process.

Also fixes a build issue with the file backend when compiling without debug.

[geky-bot]

Tests passed ✓, Code: 17136 B (+0.0%), Stack: 1440 B (+0.0%), Structs: 812 B (+0.0%)

	Code	Stack	Structs		Coverage
Default	17136 B (+0.0%)	1440 B (+0.0%)	812 B (+0.0%)	Lines	2400/2572 lines (-0.0%)
Readonly	6250 B (+0.0%)	448 B (+0.0%)	812 B (+0.0%)	Branches	1283/1616 branches (+0.0%)
Threadsafe	17984 B (+0.0%)	1440 B (+0.0%)	820 B (+0.0%)		Benchmarks
Multiversion	17208 B (+0.0%)	1440 B (+0.0%)	816 B (+0.0%)	Readed	29369693876 B (+0.0%)
Migrate	18824 B (+0.0%)	1736 B (+0.0%)	816 B (+0.0%)	Proged	1482874766 B (+0.0%)
Error-asserts	17900 B (+0.0%)	1432 B (+0.0%)	812 B (+0.0%)	Erased	1568888832 B (+0.0%)

[geky]

Hi @andyp1per, thanks for creating a PR.

Have you considered making this a filesystem-wide ifdef, say, LFS_NO_CKPROGS? That way you could also avoid the reads to validate metadata commits (here). I have a similar thing on a fork (albeit via mount flags).

I'm curious, did this impact performance noticeably? I've been viewing this as low priority as I assumed the read overhead for validation would always be overshadowed by the related prog+erase overhead.

[andyp1per]

Hi @andyp1per, thanks for creating a PR.

Have you considered making this a filesystem-wide ifdef, say, LFS_NO_CKPROGS? That way you could also avoid the reads to validate metadata commits (here). I have a similar thing on a fork (albeit via mount flags).

I'm curious, did this impact performance noticeably? I've been viewing this as low priority as I assumed the read overhead for validation would always be overshadowed by the related prog+erase overhead.

I am interested in other types of file being checked, that's why I went for a per-file option. It might also be possible to set it temporarily - so for instance our logs are self-describing, we probably want the description to be stored reliably but the data not so much.

read is about 10x faster than write and it also chews up the SPI bus, so not nothing - in my testing so far this plus avoiding checking the file size seems to get me where I need to, but I need to do another full flight test to fully check

[geky]

Hi @andyp1per, just a quick update. I think I'm going to have to leave this open for a while due to how it interacts with future work.

In the ongoing experimental branch (rbyd), the separation of file data-structure and metadata data-structure becomes quite a bit more muddy, so I'm not sure it will be possible to set this flag on a per-file basis.

I'm also growing a bit concerned we're going to run out of 32-bit flags with the current planned features...

[bmcdonnell-fb]

In the ongoing experimental branch (rbyd),

BTW, what does "rbyd" mean?

[geky]

BTW, what does "rbyd" mean?

That's a good question! It's the name of the new metadata-data structure that started the branch. The original goal was to bring mdir compaction down from $O(n^2)$ to $O(n \log n)$, which is one of the main performance issues in littlefs, but it also serves as a backbone for file B-trees and a bunch of other things.

At this point it's sort of snowballed into a number of seemingly unrelated features.

I'm not particularly happy with how it's snowballed, but also not sure there's a better way to go about it. With filesystems/storage, things get "stuck" as soon as they're released, so new features need somewhere to "stew" while details/bugs get worked out...

---

The name itself comes from combining Dhara trees (which you may know about, it and the Dhara FTL are really quite neat), with a red-black-yellow tree to make it self-balancing and order-statistic.

Maybe not the greatest name, but a name's a name.

There's a lot to write about it, but that's also a todo item...

[geky]

With filesystems/storage, things get "stuck" as soon as they're released

I should also mention I do plan to have a sort of "request-for-comments/feedback" period before stabilizing, but at the moment I'm still trying to get the details/kinks worked out.

[geky]

One neat example, order-statistic operations make things like lfs_file_fruncate possible:

littlefs/lfs.h

Lines 1133 to 1139 in 1f3c47b

	// Truncate/grow the file, but from the front
	//
	// If size is larger than the current file size, a hole is created, appearing
	// as if the file was filled with zeros.
	//
	// Returns a negative error code on failure.
	int lfsr_file_fruncate(lfs_t *lfs, lfsr_file_t *file, lfs_off_t size);

Which I think will be quite useful for maintaining logs without needing multiple files.

[andyp1per]

One neat example, order-statistic operations make things like lfs_file_fruncate possible:

littlefs/lfs.h

Lines 1133 to 1139 in 1f3c47b

	// Truncate/grow the file, but from the front
	//
	// If size is larger than the current file size, a hole is created, appearing
	// as if the file was filled with zeros.
	//
	// Returns a negative error code on failure.
	int lfsr_file_fruncate(lfs_t *lfs, lfsr_file_t *file, lfs_off_t size);

Which I think will be quite useful for maintaining logs without needing multiple files.

Presumably that would make a posix compliant version of SEEK_SET possible.

[geky]

Presumably that would make a posix compliant version of SEEK_SET possible.

It does make random writes efficient, that is another big benefit.

Though I believe the current implementation of SEEK_SET is technically POSIX compliant, if only practical for random reads and random writes to small files.

[andyp1per]

This is the bit that is not supported:

POSIX allows seeking beyond the existing end of file. If an output is performed after this seek, any read from the gap will return zero bytes. Where supported by the filesystem, this creates a sparse file. 

[geky]

This is the bit that is not supported:

POSIX allows seeking beyond the existing end of file. If an output is performed after this seek, any read from the gap will return zero bytes. Where supported by the filesystem, this creates a sparse file.

Ah, I see. Yes, that's a good point, sparse files are a nice side-effect of B-trees.

I'm curious if what use cases you've seen for sparse files (SQLite?). I've been considering them low-priority since advanced file manipulation seems relatively rare in this space.

Sparse files are actually already implemented (it'd be a bit silly not to with B-trees), but more advanced APIs like SEEK_DATA, lfs_file_punchhole, lfs_file_collapserange, etc, are currently out-of-scope (but would be nice in the future).