FrogFS (Fast Read-Only General-purpose File System) is a read-only filesystem
designed for embedded use, including, but not limited to
ESP-IDF and
ESP8266_RTOS_SDK. There is
also a cross-platform example
available. It originally was written to be used with libesphttpd and Clockwise
HTTPd, but has been separated to allow other uses. There is a simple test
Linux program in tools/test
directory to read files from a FrogFS image.
Under the hood, it uses a sorted hash table and does a binary search to locate file and directory entries. The FrogFS binary can be embedded in your application or it can be accessed via memory mapped I/O such as the SPI flash memory space on the ESP8266 or ESP32. A set of python tools are provided to compress and/or obfusicate the data stored, and then create a final FrogFS image binary. Current compression methods include heatshrink and gzip -- the later without decompression support.
To use this library with IDF, make a components
directory in your project's
root directory and within that directory run:
git clone --recursive https://github.com/jkent/frogfs
Using the CMake build system, it is straightforward to generate or embed espfs images in your binary.
target_add_frogfs(target path [NAME name] [CONFIG yaml])
Symbols will be named name or if not specified, the last path part of path.
As an example for ESP-IDF, in your project's toplevel CMakeLists.txt:
cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(my_project)
include(components/frogfs/cmake/functions.cmake)
target_add_frogfs(my_project.elf html NAME frogfs CONFIG frogfs.yaml)
Where html is the path and name of the directory containing the root to use, frogfs is the name of the symbol to use, and frogfs.yaml is the configuration overrides. In C this results in these two symbols being defined:
extern const uint8_t frogfs_bin[];
extern const size_t frogfs_bin_len;
There is also the option to create a binary without linking it with your
application. A CMake function is provided to to output a binary with a new
target generate_${name}
. If name is not specified, the last path part of
path is used:
declare_frogfs_bin(path [NAME name] [CONFIG yaml])
Again, for ESP-IDF, in your project main's CMakeLists.txt:
set(name frogfs)
declare_frogfs_bin(../html NAME ${name} CONFIG ../frogfs.yaml)
idf_component_get_property(main_args esptool_py FLASH_ARGS)
idf_component_get_property(sub_args esptool_py FLASH_SUB_ARGS)
esptool_py_flash_target(${name}-flash "${main_args}" "${sub_args}" ALWAYS_PLAINTEXT)
esptool_py_flash_to_partition(${name}-flash frogfs ${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${COMPONENT_LIB}.dir/frogfs_${name}.bin)
add_dependencies(${name}-flash generate_${name})
In this case, ../html is the directory to build from, ${name} is the name of the binary name (without the .bin) and frogfs.bin and frogfs.yaml is the configuration overrides.
In the root of frogfs
There is a frogfs_defaults.yaml
file that has sane
defaults for HTTP usage. The yaml file defines filters for the various actions
-- preprocessors and compressors -- to run before building the FrogFS image.
The config file settings can enable or disable rules with wildcard matches
or absolute matches. For example:
filters:
"index.html": no-html-minifier
"*.html": gzip
"*.txt": [zeroify, uncompressed]
All preprocessors can be prefixed with no-
to skip the preprocessor for that
glob pattern. There are currently only two compressors, gzip
and
heatshrink
. Gzip is somewhat special in that FrogFS does not know how to
decompress it. Its expected to be decompressed by a client browser. Basically,
don't store your templates gzip compressed. There are also a few other special
actions. They are skip-preprocessing
, uncompressed
, cache
, discard
and
zeroify
. Skip-preprocessing does what it sounds like, no preprocessors will
run if it is specified. Uncompressed disables any compression option, cache is
just a flag that is used to tell the browser that it should cache the file
instead of re-downloading it the next visit. Discard means leave the specified
files out of the FrogFS image. And finally zeroify null terminates data.
You can define your own preprocessors. Look at the config_default.yaml within frogfs for an example. Preprocessors must take data on stdin and produce data on stdout. Installation of preprocessors can be done with the install key or using the npm key for node.js.
When using ESP_IDF, there are two ways you can use FrogFS. There is a low
level interface and there is a VFS interface. The VFS interface is the
recommended way to use FrogFS, since it allows you to use stdio
operations
on it. There is nothing preventing you from using both at the same time,
however.
extern const uint8_t frogfs_bin[];
frogfs_config_t frogfs_config = {
.addr = frogfs_bin,
};
frogfs_fs_t *fs = frogfs_init(&frogfs_config);
assert(fs != NULL);
frogfs_deinit(fs);
You can also mount a filesystem from a flash partition. Instead of specifying addr, you would specify a part_label string.
esp_vfs_frogfs_conf_t vfs_frogfs_conf = {
.base_path = "/frogfs",
.fs = fs,
.max_files = 5,
};
esp_vfs_frogfs_register(&vfs_frogfs_conf);
You can now use the system (open, read, close) or file stream (fopen, fread, fclose) functions to access files, with the FrogFS filesystem mounted to /frogfs.
const char *frogfs_get_path(frogfs_fs_t *fs, uint16_t index);
bool frogfs_stat(frogfs_fs_t *fs, const char *path, frogfs_stat_t *s);
frogfs_file_t *frogfs_fopen(frogfs_fs_t *fs, const char *path);
void frogfs_fclose(frogfs_file_t *f);
void frogfs_fstat(frogfs_file_t *f, const char *path, frogfs_stat_t *s);
ssize_t frogfs_fread(frogfs_file_t *f, void *buf, size_t len);
ssize_t frogfs_fseek(frogfs_file_t *f, long offset, int mode);
size_t frogfs_ftell(frogfs_file_t *f);
ssize_t frogfs_faccess(frogfs_file_t *f, void **buf);