Merge pull request #92 from sz3/0.6-mode-switch

Update cimbar_js UI with "mode B" toggle + docs

DETAILS.md

@@ -84,7 +84,7 @@ These properties may appear to be magical as you consider them more, and they do
 2. wirehair requires the file contents to be stored in RAM
 	* this relates to the size limit!
 
-This constraint is less of an obstacle than it may seem -- the fountain codes are essentially being used as a wire format, and the encoder and decoder could agree on a scheme to split up, and then reassemble, larger files. Cimbar does not (yet?) implement this, however!
+The size constraint is less of an obstacle than it may seem -- the fountain codes are essentially being used as a wire format, and the encoder and decoder could agree on a scheme to split up, and then reassemble, larger files. Cimbar does not (yet?) implement this, however!
 
 ## Implementation: Decoder
 

PERFORMANCE.md

@@ -4,36 +4,48 @@
 ## Numbers of note
 
 * The barcode is `1024x1024` pixels. The individual tiles are `8x8` in a `9x9` grid (there is an empty row/column of spacing on either side)
-* **7500** or 8750 bytes per cimbar image, after error correction
+* **7500** bytes per cimbar image, after error correction
 	* There are 16 possible symbols per tile, encoding 4 bits
 	* There are 4 or 8 possible colors, encoding an additional 2-3 bits per tile.
-	* These 6-7 bits per tile work out to a maximum of 9300-10850 bytes per barcode, though in practice this number is reduced by error correction.
+	* These 6 bits per tile work out to a maximum of 9300 bytes per barcode, though in practice this number is reduced by error correction.
 	* The default ecc setting is 30/155, which is how we go from 9300 -> 7500 bytes of real data for a 4-color cimbar image.
 		* Reed Solomon is not perfect for this use case -- specifically, it corrects byte errors, and cimbar errors tend to involve 1-3 bits at a time. However, since Reed Solomon implementations are ubiquitous, it is currently in use.
 
 ## Current sustained benchmark
 
-* 4-color cimbar with ecc=30:
+* `mode B` (8x8 4-color) cimbar with ecc=30/155:
+	* 4,689,084 bytes (after compression) in 44s -> 852 kilobits/s (~106 KB/s)
+	* mode B was introduced in 0.6.0, and should work in a wide variety of scenarios
+
+* *legacy* `mode 4C` (8x8 4-color) cimbar with ecc=30/155:
 	* 4,717,525 bytes (after compression) in 45s -> 838 kilobits/s (~104 KB/s)
+	* the original configuration. Mostly replaced by mode B.
 
-* 8-color cimbar with ecc=30:
+* *deprecated* `mode 8C` (8x8 8-color) cimbar with ecc=30/155:
 	* 4,717,525 bytes in 40s -> 943 kilobits/s (~118 KB/s)
+	* removed in 0.6.0. 8-color has always been inconsistent, and needs future research
+
+* *beta* `mode S` (5x5 4-color) cimbar with ecc=40/216 (note: not finalized, and requires a special build)
+	* safely >1 Mbit/s
+	* format still a WIP. To be continued...
 
 * details:
 	* cimbar has built-in compression using zstd. What's being measured here is bits over the wire, e.g. data after compression is applied.
-	* these numbers are using https://github.com/sz3/cfc, running with 4 CPU threads on a Qualcomm Snapdragon 625
-		* perhaps I will buy a new cell phone to inflate the benchmark numbers.
-	* the sender is the cimbar.org wasm implementation. An equivalent command line is `./cimbar_send /path/to/file -s`
+	* these numbers are using https://github.com/sz3/cfc, running with 4 CPU threads on a venerable Qualcomm Snapdragon 625
+		* more modern cell CPUs run the decoder more quickly, but it turns out that this does not benefit performance much: the camera is usually the bottleneck.
+	* the sender is the cimbar.org wasm implementation. An equivalent command line is `./cimbar_send /path/to/file`
 		* cimbar.org uses the `shakycam` option to allow the receiver to detect/discard "in between" frames as part of the scan step. This allows it to spend more processing time decoding real data.
 	* burst rate can be higher (or lower)
-		* to this end, lower ecc settings *can* provide better burst rates
-	* 4-color cimbar is currently preferred, and will give more consistent transfer speeds.
-	* 8-color cimbar should be considered a prototype within a prototype. It is considerably more sensitive to lighting conditions and color tints.
+		* to this end, lower ecc settings *can* provide better burst rates. I've aimed for a balance of performance and reliability.
+	* cimbar `mode B` is preferred, and should be the most reliable.
+	* The older `mode 4C` *may* give more consistent transfer speeds in certain scenarios, but is mostly included for backwards-compatibility reasons.
 
 * other notes:
-	* having better lighting in the frame often leads to better results -- this is why cimbar.org has a (mostly) white background. cfc uses android's auto-exposure, auto-focus, etc (it's a very simple app). Good ambient light -- or a white background -- can lead to more consitent quality frame capture.
+	* having better lighting in the frame often leads to better results -- this is why cimbar.org has a (mostly) white background. cfc uses android's auto-exposure, auto-focus, etc (it's a demo app). Good ambient light -- or a white background -- can lead to more consitent quality frame capture.
+		* screen brightness on the sender is good, but ambient light is better.
 	* because of the lighting/exposure question, landscape *may* be better than portrait.
-	* cfc currently has a low resolution, so the cimbar frame should take up as much of the display as possible (trust the guide brackets)
+	* the cimbar frame should take up as much of the display as possible (trust the guide brackets)
+		* the format is designed to decode at resolutions as low as 700x700, but performance may suffer.
 	* similarly, it's best to keep the camera angle straight-on -- instead of at an angle -- to decode the whole image successfully. Decodes should still happen at higher angles, but the "smaller" part of the image may have more errors than the ECC can deal with.
 	* other things to be wary of:
 		* glare from light sources.

README.md

@@ -5,10 +5,10 @@
 
 Behold: an experimental barcode format for air-gapped data transfer.
 
-It can sustain speeds of 943+ kilobits/s (~118 KB/s) using just a computer monitor and a smartphone camera!
+It can sustain speeds of 850 kilobits/s (~106 KB/s) using just a computer monitor and a smartphone camera!
 
 <p align="center">
-<img src="https://github.com/sz3/cimbar-samples/blob/v0.5/6bit/4cecc30f.png" width="70%" title="A non-animated cimbar code" >
+<img src="https://github.com/sz3/cimbar-samples/blob/v0.6/b/4cecc30f.png" width="70%" title="A non-animated mode-B cimbar code" >
 </p>
 
 ## Explain?
@@ -31,7 +31,7 @@ No internet/bluetooth/NFC/etc is used. All data is transmitted through the camer
 
 The code is written in C++, and developed/tested on amd64+linux, arm64+android (decoder only), and emscripten+WASM (encoder only). It probably works, or can be made to work, on other platforms.
 
-Crucially, because the encoder compiles to asmjs and wasm, it can run on anything with a modern web browser. There are [releases](https://github.com/sz3/libcimbar/releases/latest) if you wish to run the encoder locally instead of via cimbar.org.
+Crucially, because the encoder compiles to asmjs and wasm, it can run on anything with a modern web browser. For offline use, you can either install cimbar.org as a progressive web app, or [download the latest release](https://github.com/sz3/libcimbar/releases/latest) of `cimbar_js.html`, save it locally, and open it in your web browser.
 
 ## Library dependencies
 

TODO.md

@@ -10,6 +10,7 @@ Performance optimizations aside, there are a number of paths that might be inter
 * proper metadata/header information?
 	* would be nice to be able to determine ecc/#colors/#symbols from the cimbar image itself?
 	* The bottom right corner is the obvious place to reclaim space to make this possible.
+	* this is complicated by potential aspect ratio changes for future cimbar modes.
 * multi-frame decoding?
 	* when decoding a static cimbar image, it would be useful to be able to use prior (unsuccessful) decode attempts to inform a future decode, and -- hopefully -- increase the probability of success. Currently, all frames are decoded independently.
 		* there is already a granular confidence metric that could be reused -- the `distance` that's tracked when decoding symbol tiles...
@@ -18,18 +19,18 @@ Performance optimizations aside, there are a number of paths that might be inter
 	* there is surely a more optimal set -- a more rigorous approach should yield lower error rates!
 	* but, more importantly, it may be possible to go up to 32 symbols, and encode 5 symbol bits per tile?
 * optimal symbol size?
-	* the symbols that make up each cell on the cimbar grid are 8x8 (in a 9x9 grid).
-	* this is because imagehash was on 8x8 tiles!
-	* smaller sizes might also work?
+	* the symbols that make up each cell on the cimbar grid are 8x8 (in a 9x9 grid). this is because imagehash was on 8x8 tiles!
+	* smaller sizes might also work? I've been looking into 5x5 (in a 6x6 grid) as a starting point. It seems promising.
 	* the limiting factor is the hamming distance between each image hash "bucket", and the 9Xth percentile decoding errors.
 * optimal color set?
 	* the 4-color (2 bit) pallettes seem reasonable. 8-color, perhaps less so?
 	* this may be a limitation of the algorithm/approach, however. Notably, since each symbol is drawn with one pallette color, all colors need sufficient contrast against the backdrop (#000 or #FFF, depending). This constrains the color space somewhat, and less distinct colors == more errors.
-	* in addition to contrast, there is interplay (that I don't currently understand) between the overall brightness of the image and the exposure time needed for high framerate capture. More clean frames == more troughput.
+	* in addition to contrast, there is interplay between the overall brightness of the image and the exposure time needed for high framerate capture. More clean frames == more troughput.
+	* the camera framerate in the CFC app is limited by auto-exposure and auto-focus behavior. A newer/better decoder app might be helpful.
 * optimal grid size?
 	* 1024x1024 is a remnant of the early prototyping process. There is nothing inherently special about it (except that it fits on a 1920x1080 screen, which seems good)
 		* the tile grid itself is 1008x1008 (1008 == 9x112 -- there are 112 tile rows and columns)
-	* a smaller grid would be less information dense, but more resilient to errors. Probably.
+	* a smaller grid *could* be more resilient to errors, at the expense of data capacity.
 * optimal grid shape?
 	* it's a square because QR codes are square. That's it. Should it be?
 	* I'm strongly considering 4:3 for the next revision.
@@ -41,6 +42,8 @@ Performance optimizations aside, there are a number of paths that might be inter
 * proper GPU support (OpenCV + openCL) on android?
 	* It *might* be useful. [CFC]((https://github.com/sz3/cfc) is the current test bed for this.
 * wasm decoder?
+	* android is going to kick CFC out of the store! (testing requirement)
+		* so it might be time to write this...
 	* probably needs to use Web Workers
 	* in-browser GPGPU support would be interesting (but I'm not counting on it)
 * ???

WASM.md

@@ -4,16 +4,30 @@
 
 ## Releases
 
-wasm and asm.js releases are available [here](https://github.com/sz3/libcimbar/releases/latest). The wasm build is what cimbar.org uses. The asm.js build can be downloaded, extracted, and run in a local web browser.
+wasm and asm.js releases are available [here](https://github.com/sz3/libcimbar/releases/latest). The wasm build is what cimbar.org uses. [cimbar_js.html](https://github.com/sz3/libcimbar/releases/latest/cimbar_js.html) can be downloaded and opened/run in a local web browser -- no install required.
 
 ## Build
 
-To build opencv.js (and the static libraries we'll need to build against opencv)...
+To build, use the `package-wasm.sh` script in a docker container:
+
+```
+docker run --mount type=bind,source="$(pwd)",target="/usr/src/app" -it emscripten/emsdk:3.1.39
+```
+Then, inside the container:
+```
+bash /usr/src/app/package-wasm.sh
+```
+
+## Alternative build for the adventurous
+
+Alternatively, if you have a local emscripten setup, you can try to run the package-wasm.sh commands piecemeal:
+
+To build opencv.js:
 ```
 cd /path/to/opencv
 mkdir opencv-build-wasm
 cd opencv-build-wasm
-python ../platforms/js/build_js.py build_wasm --build_wasm --emscripten_dir=/path/to/emscripten
+python3 ../platforms/js/build_js.py build_wasm --build_wasm --emscripten_dir=/path/to/emscripten
 ```
 
 With opencv.js built:
@@ -22,7 +36,7 @@ mkdir build-wasm
 cd build-wasm
 source /path/to/emscripten/emsdk/emsdk_env.sh
 emcmake cmake .. -DUSE_WASM=1 -DOPENCV_DIR=/path/to/opencv
-make -j7 install
+make -j5 install
 ```
 
 (do `-DUSE_WASM=2` to use asm.js instead of wasm)

package-wasm.sh

@@ -7,17 +7,20 @@ apt update
 apt install python3 -y
 
 cd opencv4/
-mkdir opencv-build-wasm && cd opencv-build-wasm
+mkdir opencv-build-wasm
+cd opencv-build-wasm
 python3 ../platforms/js/build_js.py build_wasm --build_wasm --emscripten_dir=/emsdk/upstream/emscripten
 
 cd /usr/src/app
-mkdir build-wasm && cd build-wasm
+mkdir build-wasm
+cd build-wasm
 emcmake cmake .. -DUSE_WASM=1 -DOPENCV_DIR=/usr/src/app/opencv4
 make -j5 install
 (cd ../web/ && tar -czvf cimbar.wasm.tar.gz cimbar_js.* index.html main.js)
 
 cd /usr/src/app
-mkdir build-asmjs && cd build-asmjs
+mkdir build-asmjs
+cd build-asmjs
 emcmake cmake .. -DUSE_WASM=2 -DOPENCV_DIR=/usr/src/app/opencv4
 make -j5 install
 (cd ../web/ && zip cimbar.asmjs.zip cimbar_js.js index.html main.js)

src/exe/cimbar/cimbar.cpp

@@ -177,6 +177,7 @@ int main(int argc, char** argv)
 	unsigned compressionLevel = cimbar::Config::compression_level();
 	unsigned ecc = cimbar::Config::ecc_bytes();
 	options.add_options()
+		("n,encode", "Run the encoder!", cxxopts::value<bool>())
 		("i,in", "Encoded pngs/jpgs/etc (for decode), or file to encode", cxxopts::value<vector<string>>())
 		("o,out", "Output file prefix (encoding) or directory (decoding).", cxxopts::value<string>())
 		("c,color-bits", "Color bits. [0-3]", cxxopts::value<int>()->default_value(turbo::str::str(colorBits)))
@@ -185,7 +186,6 @@ int main(int argc, char** argv)
 		("z,compression", "Compression level. 0 == no compression.", cxxopts::value<int>()->default_value(turbo::str::str(compressionLevel)))
 		("color-correct", "Toggle decoding color correction. 2 == full (fountain mode only). 1 == simple. 0 == off.", cxxopts::value<int>()->default_value("2"))
 		("color-correction-file", "Debug -- save color correction matrix generated during fountain decode, or use it for non-fountain decodes", cxxopts::value<string>())
-		("encode", "Run the encoder!", cxxopts::value<bool>())
 		("no-deskew", "Skip the deskew step -- treat input image as already extracted.", cxxopts::value<bool>())
 		("no-fountain", "Disable fountain encode/decode. Will also disable compression.", cxxopts::value<bool>())
 		("undistort", "Attempt undistort step -- useful if image distortion is significant.", cxxopts::value<bool>())

src/lib/cimb_translator/CimbDecoder.cpp

@@ -65,18 +65,23 @@ CimbDecoder::CimbDecoder(unsigned symbol_bits, unsigned color_bits, unsigned col
 	load_tiles();
 }
 
-const color_correction& CimbDecoder::get_ccm() const
+// protected
+color_correction& CimbDecoder::internal_ccm() const
 {
-	// testing purposes only.
-	// this returning a thread local would be fine, iff we only use it for debugging!
+	static thread_local color_correction _ccm;
 	return _ccm;
 }
 
+// public
+const color_correction& CimbDecoder::get_ccm() const
+{
+	// testing/debugging purposes only!!!!
+	return internal_ccm();
+}
+
 void CimbDecoder::update_color_correction(cv::Matx<float, 3, 3>&& ccm)
 {
-	// TODO: threadlocal?
-	// because this is dubious to begin with...
-	_ccm.update(std::move(ccm));
+	internal_ccm().update(std::move(ccm));
 }
 
 uint64_t CimbDecoder::get_tile_hash(unsigned symbol) const
@@ -163,9 +168,9 @@ std::tuple<uchar,uchar,uchar> CimbDecoder::get_color(int i) const
 unsigned CimbDecoder::get_best_color(float r, float g, float b) const
 {
 	// transform color with ccm
-	if (_ccm.active())
+	if (internal_ccm().active())
 	{
-		std::tuple<float, float, float> color = _ccm.transform(r, g, b);
+		std::tuple<float, float, float> color = internal_ccm().transform(r, g, b);
 		r = std::get<0>(color);
 		g = std::get<1>(color);
 		b = std::get<2>(color);

src/lib/cimb_translator/CimbDecoder.h

@@ -31,6 +31,8 @@ class CimbDecoder
 	unsigned symbol_bits() const;
 
 protected:
+	color_correction& internal_ccm() const;
+
 	uint64_t get_tile_hash(unsigned symbol) const;
 	bool load_tiles();
 
@@ -45,5 +47,4 @@ class CimbDecoder
 	unsigned _colorMode;
 	bool _dark;
 	uchar _ahashThreshold;
-	color_correction _ccm;
 };

src/lib/cimb_translator/GridConf.h

@@ -27,8 +27,5 @@ namespace cimbar
 		static constexpr unsigned cell_size = 8;
 		static constexpr unsigned cell_offset = 8;
 		static constexpr unsigned cells_per_col = 112;
-
-		static constexpr int interleave_partitions = 2;
-		static constexpr int fountain_chunks_per_frame = 10;
 	};
 }

src/lib/cimb_translator/test/CimbReaderTest.cpp

@@ -24,7 +24,7 @@ namespace {
 	{
 	public:
 		using CimbDecoder::CimbDecoder;
-		using CimbDecoder::_ccm;
+		using CimbDecoder::internal_ccm;
 	};
 }
 #include "serialize/str_join.h"
@@ -143,6 +143,7 @@ TEST_CASE( "CimbReaderTest/testCCM", "[unit]" )
 	cv::Mat sample = TestCimbar::loadSample("b/ex2434.jpg");
 
 	TestableCimbDecoder decoder(4, 2);
+	decoder.internal_ccm() = color_correction();
 	CimbReader cr(sample, decoder);
 
 	// this is the header value for the sample -- we could imitate what the Decoder does
@@ -151,10 +152,10 @@ TEST_CASE( "CimbReaderTest/testCCM", "[unit]" )
 	cr.update_metadata((char*)md.data(), md.md_size);
 	cr.init_ccm(2, cimbar::Config::interleave_blocks(), cimbar::Config::interleave_partitions(), cimbar::Config::fountain_chunks_per_frame(6, false));
 
-	assertTrue( decoder._ccm.active() );
+	assertTrue( decoder.get_ccm().active() );
 
 	std::stringstream ss;
-	ss << decoder._ccm.mat();
+	ss << decoder.get_ccm().mat();
 	assertEquals("[2.3991191, -0.41846275, -0.54654282;\n "
 				 "-0.42976046, 2.632102, -0.76466882;\n "
 				 "-0.54299992, -0.20199311, 2.2753253]", ss.str());
@@ -176,9 +177,10 @@ TEST_CASE( "CimbReaderTest/testCCM.Disabled", "[unit]" )
 	cv::Mat sample = TestCimbar::loadSample("b/ex2434.jpg");
 
 	TestableCimbDecoder decoder(4, 2);
+	decoder.internal_ccm() = color_correction();
 	CimbReader cr(sample, decoder, false, false);
 
-	assertFalse( decoder._ccm.active() );
+	assertFalse( decoder.get_ccm().active() );
 
 	std::array<unsigned, 6> expectedColors = {0, 1, 1, 2, 2, 2};
 	for (unsigned i = 0; i < expectedColors.size(); ++i)
@@ -197,6 +199,7 @@ TEST_CASE( "CimbReaderTest/testCCM.VeryNecessary", "[unit]" )
 	cv::Mat sample = TestCimbar::loadSample("b/ex380.jpg");
 
 	TestableCimbDecoder decoder(4, 2);
+	decoder.internal_ccm() = color_correction();
 	CimbReader cr(sample, decoder);
 
 	// this is the header value for the sample -- we could imitate what the Decoder does
@@ -205,10 +208,10 @@ TEST_CASE( "CimbReaderTest/testCCM.VeryNecessary", "[unit]" )
 	cr.update_metadata((char*)md.data(), md.md_size);
 	cr.init_ccm(2, cimbar::Config::interleave_blocks(), cimbar::Config::interleave_partitions(), cimbar::Config::fountain_chunks_per_frame(6, false));
 
-	assertTrue( decoder._ccm.active() );
+	assertTrue( decoder.get_ccm().active() );
 
 	std::stringstream ss;
-	ss << decoder._ccm.mat();
+	ss << decoder.get_ccm().mat();
 	assertEquals("[1.6250746, 0.0024788622, -0.45772526;\n "
 				 "-0.29126319, 2.2922182, -0.67037439;\n "
 				 "-1.2192062, -2.7447209, 5.0476217]", ss.str());