diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8681323..fb15c3a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -7,7 +7,7 @@ jobs: fail-fast: false matrix: os: [ubuntu-latest] - python-version: [3.7, 3.8, 3.9, "3.10"] + python-version: [3.8, 3.9, "3.10", "3.11"] steps: - uses: actions/checkout@v2 with: @@ -20,7 +20,7 @@ jobs: - name: install dependencies run: | - pip install -r requirements + pip install -r requirements.freeze pip install --upgrade coveralls - name: test diff --git a/ABOUT.md b/ABOUT.md index 41556d7..0de1044 100644 --- a/ABOUT.md +++ b/ABOUT.md @@ -30,7 +30,7 @@ My conclusion was that if I was going to create a proof-of-concept implementatio Not yet. The project is -- at this point -- just a proof-of-concept. It's not clear to me that cimbar is any better for data density + reliability than HCCB, or better than the myriad of unfinished attempts at color QR codes that are floating around, or (...). It could be a technological dead end. But perhaps with more refinement it might be interesting? -## Notable open questions, concerns, and ideas: +## Unresolved design questions, concerns, and ideas: * the symbol set is not optimal. There are 16, and their image hashes are all reasonable hamming distance from each other, and do *ok* when upscaled... but I drew the initial 40 or so candidates by hand in kolourpaint, and paired down the set to 16 by experimentation. * 32 distinct tiles (5 bits) be possible for 8x8 tiles @@ -52,12 +52,11 @@ Not yet. The project is -- at this point -- just a proof-of-concept. It's not cl * 8-color cimbar (3 color bits) is possible, at least in dark mode * probably light mode as well, but a palette will need to be found * 16-color cimbar does not seem possible with the current color decoding logic - * but with pre-processing for color correction and a perceptual color diff (like CIE76), ... maybe? - * this may be wishful thinking. + * ...at least not in the small (sub-8x8) tile sizes we want to use for high data density * Reed Solomon was chosen for error correction due how ubiquitous its implementations are. * it isn't a perfect fit. Most cimbar errors are 1-3 flipped bits at a time -- Reed Solomon doesn't care if one bit flipped or eight did -- a bad byte is a bad byte. - * Something built on LDPC would likely be better. + * Something that cares about bits and not bytes (LDPC? idk) would likely be better. * the focus on computer screens has surely overlooked problems that come from paper/printed/e-paper surfaces * using a black background ("dark mode") came out of getting better results from backlit screens @@ -65,8 +64,8 @@ Not yet. The project is -- at this point -- just a proof-of-concept. It's not cl * curved surfaces are a can of worms I didn't want to open -- there are some crazy ideas that could be pursued there, but they may not be worth the effort * should cimbar include "metadata" to tell the decoder what it's trying to decode -- ECC level, number of colors, (grid size...?) - * the bottom right corner of the image seems like the logical place for this. - * a slightly smaller 4th anchor pattern could give us 11 tiles (44 bits?) to work with for metadata, which is not a lot, but probably enough. + * the bottom right corner of the image seems like the logical place for this. However, differing aspect ratios may be a problem + * example: on 8x8, a slightly smaller 4th anchor pattern could give us 11 tiles (44 bits?) to work with for metadata, which is not a lot, but probably enough to be useful. ## Would you like to know more? diff --git a/README.md b/README.md index 04af0a2..7a5b98c 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ cimbar is a proof-of-concept 2D data encoding format -- much like [QR Codes](https://en.wikipedia.org/wiki/QR_code), [JAB codes](https://jabcode.org/), and [Microsoft's HCCB](https://en.wikipedia.org/wiki/High_Capacity_Color_Barcode).

- +

## How it works @@ -33,16 +33,16 @@ The main constraints cimbar must deal with are: * all tiles in the tileset must be sufficient hamming distance away from each other, where *sufficient* is determined by whether the decoder can consistently place blurry or otherwise imperfect tiles in the correct "bucket". * all colors in the colorset must be far enough away from each other -- currently as a function of RGB value scaling -- such that color bleeding, reflections, and the like, can be overcome. -In practice, this means that the source image should be around 900x900 resolution or greater, with reasonable color correction handled by the camera -- as you'd find in any modern cell phone. +Cimbar is designed to deal with some lossiness. In practice, the source image should be around 700x700 resolution or greater, in focus, and with *some* color correction handled by the camera -- as you'll hopefully find in any modern cell phone. -This python cimbar implementation is a research project. It works, but it is not very performant, and does not handle error cases with much grace. [libcimbar](https://github.com/sz3/libcimbar), the C++ implementation, has been much more heavily optimized and tested. The target goals of the proof-of-concept were: +This python cimbar implementation is a research project. It works, but it is slow, and does not handle error cases with much grace. [libcimbar](https://github.com/sz3/libcimbar), the C++ implementation, has been much more heavily optimized and tested. The target goals of the proof-of-concept were: 1. achieve data density on the order of _10kb_ per image. 2. validate a theoretical performance (and if possible, an implemented demonstration) of >= _100kb/s_ data transfer (800 kilobits/second) from a computer screen to a cell phone, using only animated cimbar codes and the cell phone camera. ## I want numbers! -* a 6-bit cimbar image contains `9300` raw bytes of data, and `7500` bytes with the default error correction level (30) -* for 7-bit cimbar, the respective numbers are `10850` and `8750` +* a `mode B` (8x8, 4-color, 30/155 ecc, 6-bits-per-tile) cimbar image contains `9300` raw bytes of data, and `7500` bytes with the default error correction level (30) +* for the old `mode 8C` (8x8, 8-color, 7-bit) cimbar, the respective numbers are `10850` and `8750` * error correction level is `N/155`. So `ecc=30` corresponds to a `30:125` ratio of error correction bytes to "real" bytes. * error correction is (for now) done via Reed Solomon, which contibutes to the rather large ratio of error correction bytes. See [ABOUT](ABOUT.md) for more technical discussion. diff --git a/cimbar/cimbar.py b/cimbar/cimbar.py index 99aafa5..77e79f8 100644 --- a/cimbar/cimbar.py +++ b/cimbar/cimbar.py @@ -5,9 +5,9 @@ Usage: ./cimbar.py ... --output= [--config=] [--dark | --light] [--colorbits=<0-3>] [--deskew=<0-2>] [--ecc=<0-200>] - [--fountain] [--preprocess=<0,1>] [--color-correct] + [--fountain] [--preprocess=<0,1>] [--color-correct=<0-2>] ./cimbar.py --encode ( | --src_data=) ( | --output=) - [--config=] [--dark | --light] + [--config=] [--dark | --light] [--colorbits=<0-3>] [--ecc=<0-150>] [--fountain] ./cimbar.py (-h | --help) @@ -26,10 +26,12 @@ --config= Choose configuration from sq8x8,sq5x5,sq5x6. [default: sq8x8] --dark Use dark palette. [default] --light Use light palette. - --color-correct Attempt color correction. + --color-correct=<0-7> Color correction. 0 is off. 1 is white balance. 3 is 2-pass on a fountain-encoded image. [default: 1] --deskew=<0-2> Deskew level. 0 is no deskew. Should usually be 0 or default. [default: 1] --preprocess=<0,1> Sharpen image before decoding. Default is to guess. [default: -1] """ +from collections import defaultdict +from io import BytesIO from os import path from tempfile import TemporaryDirectory @@ -41,8 +43,9 @@ from cimbar import conf from cimbar.deskew.deskewer import deskewer from cimbar.encode.cell_positions import cell_positions, AdjacentCellFinder, FloodDecodeOrder -from cimbar.encode.cimb_translator import CimbEncoder, CimbDecoder, avg_color +from cimbar.encode.cimb_translator import CimbEncoder, CimbDecoder, avg_color, possible_colors from cimbar.encode.rss import reed_solomon_stream +from cimbar.fountain.header import fountain_header from cimbar.util.bit_file import bit_file from cimbar.util.interleave import interleave, interleave_reverse, interleaved_writer @@ -62,23 +65,141 @@ def bits_per_op(): return conf.BITS_PER_SYMBOL + BITS_PER_COLOR +def use_split_mode(): + return getattr(conf, 'SPLIT_MODE', True) + + def num_cells(): return conf.CELL_DIM_Y*conf.CELL_DIM_X - (conf.MARKER_SIZE_X*conf.MARKER_SIZE_Y * 4) +def num_fountain_blocks(): + return bits_per_op() * 2 + + def capacity(bits_per_op=bits_per_op()): return num_cells() * bits_per_op // 8; def _fountain_chunk_size(ecc=conf.ECC, bits_per_op=bits_per_op(), fountain_blocks=conf.FOUNTAIN_BLOCKS): + fountain_blocks = fountain_blocks or num_fountain_blocks() return capacity(bits_per_op) * (conf.ECC_BLOCK_SIZE-ecc) // conf.ECC_BLOCK_SIZE // fountain_blocks +def _get_expected_fountain_headers(headers, bits_per_symbol=conf.BITS_PER_SYMBOL, bits_per_color=BITS_PER_COLOR): + import bitstring + from bitstring import Bits, BitStream + + # it'd be nice to use the frame id as well, but sometimes we skip frames. + # specifically, at the end of the input data (so when num_cunks*chunk size ~= file size) + # we will usually skip a frame (whenever the last chunk is not conveniently equal to the frame size) + # we *could* do that math and use the frame id anyway, it might be worth it... + for header in headers: + if not header.bad(): + break + assert not header.bad() # TODO: maybe just return NULL? + + color_headers = [] + for _ in range(bits_per_color * 2): + color_headers += bytes(header)[:-2] # remove frame id + + print(color_headers) + + res = [] + stream = BitStream() + stream.append(Bits(bytes=color_headers)) + while stream.pos < stream.length: + res.append(stream.read(f'uint:{bits_per_color}')) + return res + + +def _get_fountain_header_cell_index(cells, expected_vals): + # TODO: misleading to say this works for all FOUNTAIN_BLOCKS values... + fountain_blocks = conf.FOUNTAIN_BLOCKS or num_fountain_blocks() + end = capacity(BITS_PER_COLOR) * 8 // BITS_PER_COLOR + header_start_interval = capacity(bits_per_op()) * 8 // fountain_blocks // BITS_PER_COLOR + header_len = (fountain_header.length-2) * 8 // BITS_PER_COLOR + + cell_idx = [] + i = 0 + while i < end: + # maybe split this into a list of lists? idk + cell_idx += list(range(i, i+header_len)) + i += header_start_interval + + # sanity check, we're doomed if this fails + assert len(cell_idx) == len(expected_vals), f'{len(cell_idx)} == {len(expected_vals)}' + res = defaultdict(list) + for idx,exp in zip(cell_idx, expected_vals): + res[exp].append(cells[idx]) + return res + + +def _build_color_decode_lookups(ct, color_img, color_map): + res = defaultdict(list) + for exp, pos_list in color_map.items(): + for pos in pos_list: + cell = _crop_cell(color_img, pos[0], pos[1]) + color = avg_color(cell, dark=ct.dark) + res[exp].append(color) + bits = ct.decode_color(cell, 0) + if bits != exp: + print(f' wrong!!! {pos} ... {bits} == {exp}') + + # return averages + return { + k: tuple(numpy.mean(vals, axis=0)) for k,vals in res.items() + } + + +def _decode_sector_calc(midpt, x, y, num_sectors): + if num_sectors < 2: + return 0 + if (x - midpt[0])**2 + (y - midpt[1])**2 < 400**2: + return 0 + else: + return 1 + + +def _derive_color_lookups(ct, color_img, cells, fount_headers, splits=0): + header_cell_locs = _get_fountain_header_cell_index( + list(interleave(cells, conf.INTERLEAVE_BLOCKS, conf.INTERLEAVE_PARTITIONS)), + _get_expected_fountain_headers(fount_headers), + ) + print(header_cell_locs) + + color_maps = [] + if splits == 2: + center_map = defaultdict(list) + edge_map = defaultdict(list) + midX = conf.TOTAL_SIZE // 2 + midY = conf.TOTAL_SIZE // 2 + for exp,pos in header_cell_locs.items(): + for xy in pos: + if _decode_sector_calc((midX, midY), *xy, splits) == 0: + center_map[exp].append(xy) + else: + edge_map[exp].append(xy) + + lc = {exp: len(pos) for exp, pos in center_map.items()} + le = {exp: len(pos) for exp, pos in edge_map.items()} + print(f'sanity check. len(center)={lc}, len(edge)={le}') + color_maps = [center_map, edge_map] + + else: + color_map = dict() + for exp,pos in header_cell_locs.items(): + color_map[exp] = pos + color_maps = [color_map] + + return [_build_color_decode_lookups(ct, color_img, cm) for cm in color_maps] + + def detect_and_deskew(src_image, temp_image, dark, auto_dewarp=False): return deskewer(src_image, temp_image, dark, auto_dewarp=auto_dewarp) -def _decode_cell(ct, img, color_img, x, y, drift): +def _decode_cell(ct, img, x, y, drift): best_distance = 1000 for dx, dy in drift.pairs: testX = x + drift.x + dx @@ -95,8 +216,8 @@ def _decode_cell(ct, img, color_img, x, y, drift): testX = x + drift.x + best_dx testY = y + drift.y + best_dy - best_cell = color_img.crop((testX+1, testY+1, testX + conf.CELL_SIZE-2, testY + conf.CELL_SIZE-2)) - return best_bits + ct.decode_color(best_cell), best_dx, best_dy, best_distance + best_cell = (testX, testY) + return best_bits, best_cell, best_dx, best_dy, best_distance def _preprocess_for_decode(img): @@ -118,7 +239,10 @@ def _get_decoder_stream(outfile, ecc, fountain): decompressor = zstd.ZstdDecompressor().stream_writer(f) f = fountain_decoder_stream(decompressor, _fountain_chunk_size(ecc)) on_rss_failure = b'' if fountain else None - return reed_solomon_stream(f, ecc, conf.ECC_BLOCK_SIZE, mode='write', on_failure=on_rss_failure) if ecc else f + + stream = reed_solomon_stream(f, ecc, conf.ECC_BLOCK_SIZE, mode='write', on_failure=on_rss_failure) if ecc else f + fount = f if fountain else None + return stream, fount def compute_tint(img, dark): @@ -135,31 +259,119 @@ def update(c, r, g, b): else: pos = [(67, 0), (0, 67), (conf.TOTAL_SIZE-79, 0), (0, conf.TOTAL_SIZE-79)] + colors = [] for x, y in pos: iblock = img.crop((x, y, x + 4, y + 4)) - r, g, b = avg_color(iblock) - update(cc, *avg_color(iblock)) + ac = avg_color(iblock, False) + colors.append(ac) + update(cc, *ac) print(f'tint is {cc}') return cc['r'], cc['g'], cc['b'] -def _decode_iter(ct, img, color_img): +def _crop_cell(img, x, y): + return img.crop((x+1, y+1, x + conf.CELL_SIZE-1, y + conf.CELL_SIZE-1)) + + +def _decode_symbols(ct, img): cell_pos, num_edge_cells = cell_positions(conf.CELL_SPACING_X, conf.CELL_SPACING_Y, conf.CELL_DIM_X, conf.CELL_DIM_Y, conf.CELLS_OFFSET, conf.MARKER_SIZE_X, conf.MARKER_SIZE_Y) finder = AdjacentCellFinder(cell_pos, num_edge_cells, conf.CELL_DIM_X, conf.MARKER_SIZE_X) decode_order = FloodDecodeOrder(cell_pos, finder) + print('beginning decode symbols pass...') for i, (x, y), drift in decode_order: - best_bits, best_dx, best_dy, best_distance = _decode_cell(ct, img, color_img, x, y, drift) + best_bits, best_cell, best_dx, best_dy, best_distance = _decode_cell(ct, img, x, y, drift) decode_order.update(best_dx, best_dy, best_distance) - yield i, best_bits + yield i, best_bits, best_cell + + +def _calc_ccm(ct, color_lookups, cc_setting, state_info): + splits = 2 if cc_setting in (6, 7) else 0 + if cc_setting in (3, 4, 5): + possible = possible_colors(ct.dark, BITS_PER_COLOR) + if len(color_lookups[0]) < len(possible): + raise Exception("kaboomski") # not clear whether this should throw or not, really. + exp = [color for i,color in enumerate(possible) if i in color_lookups[0]] + [(255,255,255)] + exp = numpy.array(exp) + white = state_info['white'] + observed = numpy.array([v for k,v in sorted(color_lookups[0].items())] + [white]) + from colour.characterisation.correction import matrix_colour_correction_Cheung2004 + der = matrix_colour_correction_Cheung2004(observed, exp) + + # not sure which of this would be better... + if ct.ccm is None or cc_setting == 4: + ct.ccm = der + else: # cc_setting == 3,5 + ct.ccm = der.dot(ct.ccm) + + if splits: # 6,7 + from colour.characterisation.correction import matrix_colour_correction_Cheung2004 + exp = numpy.array(possible_colors(ct.dark, BITS_PER_COLOR) + [(255,255,255)]) + white = state_info['white'] + ccms = list() + i = 0 + while i < splits: + observed = numpy.array([v for k,v in sorted(color_lookups[i].items())] + [white]) + der = matrix_colour_correction_Cheung2004(observed, exp) + ccms.append(der) + i += 1 + + if ct.ccm is None or cc_setting == 7: + ct.ccm = ccms + else: + ct.ccm = [der.dot(ct.ccm) for der in ccms] + + if cc_setting == 5: + ct.colors = color_lookups[0] + if cc_setting == 10: + ct.disable_color_scaling = True + ct.colors = color_lookups[0] + + +def _decode_iter(ct, img, color_img, state_info={}): + decoding = sorted(_decode_symbols(ct, img)) + if use_split_mode(): + for i, bits, _ in decoding: + yield i, bits + yield -1, None + + # state_info can be set at any time, but it will probably be set by the caller *after* the empty yield above + if state_info.get('color_correct') == 1: + white = state_info['white'] + from colormath.chromatic_adaptation import _get_adaptation_matrix + ct.ccm = _get_adaptation_matrix(numpy.array([*white]), + numpy.array([255, 255, 255]), 2, 'von_kries') + + if state_info.get('headers'): + cc_setting = state_info['color_correct'] + splits = 2 if cc_setting in (6, 7) else 0 + + cells = [cell for _, __, cell in decoding] + color_lookups = _derive_color_lookups(ct, color_img, cells, state_info.get('headers'), splits) + print('color lookups:') + print(color_lookups) + + _calc_ccm(ct, color_lookups, cc_setting, state_info) + + print('beginning decode colors pass...') + midX = conf.TOTAL_SIZE // 2 + midY = conf.TOTAL_SIZE // 2 + for i, bits, cell in decoding: + testX, testY = cell + best_cell = _crop_cell(color_img, testX, testY) + decode_sector = 0 if ct.ccm is None else _decode_sector_calc((midX, midY), testX, testY, len(ct.ccm)) + if use_split_mode(): + yield i, ct.decode_color(best_cell, 0) + else: + yield i, bits + (ct.decode_color(best_cell, 0) << conf.BITS_PER_SYMBOL) -def decode_iter(src_image, dark, should_preprocess, should_color_correct, deskew, auto_dewarp): +def decode_iter(src_image, dark, should_preprocess, color_correct, deskew, auto_dewarp, state_info={}): tempdir = None if deskew: tempdir = TemporaryDirectory() - temp_img = path.join(tempdir.name, path.basename(src_image)) + temp_img = path.join(tempdir.name, path.basename(src_image)) # or /tmp dims = detect_and_deskew(src_image, temp_img, dark, auto_dewarp) if should_preprocess < 0: should_preprocess = dims[0] < conf.TOTAL_SIZE or dims[1] < conf.TOTAL_SIZE @@ -170,12 +382,12 @@ def decode_iter(src_image, dark, should_preprocess, should_color_correct, deskew ct = CimbDecoder(dark, symbol_bits=conf.BITS_PER_SYMBOL, color_bits=conf.BITS_PER_COLOR) img = _preprocess_for_decode(color_img) if should_preprocess else color_img - if should_color_correct: - from colormath.chromatic_adaptation import _get_adaptation_matrix - ct.ccm = _get_adaptation_matrix(numpy.array([*compute_tint(color_img, dark)]), - numpy.array([255, 255, 255]), 2, 'von_kries') + if color_correct: + white = compute_tint(color_img, dark) + state_info['white'] = white + state_info['color_correct'] = color_correct - yield from _decode_iter(ct, img, color_img) + yield from _decode_iter(ct, img, color_img, state_info) if tempdir: # cleanup with tempdir: @@ -187,15 +399,52 @@ def decode(src_images, outfile, dark=False, ecc=conf.ECC, fountain=False, force_ cells, _ = cell_positions(conf.CELL_SPACING_X, conf.CELL_SPACING_Y, conf.CELL_DIM_X, conf.CELL_DIM_Y, conf.CELLS_OFFSET, conf.MARKER_SIZE_X, conf.MARKER_SIZE_Y) interleave_lookup, block_size = interleave_reverse(cells, conf.INTERLEAVE_BLOCKS, conf.INTERLEAVE_PARTITIONS) - dstream = _get_decoder_stream(outfile, ecc, fountain) + dstream, fount = _get_decoder_stream(outfile, ecc, fountain) + dupe_stream = dupe_pass = None + if color_correct >= 3 and not fount: + dupe_stream, fount = _get_decoder_stream('/dev/null', ecc, True) with dstream as outstream: for imgf in src_images: - with interleaved_writer(f=outstream, bits_per_op=bits_per_op(), mode='write', keep_open=True) as iw: - decoding = {i: bits for i, bits in decode_iter(imgf, dark, force_preprocess, color_correct, deskew, - auto_dewarp)} - for i, bits in sorted(decoding.items()): - block = interleave_lookup[i] // block_size - iw.write(bits, block) + if use_split_mode(): + first_pass = interleaved_writer( + f=outstream, bits_per_op=conf.BITS_PER_SYMBOL, mode='write', keep_open=True + ) + if dupe_stream: + dupe_pass = interleaved_writer( + f=dupe_stream, bits_per_op=conf.BITS_PER_SYMBOL, mode='write', keep_open=True + ) + second_pass = interleaved_writer( + f=outstream, bits_per_op=BITS_PER_COLOR, mode='write', keep_open=True + ) + else: + first_pass = interleaved_writer(f=outstream, bits_per_op=bits_per_op(), mode='write', keep_open=True) + second_pass = None + + # this is a bit goofy, might refactor it to have less "loop through writers" weirdness + iw = first_pass + state_info = {} + for i, bits in decode_iter( + imgf, dark, force_preprocess, color_correct, deskew, auto_dewarp, state_info + ): + if i == -1: + # flush and move to the second writer + with iw: + pass + if dupe_pass: + with dupe_pass: + pass + iw = second_pass + if fount: + state_info['headers'] = fount.headers + continue + block = interleave_lookup[i] // block_size + iw.write(bits, block) + if dupe_pass: + dupe_pass.write(bits, block) + + # flush iw + with iw: + pass def _get_image_template(width, dark): @@ -225,7 +474,7 @@ def _get_image_template(width, dark): return img -def _get_encoder_stream(src, ecc, fountain, compression_level=6): +def _get_encoder_stream(src, ecc, fountain, compression_level=16): # various checks to set up the instream. # the hierarchy is raw bytes -> zstd -> fountain -> reedsolomon -> image f = open(src, 'rb') @@ -253,10 +502,28 @@ def encode_iter(src_data, ecc, fountain): cells, _ = cell_positions(conf.CELL_SPACING_X, conf.CELL_SPACING_Y, conf.CELL_DIM_X, conf.CELL_DIM_Y, conf.CELLS_OFFSET, conf.MARKER_SIZE_X, conf.MARKER_SIZE_Y) assert len(cells) == num_cells() - for x, y in interleave(cells, conf.INTERLEAVE_BLOCKS, conf.INTERLEAVE_PARTITIONS): - bits = f.read() - yield bits, x, y, frame_num + + if use_split_mode(): + symbols = [] + for x, y in interleave(cells, conf.INTERLEAVE_BLOCKS, conf.INTERLEAVE_PARTITIONS): + bits = f.read(conf.BITS_PER_SYMBOL) + symbols.append(bits) + + # there are better ways to do this than reverse+pop... + # the important part is that it's a 2-pass approach + symbols.reverse() + + for x, y in interleave(cells, conf.INTERLEAVE_BLOCKS, conf.INTERLEAVE_PARTITIONS): + bits = symbols.pop() | (f.read(BITS_PER_COLOR) << conf.BITS_PER_SYMBOL) + yield bits, x, y, frame_num + + else: + for x, y in interleave(cells, conf.INTERLEAVE_BLOCKS, conf.INTERLEAVE_PARTITIONS): + bits = f.read() + yield bits, x, y, frame_num + frame_num += 1 + print(f'encoded {frame_num} frames') def encode(src_data, dst_image, dark=False, ecc=conf.ECC, fountain=False): @@ -280,7 +547,7 @@ def save_frame(img, frame): def main(): - args = docopt(__doc__, version='cimbar 0.5.13') + args = docopt(__doc__, version='cimbar 0.6.0') global BITS_PER_COLOR BITS_PER_COLOR = int(args.get('--colorbits')) @@ -304,7 +571,7 @@ def main(): deskew = get_deskew_params(args.get('--deskew')) should_preprocess = int(args.get('--preprocess')) - color_correct = args['--color-correct'] + color_correct = int(args.get('--color-correct')) src_images = args[''] dst_data = args[''] or args['--output'] decode(src_images, dst_data, dark, ecc, fountain, should_preprocess, color_correct, **deskew) @@ -313,3 +580,4 @@ def main(): if __name__ == '__main__': main() + diff --git a/cimbar/conf.py b/cimbar/conf.py index 9db45ed..b1ae384 100644 --- a/cimbar/conf.py +++ b/cimbar/conf.py @@ -1,7 +1,7 @@ import sys -class sq8x8: +class og8x8: TOTAL_SIZE = 1024 BITS_PER_SYMBOL = 4 BITS_PER_COLOR = 2 @@ -19,6 +19,27 @@ class sq8x8: INTERLEAVE_BLOCKS = ECC_BLOCK_SIZE MARKER_SIZE_X = round(54 / CELL_SPACING_X) MARKER_SIZE_Y = round(54 / CELL_SPACING_Y) # 6 or 9, probably + SPLIT_MODE=False # legacy + + +class sq8x8: + TOTAL_SIZE = 1024 + BITS_PER_SYMBOL = 4 + BITS_PER_COLOR = 2 + CELL_SIZE = 8 + CELL_SPACING_X = CELL_SIZE + 1 + CELL_DIM_X = 112 + CELLS_OFFSET = 8 + ECC = 30 + ECC_BLOCK_SIZE = 155 + INTERLEAVE_PARTITIONS = 2 + FOUNTAIN_BLOCKS = 0 # dynamic + + CELL_DIM_Y = CELL_DIM_X + CELL_SPACING_Y = CELL_SPACING_X + INTERLEAVE_BLOCKS = ECC_BLOCK_SIZE + MARKER_SIZE_X = round(54 / CELL_SPACING_X) + MARKER_SIZE_Y = round(54 / CELL_SPACING_Y) # 6 or 9, probably class sq5x5: @@ -29,10 +50,10 @@ class sq5x5: CELL_SPACING_X = CELL_SIZE + 1 CELL_DIM_X = 162 CELLS_OFFSET = 9 - ECC = 40 - ECC_BLOCK_SIZE = 216 + ECC = 40 # 32? + ECC_BLOCK_SIZE = 216 # 162? INTERLEAVE_PARTITIONS = 2 - FOUNTAIN_BLOCKS = 10 + FOUNTAIN_BLOCKS = 0 # dynamic CELL_DIM_Y = CELL_DIM_X CELL_SPACING_Y = CELL_SPACING_X @@ -42,19 +63,61 @@ class sq5x5: class sq5x6: - TOTAL_SIZE = 1006 + TOTAL_SIZE = 966 + BITS_PER_SYMBOL = 2 + BITS_PER_COLOR = 2 + CELL_SIZE = 5 + CELL_SPACING_X = CELL_SIZE + CELL_SPACING_Y = CELL_SIZE + 1 + CELL_DIM_Y = 158 + CELL_DIM_X = 190 + CELLS_OFFSET = 9 + ECC = 31 + ECC_BLOCK_SIZE = 161 + INTERLEAVE_PARTITIONS = 23 # or just leave ecc locked, and do 2,10? + FOUNTAIN_BLOCKS = 23 + + INTERLEAVE_BLOCKS = ECC_BLOCK_SIZE + MARKER_SIZE_X = round(54 / CELL_SPACING_X) + MARKER_SIZE_Y = round(54 / CELL_SPACING_Y) # 6 or 9, probably + + +# this one is very flexible, probably good for experimenting with +class sq5x6alt: + TOTAL_SIZE = 958 + BITS_PER_SYMBOL = 2 + BITS_PER_COLOR = 2 + CELL_SIZE = 5 + CELL_SPACING_X = CELL_SIZE + CELL_SPACING_Y = CELL_SIZE + 1 + CELL_DIM_Y = 157 + CELL_DIM_X = 188 + CELLS_OFFSET = 9 + ECC = 35 + ECC_BLOCK_SIZE = 182 + INTERLEAVE_PARTITIONS = 2 + FOUNTAIN_BLOCKS = 0 # dynamic + + INTERLEAVE_BLOCKS = ECC_BLOCK_SIZE + MARKER_SIZE_X = round(54 / CELL_SPACING_X) + MARKER_SIZE_Y = round(54 / CELL_SPACING_Y) # 6 or 9, probably + + +# swing for the fences +class sq5x6beeg: + TOTAL_SIZE = 1051 BITS_PER_SYMBOL = 2 BITS_PER_COLOR = 2 CELL_SIZE = 5 CELL_SPACING_X = CELL_SIZE CELL_SPACING_Y = CELL_SIZE + 1 - CELL_DIM_Y = 165 - CELL_DIM_X = 198 + CELL_DIM_Y = 172 + CELL_DIM_X = 207 CELLS_OFFSET = 9 ECC = 33 ECC_BLOCK_SIZE = 163 INTERLEAVE_PARTITIONS = 3 - FOUNTAIN_BLOCKS = 9 + FOUNTAIN_BLOCKS = 0 # dynamic INTERLEAVE_BLOCKS = ECC_BLOCK_SIZE MARKER_SIZE_X = round(54 / CELL_SPACING_X) diff --git a/cimbar/encode/cimb_translator.py b/cimbar/encode/cimb_translator.py index 21dee9c..0fd43e6 100644 --- a/cimbar/encode/cimb_translator.py +++ b/cimbar/encode/cimb_translator.py @@ -6,6 +6,7 @@ CIMBAR_ROOT = path.abspath(path.join(path.dirname(path.realpath(__file__)), '..', '..')) +DEFAULT_COLOR_CORRECT = {'r_min': 0, 'r_max': 255.0, 'g_min': 0, 'g_max': 255.0, 'b_min': 0, 'b_max': 255.0} def possible_colors(dark, bits=0): @@ -22,10 +23,10 @@ def possible_colors(dark, bits=0): ] elif dark and bits < 3: colors = [ + (0, 0xFF, 0), (0, 0xFF, 0xFF), (0xFF, 0xFF, 0), (0xFF, 0, 0xFF), - (0, 0xFF, 0), ] else: # dark and bits == 3 (>=??) colors = [ @@ -42,7 +43,7 @@ def possible_colors(dark, bits=0): (0x7F, 0xFF, 0), # lime green ... greens tend to look way too similar, and may not be reliable (0, 0xFF, 0x7F), # sea green or something ] - return colors + return colors[:2**bits] def load_tile(name, dark, replacements={}): @@ -61,13 +62,19 @@ def load_tile(name, dark, replacements={}): return img -def avg_color(img): +def avg_color(img, dark): nim = numpy.array(img) w,h,d = nim.shape nim.shape = (w*h, d) return tuple(nim.mean(axis=0)) +def simple_color_scale(r, g, b): + m = max(r, g, b, 1) + scale = 255 / m + return r * scale, g * scale, b * scale + + def relative_color(c): r, g, b = c rg = r - g @@ -83,15 +90,19 @@ def relative_color_diff(c1, c2): class CimbDecoder: - def __init__(self, dark, symbol_bits, color_bits=0, ccm=None): + def __init__(self, dark, symbol_bits, color_bits=0, color_correct=DEFAULT_COLOR_CORRECT, ccm=None): self.dark = dark self.symbol_bits = symbol_bits self.hashes = {} + self.color_correct = color_correct self.ccm = ccm + self.disable_color_scaling = False all_colors = possible_colors(dark, color_bits) self.colors = {c: all_colors[c] for c in range(2 ** color_bits)} + self.color_metrics = [] + self.color_clusters = None for i in range(2 ** symbol_bits): name = path.join(CIMBAR_ROOT, 'bitmap', f'{symbol_bits}', f'{i:02x}.png') @@ -121,30 +132,26 @@ def _check_color(self, c, d): #return (c[0] - d[0])**2 + (c[1] - d[1])**2 + (c[2] - d[2])**2 return relative_color_diff(c, d) - def _scale_color(self, c, adjust, down): + def _scale_adjust(self, c, adjust, down): c = int((c - down) * adjust) - if c > (245 - down): + if c > 245: c = 255 + if c < 0: + c = 0 return c - def _correct_all_colors(self, r, g, b): - if self.ccm is not None: - r, g, b = self.ccm.dot(numpy.array([r, g, b])) - return r, g, b - - def _best_color(self, r, g, b): - r, g, b = self._correct_all_colors(r, g, b) - - # probably some scaling will be good. + def scale_color(self, r, g, b): + if self.disable_color_scaling: + return r, g, b if self.dark: max_val = max(r, g, b, 1) - min_val = min(r, g, b, 48) + min_val = min(r, g, b, max_val-100) if min_val >= max_val: min_val = 0 adjust = 255.0 / (max_val - min_val) - r = self._scale_color(r, adjust, min_val) - g = self._scale_color(g, adjust, min_val) - b = self._scale_color(b, adjust, min_val) + r = self._scale_adjust(r, adjust, min_val) + g = self._scale_adjust(g, adjust, min_val) + b = self._scale_adjust(b, adjust, min_val) else: min_val = min(r, g, b) max_val = max(r, g, b, 1) @@ -152,9 +159,37 @@ def _best_color(self, r, g, b): r = g = b = 0 else: adjust = 255.0 / (max_val - min_val) - r = self._scale_color(r, adjust, min_val) - g = self._scale_color(g, adjust, min_val) - b = self._scale_color(b, adjust, min_val) + r = self._scale_adjust(r, adjust, min_val) + g = self._scale_adjust(g, adjust, min_val) + b = self._scale_adjust(b, adjust, min_val) + return r, g, b + + def _correct_all_colors(self, r, g, b, sector): + if isinstance(self.ccm, list): + ccm = self.ccm[sector] + else: + ccm = self.ccm + if ccm is not None: + r, g, b = ccm.dot(numpy.array([r, g, b])) + return r, g, b + + def _update_metrics(self, i, c, color_in): + stats = self.color_metrics[i] + real_distance = self._check_color(c, color_in) + if real_distance < stats[0]: + self.color_metrics[i] = (real_distance, color_in) + + def best_color(self, r, g, b, sector): + r, g, b = self._correct_all_colors(r, g, b, sector) + #print(f'{r} {g} {b}') + + # probably some scaling will be good. + r, g, b = self.scale_color(r, g, b) + + color_in = (r, g, b) + self.color_metrics.append(color_in) + if self.color_clusters: + return self.color_clusters.categorize(color_in) best_fit = 0 best_distance = 1000000 @@ -168,13 +203,13 @@ def _best_color(self, r, g, b): # break return best_fit - def decode_color(self, img_cell): + def decode_color(self, img_cell, sector): if len(self.colors) <= 1: return 0 - r, g, b = avg_color(img_cell) - bits = self._best_color(r, g, b) - return bits << self.symbol_bits + r, g, b = avg_color(img_cell, self.dark) + # count colors? + return self.best_color(r, g, b, sector) class CimbEncoder: diff --git a/cimbar/fountain/fountain_decoder_stream.py b/cimbar/fountain/fountain_decoder_stream.py index c233d4c..ee40f71 100644 --- a/cimbar/fountain/fountain_decoder_stream.py +++ b/cimbar/fountain/fountain_decoder_stream.py @@ -11,6 +11,7 @@ def __init__(self, f, chunk_size): self.fountain = None self.buffer = b'' self.done = False + self.headers = [] @property def closed(self): @@ -43,6 +44,12 @@ def write(self, buffer): # get chunk_id and total_size from header hdr = fountain_header(buffer[0:fountain_header.length]) + self.headers.append(hdr) + # sanity check/fail if hdr is bad? Will be all 0s if decode failed... + if hdr.bad(): + print('failed fountain decode! ...move along') + return False + if not self.fountain: self._reset(hdr.total_size) diff --git a/cimbar/fountain/fountain_encoder_stream.py b/cimbar/fountain/fountain_encoder_stream.py index 5f8dadb..4249492 100644 --- a/cimbar/fountain/fountain_encoder_stream.py +++ b/cimbar/fountain/fountain_encoder_stream.py @@ -2,7 +2,7 @@ class fountain_encoder_stream: - def __init__(self, f, chunk_size, encode_id=0): + def __init__(self, f, chunk_size, encode_id=108): self.buffer = b'' self.read_size = chunk_size self.chunk_size = chunk_size - fountain_header.length diff --git a/cimbar/fountain/header.py b/cimbar/fountain/header.py index 2e6378e..5a84f3a 100644 --- a/cimbar/fountain/header.py +++ b/cimbar/fountain/header.py @@ -32,3 +32,7 @@ def from_encoded(cls, encoded_bytes): total_size = total_size | ((encode_id & 0x80) << 17) encode_id = encode_id & 0x7F return encode_id, total_size, chunk_id + + def bad(self): + return self.encode_id == 0 and self.total_size == 0 and self.chunk_id + diff --git a/cimbar/grader.py b/cimbar/grader.py index e65cfa0..a26ff1e 100644 --- a/cimbar/grader.py +++ b/cimbar/grader.py @@ -22,8 +22,7 @@ from docopt import docopt -from cimbar.cimbar import bits_per_op as bpo -from cimbar.conf import BITS_PER_SYMBOL +from cimbar.conf import BITS_PER_SYMBOL, BITS_PER_COLOR from cimbar.util.bit_file import bit_file @@ -99,6 +98,26 @@ def grade(self, expected_bits, actual_bits): self.mismatch_by_symbol[actual_symbols] += symbol_err self.mismatch_by_color[actual_color] += color_err + def grade_symbol(self, expected_bits, actual_bits): + err = bin(expected_bits ^ actual_bits).count('1') + if err: + self.error_bits += err + self.error_tiles += 1 + + self.symbol_error_bits += err + self.errors_by_symbol[expected_bits] += err + self.mismatch_by_symbol[actual_bits] += err + + def grade_color(self, expected_bits, actual_bits): + err = bin(expected_bits ^ actual_bits).count('1') + if err: + self.error_bits += err + self.error_tiles += 1 + + self.color_error_bits += err + self.errors_by_color[expected_bits] += err + self.mismatch_by_color[actual_bits] += err + def print_report(self): _print_sorted(self.errors_by_symbol) _print_sorted(self.errors_by_color) @@ -114,7 +133,7 @@ def print_report(self): print(f'color error bits: {self.color_error_bits}') -def evaluate(src_file, dst_file, bits_per_op, dark): +def evaluate_interleaved(src_file, dst_file, bits_per_op): g = Grader() total_bits = getsize(src_file) * 8 @@ -131,14 +150,41 @@ def evaluate(src_file, dst_file, bits_per_op, dark): return g.error_bits +def evaluate_split(src_file, dst_file, bits_per_symbol, bits_per_color): + g = Grader() + + bits_per_op = bits_per_symbol + bits_per_color + total_bits = getsize(src_file) * 8 + symbol_bits = total_bits * bits_per_symbol // bits_per_op + + i = 0 + with bit_file(src_file, bits_per_symbol) as sf, bit_file(dst_file, bits_per_symbol) as df: + + while i < symbol_bits: + expected_bits = sf.read(bits_per_symbol) + actual_bits = df.read(bits_per_symbol) + g.grade_symbol(expected_bits, actual_bits) + i += bits_per_symbol + + while i < total_bits: + expected_bits = sf.read(bits_per_color) + actual_bits = df.read(bits_per_color) + g.grade_color(expected_bits, actual_bits) + i += bits_per_color + + g.print_report() + print(f'total bits: {total_bits}') + return g.error_bits + + def main(): - args = docopt(__doc__, version='cimbar fitness check 0.0.1') + args = docopt(__doc__, version='cimbar fitness check 0.1') src_file = args[''] dst_file = args[''] - dark = args.get('--dark') - bits_per_op = int(args.get('--bits-per-op') or bpo()) - evaluate(src_file, dst_file, bits_per_op, dark) + bits_per_symbol = int(args.get('--bits-per-symbol') or BITS_PER_SYMBOL) + bits_per_color = int(args.get('--bits-per-color') or BITS_PER_COLOR) + evaluate_split(src_file, dst_file, bits_per_symbol, bits_per_color) if __name__ == '__main__': diff --git a/cimbar/util/bit_file.py b/cimbar/util/bit_file.py index 5e9da59..733ca0a 100644 --- a/cimbar/util/bit_file.py +++ b/cimbar/util/bit_file.py @@ -41,14 +41,15 @@ def write(self, bits): bits = Bits(uint=bits, length=self.bits_per_op) self.stream.append(bits) - def read(self): + def read(self, bits_per_op=None): + bits_per_op = bits_per_op or self.bits_per_op if self.read_count and self.stream.bitpos == self.stream.length: self.stream.clear() self.stream.append(Bits(bytes=self.f.read(self.read_size))) self.read_count -= 1 try: - bits = self.stream.read(f'uint:{self.bits_per_op}') + bits = self.stream.read(f'uint:{bits_per_op}') except bitstring.ReadError: try: bits = self.stream.read('uint') diff --git a/cimbar/util/clustering.py b/cimbar/util/clustering.py new file mode 100644 index 0000000..c31a893 --- /dev/null +++ b/cimbar/util/clustering.py @@ -0,0 +1,39 @@ +import json +import sys + +import numpy +from sklearn.cluster import KMeans + + +class ClusterSituation(): + def __init__(self, data, num_clusters=4): + if not data: + raise Exception("what're ya doin', there's no datum points!") + + self.data = numpy.array(data) + self.num_clusters = num_clusters + self.kmeans = KMeans(n_clusters=num_clusters, random_state=0) + self.kmeans.fit(self.data) + self.labels = self.kmeans.labels_ + self.index = None + + def centers(self): + print(self.kmeans.cluster_centers_) + return self.kmeans.cluster_centers_ + + def categorize(self, point): + cat = self.kmeans.predict([point,])[0] + if self.index: + cat = self.index[cat] + return cat + + def plot(self, filename): + from matplotlib import pyplot + + fig = pyplot.figure() + ax = fig.add_subplot(111, projection='3d') + ax.scatter(self.data[:,0], self.data[:,1], self.data[:,2], c=self.kmeans.labels_) + ax.set_ylim(ax.get_ylim()[::-1]) + pyplot.xlabel('red') + pyplot.ylabel('green') + fig.savefig(filename) diff --git a/requirements b/requirements index cb4f91c..ce4a634 100644 --- a/requirements +++ b/requirements @@ -1,4 +1,5 @@ -bitstring +bitstring==3.1.9 +colour-science docopt imagehash numpy diff --git a/requirements.freeze b/requirements.freeze new file mode 100644 index 0000000..f4935b4 --- /dev/null +++ b/requirements.freeze @@ -0,0 +1,14 @@ +bitstring==3.1.9 +colour-science +docopt==0.6.2 +ImageHash==4.3.1 +imageio==2.34.0 +numpy==1.24.4 +opencv-python==4.9.0.80 +pillow==10.2.0 +PyWavelets==1.4.1 +reedsolo==1.7.0 +scipy==1.10.1 +typing-extensions==4.9.0 +zstandard==0.22.0 +https://github.com/sz3/pywirehair/archive/master.zip diff --git a/samples b/samples index 1e86f11..7443f61 160000 --- a/samples +++ b/samples @@ -1 +1 @@ -Subproject commit 1e86f113f2be91c07ba36aa8f4f16bedf6f092d3 +Subproject commit 7443f6144991600515283476f3b6fbee21dbbd57 diff --git a/tests/test_cimb_translator.py b/tests/test_cimb_translator.py index 90ed20f..aeb3d2a 100644 --- a/tests/test_cimb_translator.py +++ b/tests/test_cimb_translator.py @@ -29,8 +29,8 @@ def test_decode_dark(self): self.assertEqual(decoded, 5) self.assertEqual(error, 0) - color = cimb.decode_color(img) - self.assertEqual(color, 0 << 4) + color = cimb.decode_color(img, 0) + self.assertEqual(color, 1) img2 = Image.open(path.join(CIMBAR_ROOT, 'tests', 'sample', '25.png')) img2 = img2.convert('RGB') @@ -38,5 +38,5 @@ def test_decode_dark(self): self.assertEqual(decoded, 5) self.assertEqual(error, 0) - color = cimb.decode_color(img2) - self.assertEqual(color, 1 << 4) + color = cimb.decode_color(img2, 0) + self.assertEqual(color, 2) diff --git a/tests/test_cimbar.py b/tests/test_cimbar.py index f834a72..4edea78 100644 --- a/tests/test_cimbar.py +++ b/tests/test_cimbar.py @@ -2,13 +2,14 @@ from os import path from tempfile import TemporaryDirectory from unittest import TestCase +from unittest.mock import patch import cv2 import numpy from cimbar.cimbar import encode, decode, bits_per_op from cimbar.encode.rss import reed_solomon_stream -from cimbar.grader import evaluate as evaluate_grader +from cimbar.grader import evaluate_split, evaluate_interleaved CIMBAR_ROOT = path.abspath(path.join(path.dirname(path.realpath(__file__)), '..')) @@ -78,7 +79,7 @@ def validate_output(self, out_path): self.assertEqual(contents, self._src_data()[:7500]) def validate_grader(self, out_path, target): - num_bits = evaluate_grader(self.decode_clean, out_path, bits_per_op(), True) + num_bits = evaluate_split(self.decode_clean, out_path, bits_per_op(), 2) self.assertLess(num_bits, target) def test_decode_simple(self): @@ -112,7 +113,8 @@ def test_decode_perspective_rotate(self): decode([skewed_image], out_no_ecc, dark=True, ecc=0, force_preprocess=True) self.validate_grader(out_no_ecc, 4000) - def test_decode_sample(self): + @patch('cimbar.cimbar.use_split_mode', lambda: False) + def test_decode_sample_mode4c(self): clean_image = 'samples/6bit/4color_ecc30_0.png' warped_image = 'samples/6bit/4_30_802.jpg' @@ -122,7 +124,20 @@ def test_decode_sample(self): warped_bits = self._temp_path('outfile_warped.txt') decode([warped_image], warped_bits, dark=True, ecc=0, force_preprocess=True, auto_dewarp=False) - num_bits = evaluate_grader(clean_bits, warped_bits, bits_per_op(), True) + num_bits = evaluate_interleaved(clean_bits, warped_bits, bits_per_op()) + self.assertLess(num_bits, 350) + + def test_decode_sample_modeb(self): + clean_image = 'samples/b/tr_0.png' + warped_image = 'samples/b/ex2434.jpg' + + clean_bits = self._temp_path('outfile_clean.txt') + decode([clean_image], clean_bits, dark=True, ecc=0, auto_dewarp=False) + + warped_bits = self._temp_path('outfile_warped.txt') + decode([warped_image], warped_bits, dark=True, ecc=0, force_preprocess=True, auto_dewarp=False) + + num_bits = evaluate_split(clean_bits, warped_bits, bits_per_op(), 2) self.assertLess(num_bits, 350) @@ -145,10 +160,10 @@ def test_roundtrip(self): encode(self.src_file, dst_image, dark=True, fountain=True) out_path = path.join(self.temp_dir.name, 'out.txt') - decode([dst_image], out_path, dark=True, deskew=False, auto_dewarp=False, fountain=True) + decode([dst_image], out_path, dark=True, deskew=False, auto_dewarp=False, fountain=True, color_correct=3) with open(out_path, 'rb') as f: contents = f.read() with open(self.src_file, 'rb') as f: expected = f.read() - self.assertEquals(contents, expected) + self.assertEqual(contents, expected) diff --git a/tests/test_fountain.py b/tests/test_fountain.py index 777e6b7..d20875f 100644 --- a/tests/test_fountain.py +++ b/tests/test_fountain.py @@ -47,7 +47,7 @@ def test_encode(self): data = b'0123456789' * 100 inbuff = BytesIO(data) - fes = fountain_encoder_stream(inbuff, 400) + fes = fountain_encoder_stream(inbuff, 400, encode_id=0) r = fes.read(400) self.assertEqual(b'\x00\x00\x03\xe8\x00\x00' + data[:394], r)