Skip to content

Commit

Permalink
Fix garbled JSON-lines output sometimes
Browse files Browse the repository at this point in the history
In `kart diff -o json-lines --add-feature-count-estimate={value}`,
two threads would write to the output. This correctly used a lock to
prevent races, but there is an intermediate bytearray which is written
just beforehand (introduced with #1025) and this also needed to use
a lock to prevent races.

This change moves that intermediate bytearray under the same lock as
writing the output buffer, thus avoiding the race condition
  • Loading branch information
craigds committed Feb 6, 2025
1 parent 943e87a commit e9b7a55
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 3 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ _When adding new entries to the changelog, please include issue/PR numbers where

## Unreleased

- diff: Fixed garbled json-lines output sometimes when using `--add-feature-count-estimate` [#1040](https://github.com/koordinates/kart/issues/1040)
- diff/show: Faster output for some large repositories (varies wildly) [#1038](https://github.com/koordinates/kart/issues/1038)
- show: Added `--no-sort-keys` option to disable sorting of features by name/PK. Previously added to `diff` only
- show: Added `--add-feature-count-estimate` option to add a feature count estimate to `json-lines` output. Previously added to `diff` only
Expand Down
6 changes: 3 additions & 3 deletions kart/json_diff_writers.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,10 +247,10 @@ def __init__(self, *args, diff_estimate_accuracy=None, delta_filter=None, **kwar
self._output_buffer = bytearray()

def dump(self, obj):
# https://jcristharif.com/msgspec/perf-tips.html#line-delimited-json
msgspec_json_encoder.encode_into(obj, self._output_buffer)
self._output_buffer.extend(b"\n")
with self._output_lock:
# https://jcristharif.com/msgspec/perf-tips.html#line-delimited-json
msgspec_json_encoder.encode_into(obj, self._output_buffer)
self._output_buffer.extend(b"\n")
self.fp.buffer.write(self._output_buffer)

def write_header(self):
Expand Down

0 comments on commit e9b7a55

Please sign in to comment.