From b955880dd39174db5d1924e728eecbf443d06200 Mon Sep 17 00:00:00 2001 From: Joao Paulo Magalhaes Date: Sun, 5 May 2024 18:54:09 +0200 Subject: [PATCH] event-based parser policy pt6: docs --- README.md | 32 ++++++---- changelog/current.md | 108 +++++++++++++++++++++++++++++++++ doc/Doxyfile | 2 + doc/doxy_main.md | 1 + doc/index.rst | 4 ++ doc/sphinx_other_languages.rst | 12 +++- doc/sphinx_quicklinks.rst | 7 ++- doc/sphinx_using.rst | 3 +- doc/sphinx_yaml_standard.rst | 66 +++++++++++--------- 9 files changed, 191 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 8faeaa375..dff3d55d5 100644 --- a/README.md +++ b/README.md @@ -729,13 +729,22 @@ See also [the roadmap](./ROADMAP.md) for a list of future work. ryml deliberately makes no effort to follow the standard in the following situations: -* Containers are not accepted as mapping keys: keys must be scalars. +* ryml's tree does NOT accept containers are as mapping keys: keys + must be scalars. HOWEVER, this is a limitation only of the tree. The + event-based parser engine DOES parse container keys. The parser + engine is the result of a recent refactor and its usage is meant to + be used by other programming languages to create their native + data-structures. This engine is fully tested and fully conformant + (other than the general error permissiveness noted below). But + because it is recent, it is still undocumented, and it requires some + API cleanup before being ready for isolated use. Please get in touch + if you are interested in integrating the event-based parser engine + without the standalone `ryml::parse_*()` * Tab characters after `:` and `-` are not accepted tokens, unless ryml is compiled with the macro `RYML_WITH_TAB_TOKENS`. This requirement exists because checking for tabs introduces branching into the parser's hot code and in some cases costs as much as 10% in parsing time. -* Anchor names must not end with a terminating colon: eg `&anchor: key: val`. * Non-unique map keys are allowed. Enforcing key uniqueness in the parser or in the tree would cause log-linear parsing complexity (for root children on a mostly flat tree), and would increase code size @@ -754,18 +763,17 @@ following situations: reflects the usual practice of having at most 1 or 2 tag directives; also, be aware that this feature is under consideration for removal in YAML 1.3. - -Also, ryml tends to be on the permissive side where the YAML standard -dictates there should be an error; in many of these cases, ryml will -tolerate the input. This may be good or bad, but in any case is being -improved on (meaning ryml will grow progressively less tolerant of -YAML errors in the coming releases). So we strongly suggest to stay -away from those dark corners of YAML which are generally a source of -problems, which is a good practice anyway. +* ryml tends to be on the permissive side in several cases where the + YAML standard dictates that there should be an error; in many of these + cases, ryml will tolerate the input. This may be good or bad, but in + any case is being improved on, meaning ryml will grow progressively + less tolerant of YAML errors in the coming releases. So we strongly + suggest to stay away from those dark corners of YAML which are + generally a source of problems; this is good practice anyway. If you do run into trouble and would like to investigate conformance -of your YAML code, beware of existing online YAML linters, many of -which are not fully conformant; instead, try using +of your YAML code, **beware** of existing online YAML linters, many of +which are not fully conformant. Instead, try using [https://play.yaml.io](https://play.yaml.io), an amazing tool which lets you dynamically input your YAML and continuously see the results from all the existing parsers (kudos to @ingydotnet and the people diff --git a/changelog/current.md b/changelog/current.md index e69de29bb..d005e6836 100644 --- a/changelog/current.md +++ b/changelog/current.md @@ -0,0 +1,108 @@ +**All the changes described come from a single PR: [#PR414](https://github.com/biojppm/rapidyaml/pull/414).** + + +### Parser refactor + +The parser was completely refactored ([#PR414](https://github.com/biojppm/rapidyaml/pull/414)). This was a large and hard job carried out over several months, and the result is: + +- A new event-based parser engine is now in place, enabling the improvements described below. This engine uses a templated event handler, where each event is a function call, which spares branches on the event handler. The parsing code was fully rewritten, and is now much more simple (albeit longer), and much easier to work with and fix. +- YAML standard-conformance was improved significantly. Along with many smaller fixes and additions, (too many to list here), the main changes are the following: + - The parser engine can now successfully parse container keys, emitting all the events in the correct , **but** as before, the ryml tree cannot accomodate these (and this constraint is no longer enforced by the parser, but instead by `EventHandlerTree`). For an example of a handler which can accomodate key containers, see the one which is used for the test suite at `test/test_suite/test_suite_event_handler.hpp` + - Anchor keys can now be terminated with colon (eg, `&anchor: key: val`), as dictated by the standard. +- The parser engine can now be used to create native trees in other programming languages, or in cases where the user *must* have container keys. +- Parsing performance improved (benchmark results incoming) from reduced parser branching. +- Emitting performance improved (benchmark results incoming), as the emitting code no longer has to read the full scalars to decide on an appropriate emit style. + + +### Strict JSON parser + +- A strict JSON parser was added. Use the `parse_json_...()` family of functions to parse json in stricter mode (and faster) than flow-style YAML. + + +### YAML style preserved while parsing + +- The YAML style information is now fully preserved through parsing/emitting round trips. This was made possible because the event model of the new parsing engine now incorporates style varieties. So, for example: + - a scalar parsed from a plain/single-quoted/double-quoted/block-literal/block-folded scalar will be emitted always using its original style in the YAML source + - a container parsed in block-style will always be emitted in block-style + - a container parsed in flow-style will always be emitted in flow-style + Because of this, the style of YAML emitted by ryml changes from previous releases. +- Scalar filtering was improved and is now done directly in the source being parsed (which may be in place or in the arena), except in the cases where the scalar expands and does not fit its initial range, in which case the scalar is filtered out of place to the tree's arena. + - Filtering can now be disabled while parsing, to ensure a fully-readonly parse (but this feature is still experimental and somewhat untested, given the scope of the rewrite work). + - The parser now offers methods to filter scalars in place or out of place. +- Style flags were added to `NodeType_e`: + ``` + FLOW_SL ///< mark container with single-line flow style (seqs as '[val1,val2], maps as '{key: val,key2: val2}') + FLOW_ML ///< mark container with multi-line flow style (seqs as '[\n val1,\n val2\n], maps as '{\n key: val,\n key2: val2\n}') + BLOCK ///< mark container with block style (seqs as '- val\n', maps as 'key: val') + KEY_LITERAL ///< mark key scalar as multiline, block literal | + VAL_LITERAL ///< mark val scalar as multiline, block literal | + KEY_FOLDED ///< mark key scalar as multiline, block folded > + VAL_FOLDED ///< mark val scalar as multiline, block folded > + KEY_SQUO ///< mark key scalar as single quoted ' + VAL_SQUO ///< mark val scalar as single quoted ' + KEY_DQUO ///< mark key scalar as double quoted " + VAL_DQUO ///< mark val scalar as double quoted " + KEY_PLAIN ///< mark key scalar as plain scalar (unquoted, even when multiline) + VAL_PLAIN ///< mark val scalar as plain scalar (unquoted, even when multiline) + ``` +- Style predicates were added to `NodeType`, `Tree`, `ConstNodeRef` and `NodeRef`: + ``` + bool is_container_styled() const; + bool is_block() const + bool is_flow_sl() const; + bool is_flow_ml() const; + bool is_flow() const; + + bool is_key_styled() const; + bool is_val_styled() const; + bool is_key_literal() const; + bool is_val_literal() const; + bool is_key_folded() const; + bool is_val_folded() const; + bool is_key_squo() const; + bool is_val_squo() const; + bool is_key_dquo() const; + bool is_val_dquo() const; + bool is_key_plain() const; + bool is_val_plain() const; + ``` +- Style modifiers were also added: + ``` + void set_container_style(NodeType_e style); + void set_key_style(NodeType_e style); + void set_val_style(NodeType_e style); + ``` +- Emit helper predicates were added, and are used when an emitted node was built programatically without style flags: + ``` + /** choose a YAML emitting style based on the scalar's contents */ + NodeType_e scalar_style_choose(csubstr scalar) noexcept; + /** query whether a scalar can be encoded using single quotes. + * It may not be possible, notably when there is leading + * whitespace after a newline. */ + bool scalar_style_query_squo(csubstr s) noexcept; + /** query whether a scalar can be encoded using plain style (no + * quotes, not a literal/folded block scalar). */ + bool scalar_style_query_plain(csubstr s) noexcept; + ``` + +### Breaking changes + +As a result of the refactor, there are some limited changes with impact in client code. Even though this was a large refactor, effort was directed at keeping maximal backwards compatibility, and the changes are not wide. But they still exist: + +- The existing `parse_...()` methods in the `Parser` class were all removed. Use the corresponding `parse_...(Parser*, ...)` function from the header [`c4/yml/parse.hpp`](https://github.com/biojppm/master/src/c4/yml/parse.hpp) (link valid after this branch is merged). +- When instantiated by the user, the parser now needs to receive a `EventHandlerTree` object, which is responsible for building the tree. Although fully functional and tested, the structure of this class is still somewhat experimental and is still likely to change. There is an alternative event handler implementation responsible for producing the events for the YAML test suite in `test/test_suite/test_suite_event_handler.hpp`. +- The declaration and definition of `NodeType` was moved to a separate header file `c4/yml/node_type.hpp` (previously it was in `c4/yml/tree.hpp`). +- Some of the node type flags were removed, and several flags (and combination flags) were added. + - Most of the existing flags are kept, as well as their meaning. + - `KEYQUO` and `VALQUO` are now masks of the several style flags for quoted scalars. In general, however, client code using these flags and `.is_val_quoted()` or `.is_key_quoted()` is not likely to require any changes. + + +### New type for node IDs + +A type `id_type` was added to signify the integer type for the node id, defaulting to the backwards-compatible `size_t` which was previously used in the tree. In the future, this type is likely to change, *and probably to a signed type*, so client code is encouraged to always use `id_type` instead of the `size_t`, and specifically not to rely on the signedness of this type. + + +### Reference resolver is now exposed + +The reference (ie, alias) resolver object is now exposed in +[`c4/yml/reference_resolver.hpp`](https://github.com/biojppm/master/src/c4/yml/reference_resolver.hpp) (link valid after this PR is merged). Previously this object was temporarily instantiated in `Tree::resolve()`. Exposing it now enables the user to reuse this object through different calls, saving a potential allocation on every call. diff --git a/doc/Doxyfile b/doc/Doxyfile index b33ec7dff..6358b1f13 100644 --- a/doc/Doxyfile +++ b/doc/Doxyfile @@ -952,6 +952,8 @@ WARN_LOGFILE = INPUT = \ ./doxy_main.md \ ../src \ + ../test/test_suite/test_suite_event_handler.hpp \ + ../test/test_suite/test_suite_event_handler.cpp \ ../samples/quickstart.cpp \ ../ext/c4core/src/c4/substr.hpp \ ../ext/c4core/src/c4/charconv.hpp \ diff --git a/doc/doxy_main.md b/doc/doxy_main.md index 728a3c426..5a64e9ae2 100644 --- a/doc/doxy_main.md +++ b/doc/doxy_main.md @@ -11,6 +11,7 @@ * @ref doc_tree * @ref doc_node_classes * For serialization/deserialization, see @ref doc_serialization. + * @ref doc_ref_utils - how to resolve references to anchors * @ref doc_tag_utils - how to resolve tags * @ref doc_callbacks - how to set up error/allocation/deallocation callbacks either globally for the library, or for specific objects diff --git a/doc/index.rst b/doc/index.rst index 8138fbf42..b26b178cd 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -47,6 +47,8 @@ ryml is written in C++11, and compiles cleanly with: `_. +---- + Table of contents ================= @@ -54,6 +56,7 @@ Table of contents :maxdepth: 3 Doxygen docs + YAML playground ./sphinx_quicklinks ./sphinx_is_it_rapid ./sphinx_yaml_standard @@ -61,6 +64,7 @@ Table of contents ./sphinx_other_languages ./sphinx_alternative_libraries +---- API teaser ========== diff --git a/doc/sphinx_other_languages.rst b/doc/sphinx_other_languages.rst index f9dc9c909..461c6d02d 100644 --- a/doc/sphinx_other_languages.rst +++ b/doc/sphinx_other_languages.rst @@ -8,11 +8,21 @@ out the general approach, other languages are likely to follow (all of this is possible because we’re using `SWIG `__, which makes it easy to do so). + JavaScript ---------- A JavaScript+WebAssembly port is available, compiled through -`emscripten `__. +`emscripten `__. Here's a quick example of +how to compile ryml with emscripten using ``emcmake``: + +.. code:: bash + + git clone --recursive https://github.com/biojppm/rapidyaml + cd rapidyaml + emcmake cmake -S . -B build \ + -DCMAKE_CXX_FLAGS="-s DISABLE_EXCEPTION_CATCHING=0" + Here's a quick example on how to configure, compile and run the tests using `emscripten`: diff --git a/doc/sphinx_quicklinks.rst b/doc/sphinx_quicklinks.rst index 63f87d837..b4c8c7c1e 100644 --- a/doc/sphinx_quicklinks.rst +++ b/doc/sphinx_quicklinks.rst @@ -3,7 +3,11 @@ Quick links * `API documentation (Doxygen) <./doxygen/index.html>`_ -* `Github repo `_ +* `YAML playground `_ + +* YAML Test Suite `online `_ / `Github `_ + +* `rapidyaml Github repo `_ * `Issues `_ @@ -17,7 +21,6 @@ Quick links * `README [0.6.0] `_ - * Since latest release (master branch): * `README [master] `_ diff --git a/doc/sphinx_using.rst b/doc/sphinx_using.rst index 89e5254aa..d498282d8 100644 --- a/doc/sphinx_using.rst +++ b/doc/sphinx_using.rst @@ -210,8 +210,7 @@ of ryml: low-level multi-platform utilities for C++. When ``RYML_STANDALONE=ON``, c4core is incorporated into ryml as if it is the same library. Defaults to ``ON``. -- ``RYML_INSTALL=ON/OFF``. enable/disable install target. Defaults to - ``ON``. +- ``RYML_INSTALL=ON/OFF``. enable/disable install target. Defaults to ``ON``. If you’re developing ryml or just debugging problems with ryml itself, the following cmake variables can be helpful: diff --git a/doc/sphinx_yaml_standard.rst b/doc/sphinx_yaml_standard.rst index 28ea3bc48..fdf5c13e8 100644 --- a/doc/sphinx_yaml_standard.rst +++ b/doc/sphinx_yaml_standard.rst @@ -10,22 +10,43 @@ appear cases which ryml fails to parse. Your `bug reports or pull requests `__ are very welcome. -See also `the roadmap `__ for a list of future work. +.. note:: + + If you do run into trouble and would like to investigate + conformance of your YAML code, **do not use existing online YAML + linters**, many of which are not fully conformant; instead, try + using `https://play.yaml.io + `__, + an amazing tool which lets you dynamically input your YAML and + continuously see the results from all the existing parsers (kudos + to @ingydotnet and the people from the YAML test suite). And of + course, if you detect anything wrong with ryml, please `open an + issue `__ so that we + can improve. + -Known limitations ------------------ +Deliberate deviations +--------------------- ryml deliberately makes no effort to follow the standard in the following situations: -- Containers are not accepted as mapping keys: keys must be scalars. +- ryml's tree does NOT accept containers are as mapping keys: keys + must be scalars. HOWEVER, this is a limitation only of the tree. The + event-based parser engine DOES parse container keys. The parser + engine is the result of a recent refactor and its usage is meant to + be used by other programming languages to create their native + data-structures. This engine is fully tested and fully conformant + (other than the general error permissiveness noted below). But + because it is recent, it is still undocumented, and it requires some + API cleanup before being ready for isolated use. Please get in touch + if you are interested in integrating the event-based parser engine + without the standalone `ryml::parse_*()` - Tab characters after ``:`` and ``-`` are not accepted tokens, unless ryml is compiled with the macro ``RYML_WITH_TAB_TOKENS``. This requirement exists because checking for tabs introduces branching into the parser’s hot code and in some cases costs as much as 10% in parsing time. -- Anchor names must not end with a terminating colon: eg - ``&anchor: key: val``. - Non-unique map keys are allowed. Enforcing key uniqueness in the parser or in the tree would cause log-linear parsing complexity (for root children on a mostly flat tree), and would increase code size @@ -45,31 +66,22 @@ following situations: tag directives; also, be aware that this feature is under consideration for removal in YAML 1.3. -Also, ryml tends to be on the permissive side where the YAML standard -dictates there should be an error; in many of these cases, ryml will -tolerate the input. This may be good or bad, but in any case is being -improved on (meaning ryml will grow progressively less tolerant of YAML -errors in the coming releases). So we strongly suggest to stay away from -those dark corners of YAML which are generally a source of problems, -which is a good practice anyway. -.. note:: +Known (unintended) deviations +----------------------------- + +ryml tends to be on the permissive side in several cases where the +YAML standard dictates that there should be an error; in many of these +cases, ryml will tolerate the input without producing an error. This +is being improved on, meaning **ryml will grow progressively less +tolerant of YAML errors** in the coming releases. So we strongly +suggest to stay away from those dark corners of YAML which are +generally a source of problems; this is good practice anyway. - If you do run into trouble and would like to investigate - conformance of your YAML code, **do not use existing online YAML - linters**, many of which are not fully conformant; instead, try - using `https://play.yaml.io - `__, - an amazing tool which lets you dynamically input your YAML and - continuously see the results from all the existing parsers (kudos - to @ingydotnet and the people from the YAML test suite). And of - course, if you detect anything wrong with ryml, please `open an - issue `__ so that we - can improve. YAML test suite -=============== +--------------- As part of its CI testing, ryml uses the `YAML test suite `__. This is an extensive @@ -99,7 +111,7 @@ several hundred thousand individual tests to which ryml is subjected, which are added to the unit tests in ryml, which also employ the same extensive combinatorial approach. -Also, note that in `their own words `__, the +Also, note that in `their own words `__, the tests from the YAML test suite *contain a lot of edge cases that don’t play such an important role in real world examples*. And yet, despite the extreme focus of the test suite, currently ryml only fails a minor