RFC: SDoc format vs Python

[stanislaw]

Is it crazy enough to consider replacing SDoc or adding another format where the requirements could be described in pure Python?

from dataclasses import dataclass
from typing import Union, List

@dataclass
class Requirement:
    TITLE: str
    STATEMENT: str


@dataclass
class Document:
    children: List[Union[Requirement]]


requirement = Requirement(
    TITLE="FOOBAR",
    STATEMENT="""
StrictDoc is software for technical documentation and requirements management.

Summary of StrictDoc features:

- The documentation files are stored as human-readable text files.
- A simple domain-specific language DSL is used for writing the documents. The
  text format for encoding this language is called SDoc (strict-doc).
- StrictDoc reads \*.sdoc files and builds an in-memory representation of a
  document tree.
    """
)

document = Document(children=[requirement])
print(document)

There are so many arguments for lowering the requirements content to the level of pretty much Python and code:

type-safety, modularity, no need to deal with a custom format (SDoc) and handling its model, ...

This change would move it much closer to the other formats like SPDX or ReqIF because you cannot be any stronger than being pure software.

This is a data as code approach in its direct form.

[stanislaw]

cc @haxtibal @thseiler @nicpappler @johanenglund @fkromer @BenGardiner @richardbarlow @RobertoBagnara

[richardbarlow]

I think it's a nice idea and can completely understand the benefits it brings. However I have a big concern over how we would go about validating a system like this and how auditors would cope with it.

Unfortunately the flexibility it would bring is also a huge risk. Validation of the tool now expands to validating the whole of the python language and interpreter as the requirement definitions themselves could cause the tool to misbehave and produce incorrect output. I can't see how StrictDoc can, by design, ensure that the user of the tool doesn't cause the tool itself to misbehave. This is a critical requirement.

I think there are certain places where a Domain Specific Language is appropriate and I think that StrictDoc is one of these. Having an intentionally constrained and declarative format in this application is a must to ensure that the system can be validated.

Additionally I believe that a DSL, if well designed, can present a more pleasant user experience when it comes to editing, modifying and diffing changes. I think there are some things about the SDoc format that could be improved, but as it stands right now I think it's more than adequate.

Something that we currently do with our requirements capture solution is supplement StrictDoc with additional checks and constraints on the data using the python 'API'. This gives us the benefits of having the document model in python for (relatively) easy post-processing, without modifying StrictDoc itself. We then perform our own validation of our additional tooling against a specific StrictDoc release. I can see that capturing the data natively in python could help here, but I see it as one of StrictDoc's features that it abstracts away the underlying data format, enforces its own constraints on that format and then exposes a python interface that is validated.

Depending on the outcome of this discussion, I think a few more requirements for StrictDoc will become apparent, such as the need for a declarative data format.

[haxtibal]

Validation of the tool now expands to validating the whole of the python language and interpreter

No, IIUC we would merely borrow the python syntax, not the interpreter or other parts of the python language. strictdoc would have it's own python parser (we already have that), and ...

I can't see how StrictDoc can, by design, ensure that the user of the tool doesn't cause the tool itself to misbehave. This is a critical requirement.

... strictdoc would have it's own validator, that checks types, relations and whether the AST is correctly restricted to the strictdoc model. This is basically not different to what we have now.

Maybe we would carefully pick specific tools from the python ecosystem like mypy for type checking, but this shouldn't concern auditors more than any other dependency we already have.

Additionally I believe that a DSL, if well designed, can present a more pleasant user experience when it comes to editing, modifying and diffing changes.

I tend to agree. The stirctdoc language is an exact fit to what the strictdoc model supports. The python language is far more expressive, and it may not be immediately clear to users what is the supported subset. For example, one could be tempted to think "hey cool this is python, so I can write "

@dataclass
class Requirement:
    TITLE: str = field(default_factory=my_custom_initializer)

[fkromer]

@stanislaw

In my opinion adding a Python API for representing requirements and tests in code would be extremely powerful and beneficial. We should provide a Python API and migrate the code base that the SDoc format makes use of the Python API. About the long time continuation of the SDoc format I personally don’t care but I understand if people want to continue maintaining it (with the provided approach they actually could).

The additional layer of complexity introduced for SDoc parsing would be optional in case we continue to support the SDoc format as is (we should at least for some migration time period). With a Python API we can add new features way faster. I’m not an expert about the strictdoc code base but I could imagine it would be easier to add advanced features. E.g. for ultra fast and scalable cross language requirement traceability we could make use of existing Rust projects like duvet. I don’t understand the argument that a Python API is harder to validate than a parsing based SDoc DSL. Python code is super easy to test. The way how a „SDoc document“ is modeled with Python is another topic to be discussed. It should be pretty easy to validate as well however. @richardbarlow Could you explain what you mean with validation exactly?

[richardbarlow]

No, IIUC we would merely borrow the python syntax, not the interpreter or other parts of the python language. strictdoc would have it's own python parser (we already have that), and ...

@haxtibal if that is what is being proposed then I think the potentially perceived benefits of using python to describe requirements is greatly reduced. It means that to a causal observer it looks like python, but is in fact a reduced subset that is not actually executable (as you point out with your example of someone attempting to use the dataclass field function).

At this point it is essentially a DSL, but not optimised for the domain. We use StrictDoc primarily for capturing requirements and assessing risks for our medical device, which has a large hardware element to it. Therefore there are users of the system who are not software developers and, while the python syntax may appear to be familiar to a software dev, it is possibly not the easiest for someone who is used to reading natural language documents.

In my opinion adding a Python API for representing requirements and tests in code would be extremely powerful and beneficial. We should provide a Python API and migrate the code base that the SDoc format makes use of the Python API. About the long time continuation of the SDoc format I personally don’t care but I understand if people want to continue maintaining it (with the provided approach they actually could).

@fkromer I agree that separating the core functionality of StrictDoc from the SDoc format via a documented and maintained API would be hugely beneficial. I think this topic starts to raise a question I've been thinking about for a while: what actually is StrictDoc? Is it the format (SDoc) for capturing requirements of an arbitrary system (i.e. not just software systems) that happens to have some functionality to interact with that data (convert to other forms, assert constraints, provide GUI editor, etc). Or is StrictDoc primarily the tooling itself and the SDoc format is a side-effect and StrictDoc could (should?) support other requirement capture formats as first-class citizens? I don't necessarily mind either way, but I think it would be good if it was more clear which of the two the project is aiming to be.

For us, having our requirements in a human and machine readable form is very important and that is what attracted so StrictDoc in the first place. While a python-esque description language would be reasonably approachable for most software developers, I don't think it would be the most optimal for non-software people.

I don’t understand the argument that a Python API is harder to validate than a parsing based SDoc DSL. Python code is super easy to test. The way how a „SDoc document“ is modeled with Python is another topic to be discussed. It should be pretty easy to validate as well however. @richardbarlow Could you explain what you mean with validation exactly?

I think that validating the Python API separately to the SDoc format implementation is a very sensible thing to do! As you say, it is generally much easier to validate a python interface than the combination of a DSL + complex back-end behaviour. I too am not an expert on the StrictDoc codebase, but I think it's safe to say that the split between the DSL and back-end behaviour is already there, but what is missing is the exposure of a controlled API between them that can be fully verified and validated against StrictDocs high level requirements.
Even with this division we would still be looking for whole-system validation (SDoc + back-end), but it might be that StrictDocs implementation of this validation could be simplified.

Regarding what validation means for us I wrote in a bit more detail how we see our needs being met by StrictDoc here: #2147 (comment)

[fkromer]

@richardbarlow @BenGardiner

Thanks for clarifying your arguments for SDoc (non technical users, copy&paste of documents, ...). Valid points...

@richardbarlow

what actually is StrictDoc? ... I think it would be good if it was more clear which of the two the project is aiming to be.

I agree.

@richardbarlow @haxtibal

I too am not an expert on the StrictDoc codebase, but I think it's safe to say that the split between the DSL and back-end behaviour is already there, but what is missing is the exposure of a controlled API between them that can be fully verified and validated against StrictDocs high level requirements.
Even with this division we would still be looking for whole-system validation (SDoc + back-end), but it might be that StrictDocs implementation of this validation could be simplified.

Regarding what validation means for us I wrote in a bit more detail how we see our needs being met by StrictDoc here: #2147 (comment)

W.r.t. to validation:

strictdoc is CLI wise a comparably simple input output data processing system. Exemplary defined input (SDoc) is processed to corresponding output (html, ...) which can be easily compared. A lot of people in the medical domain (e.g. covered in some Agile Embedded Podcast Episodes about testing) consider test coverage as a very weak metric I personally agree with. Would you trust a 100% test coverage based on bad quality tests? I would not. (Considering the change in test coverage is helpful metric however.) Shifting as much as possible tests into E2E tests could help in the validation. From a validators point of view it's easier to trust high level tests than lower level tests. Validators depend on lower level tests only in case production code is not covered with tests. What might be helpful could be to adapt a behavior driven development (BDD) style of E2E tests (e.g. using pytest-bdd, ...) or even a Keyword Driven Development style of tests (e.g. Robot Framework). For validators it's easier to understand and map behavior and keyword driven tests to requirements. An additional benefit of using something like the Robot Framework could be to be able to consolidate high level and low level tests and reports into a single report. The main purpose of integration and unit tests is test driving design (TDD) and giving development trust in their implementation. If the already existing Python API is sufficient to implement lower level tests for requirements there is no need to extend it from a validation point of view. The purpose of a "Python DSL API" would be to provide an alternative DSL to SDoc only and would be overhead.

Is it required to validate strictdoc Server UI wise at all? The output of it should be SDoc which is processed CLI wise afterwards, right? (I've never used the server so far.)

[BenGardiner]

I think you list good arguments and I can add to that: ease of creating sdoc in tests and also generating 'templatized' or other repetitive docs

So I think the existence of this 'format' is justified but I don't see replacing your excellent strict grammar because, as Richard points out there will always be need for validation and audit when these requirements are applied to fields that need that rigor. (Perhaps eventually all fields 🤷‍♂️)

Anyhow, my 0.02$ : introduce python as sdoc? Yes. Replace sdoc grammars with it? No

[haxtibal]

Anyhow, my 0.02$ : introduce python as sdoc? Yes. Replace sdoc grammars with it? No

If the ultimate goal is not to replace sdoc, but to support both, than the new syntax seems to be more a maintenance burden than it helps.