Json output, progress mode

[lavocatt]

This design document opens the discussion about a new mode for the json output of osbuild, the progress mode.

This work is based on the work of @achilleas-k and is accessible:

• on the pull request osbuild: Add a new progress mode for --json output osbuild#949 for osbuild
• on the pull request log osbuild messages in real time osbuild-composer#2213 for composer

In the rest of the document I'll discuss the decisions taken so far:

Progress mode:

A new argument is accessible for osbuild --json-mode, the argument takes two possible values batch and progress.

• The first one lets osbuild outputs a legacy json
• the second one, progress, adds a new type of json message.

Details on the progress functionality

Each time a log message is produced by a stage, a pipeline, or even any component of osbuild, a json message is printed on stdout.
Each message is printed on one line and is followed by a carriage return.
These carriage returns are then used in composer to split the different json messages from each other.

Format of the json message

{
    "message": "some.message",
    "error": null,

    "context": {
        "origin": "some.origin",
        "pipeline": {
            "name": "some.pipeline",
            "id": "pipeline.id",
            "stage": {
                "name": "some.stage",
                "id": "stage.id"
            }
        },
        "id": "some.id"
    },
    "progress": {
        "name": "pipelines",
        "total": 3,
        "done": null,
        "unit": null
    },

    "timestamp": 1643104972.0953932
}

At this time the message contains also a context that can be populated by a pipeline and a stage. This can be used then by composer to gather messages from a same source and improve the user readability.

At this time the error field seems unused and could be in the future used to get back structured error to composer.
unit field is also unused.

Composer intergration

A simple loop in composer takes care of unmarshalling the json messages coming from osbuild. The algorithm does:

while the a scanner that reads osbuild stdout can get a full line terminated by a carriage return and does not get EOF:
    try to strictly decode the line as a Message kind
        if this is a success, print out the message if we are in debug mode
        else if the decoding was a failure
             try to decode the line as legacy osbuild's output.
                 if this succeed, break the loop and proceed as usual
                 else this is a failure, return an error

At this time composer ignore the progress information and the timing information.

future upgrades

• in progress mode, do not put the logs into the legacy report at the end of the osbuild execution
• be able to give a severity to a message (DEBUG, INFO, WARN) and reflect this level of severity to composer to filter depending on the level in the configuration file of the worker
• be able to send back detailed and structured errors to composer, meaning, catch the exceptions, extract the message and the location of the failure in the code, encapsulate this as a valid json and print it out on the stdout.
• Use the timing information and the progress information to gather data for analytics.