Establish the structure/schema of an incoming messages on MQTT

[joepavitt]

Given our traffic monitoring for the MQTT brokers, establish, and record, the payload structure/schema seen on each topic.

This would extend the work in #4919 where Nick established a foundation Async API schema for the topic hierarchy.

[hardillb]

I've added some basic tests to the mqtt-schema-agent to test for the following:

• Strings
  • XML
  • JSON (and then fed to the schema generator that @knolleary provided in the mqtt-schema-agent project)
• Anything else

At the moment this is pushed to the topic API, but not stored yet. Need to work out what columns we need to add to the MQTTTopicSchema table

[joepavitt]

Just to emphasise something I mentioned yesterday re: workflow that may impact DB columns, ideally we have a review process in place (as requested by customers). We propose that we think a given structure is relevant, e.g. string, json, etc. and then the user confirms it.

once confirmed it'd show as part of the AsyncAPI docs and the view in the topic hierarchy.

Designs can be found here: #4963

[hardillb]

I'm thinking we need 2 columns

• inferred payload schema
• user payload schema

The first carries the value from the agent, the UI lets the user either "accept" this (copy it to the user payload column) or provide their own.

This way users can always override the generated value?

[knolleary]

We already have the metadata column as a JSON blob. That should be treated as the official record of any meta data against the topic that the user has entered/confirmed etc. That can cover description, payload etc.

We should add a second column for the inferred metadata.

It does feel like this can get messy quite quickly after the user has 'confirmed' a payload schema - and we continue to detect and infer things under the covers. Not sure how we'd highlight to the user there's a difference. Lets take this one step at a time.

[knolleary]

Have raised FlowFuse/mqtt-schema-agent#12 to improve some of the edge case handling around non-string payloads.

[joepavitt]

@knolleary this is good to close now, right?

[joepavitt]

I believe it's closed by FlowFuse/mqtt-schema-agent#6