spline agent migration and JSON schema changes

[juja76]

hi there,

first of all wanted to thank you for the tremendous effort you have put in place for bringing the Spline to the community.

I'm using Spline since version 0.5.0 (+spark 2.3) and had no problem upgrading to 0.5.6. Recently, I have upgraded to 0.7.7 (+spark 3.2) and I see significant changes in terms of the JSON schema for generated lineage files. This, in turn, has got severe impact on the lineage data processing logic. Thus, I have three questions for you:

1. are there any documentation that describes those changes, I need it to estimate the effort for the upgrade of the lineage data processing pipelines?
2. is there any way to upgrade to the 0.7.* versions branch, but still to generate a JSON lineage files compatible with the previous version of the spline agent, for example 0.5.6.
3. (in the case 2 is not possible) What is the highest spline version I can use that would provide me with the JSONs compatible with 0.5.6 version, but still provide full support for spark 3.2 and 3.3.

thank you in advance

[wajda]

Hi @juja76,

I'm using Spline since version 0.5.0 (+spark 2.3) and had no problem upgrading to 0.5.6. Recently, I have upgraded to 0.7.7 (+spark 3.2) and I see significant changes in terms of the JSON schema for generated lineage files.

Can you more specific on that one please? What dispatcher have you been using, and what changes in the JSON model break your pipeline?
If you are using embedded HTTP dispatcher connected to your custom endpoint that there should not be any (visible) incompatible changes between 0.5.6 and 0.7.7.

are there any documentation that describes those changes, I need it to estimate the effort for the upgrade of the lineage data processing pipelines?

There is no such doc as there should be no need for it as the agent is meant to be backward compatible for the majority of users.

is there any way to upgrade to the 0.7.* versions branch, but still to generate a JSON lineage files compatible with the previous version of the spline agent, for example 0.5.6.

Yes there is. The Spline agent supports all JSON models retrospectively starting from 0.4+
Basically, in 0.x version series, only two incompatible model versions have been introduced - model ver. 1 and 1.1 (do not confuse with the agent version, it's unrelated). The way how they are selected depends on the lineage dispatcher in use. HTTP dispatcher negotiates the version with the server and falls back to the oldest supported one (ver 1). Starting from agent 0.7.9 you can use the spline.lineageDispatcher.http.apiVersion config property to enforce the desired model version in the HTTP dispatcher. (see https://github.com/AbsaOSS/spline-spark-agent/blob/release/0.7/core/src/main/resources/spline.default.properties#L42). Other dispatchers, if I'm not mistaken, just use the latest model version.

(in the case 2 is not possible) What is the highest spline version I can use that would provide me with the JSONs compatible with 0.5.6 version, but still provide full support for spark 3.2 and 3.3.

We always recommend to use the latest release with the same major version number - 0.x.y in your case (0.7.12 at the time of writing this).

[juja76]

Thank you for the support!

There are too many changes in the JSON model to list them all, I'll provide you with just a few examples:

1. The attributes map is moved. The mapping information about the attributes in version 0.5.6 was available in the JSON document at 'extraInfo.attributes' but in version 0.7.7 at 'attributes'.
2. The IO metrics are missing. In version 0.5.6 the read and write metrics were available at 'extraInfo.readMetrics' and 'extraInfo.writeMetrics'. Not available in 0.7.7
3. The attribute that holds info about reads operations schema has been renamed. In version 0.5.6 the attribute for the output schema was available at 'reads.schema' in version 0.7.7 at 'reads.output'. The same applies to 'other' operations.

We are using custom implementation of the dispatcher, it is very similar to HDFSLineageDispatcher that is provided with the distributive. It is implementing LineageDispatcher trait and on 'send' it streams down the lineage metadata into HDFS or S3 filesystem depending on the environment it is running in and using specific logic to infer the path where it suppose to be written from path of write operation. These are the settings we supply in spline.default.properties

=======================================

spline.internal.execPlan.uuid.version=5

spline.mode=BEST_EFFORT
spline.file.name=_SPLINE_LINEAGE

spline.lineageDispatcher=generic
spline.lineageDispatcher.generic.className=xxx.spline.harvester.dispatcher.GenericLineageDispatcher

spline.IWDStrategy=default
spline.IWDStrategy.default.className=za.co.absa.spline.harvester.iwd.DefaultIgnoredWriteDetectionStrategy
spline.IWDStrategy.default.onMissingMetrics=IGNORE_LINEAGE

spline.postProcessingFilter=composite
spline.postProcessingFilter.composite.className=za.co.absa.spline.harvester.postprocessing.CompositePostProcessingFilter

spline.user_extra_meta_provider.className=za.co.absa.spline.harvester.extra.NoopUserExtraMetaDataProvider

=======================================

Following your analogy with HttpDispatcher, I can guess that I'll have to do something like:

spline.lineageDispatcher.generic.apiVersion=1

Is that correct?
Just to let you know, that in the custom dispatcher implementation I don't reference spline.lineageDispatcher.generic.apiVersion property directly, so I'm not sure it will work.

Or I have to implements something like what has been done in HttpDispatcher:

private val modelMapper = ModelMapper.forApiVersion(ProducerApiVersion.V1)

and on send()

val execPlanDTO = modelMapper.toDTO(plan)
sendJson(execPlanDTO.toJson, executionPlansEndpoint)

please advice how shall I proceed?

[wajda]

spline.lineageDispatcher.generic.apiVersion=1

If you are using a custom lineage dispatcher implementation then you don't need to set any properties like that unless you explicitly read it and use it in your dispatcher code.

private val modelMapper = ModelMapper.forApiVersion(ProducerApiVersion.V1)

Yes, this is the correct way to do it on the custom dispatcher level.

[juja76]

I got it running.
thanks a lot!

[juja76]

Dear Alex,

I have a follow up question that is not technical but directly related to the technical issue we have discussed with you in that thread.

In our organisation, we are laying down long term plan for our data ecosystem. And we would like to understand better if you have any plans in the future (2-3 years) to discontinue the support for the ModelMapper feature that allows us to produce the JSON files compatible with model version V1, still using latest versions of Spline agent?

Thank you in advance,
Alexander

[wajda]

Well, I don't have any crystal ball to look into, but if you ask me, I would say that for the next 2-3 years, with respect to the V1 support, there is nothing to worry about :)
Our vision for Spline is a long-term backward and forward compatibility. Spline is meant to be a central service in the organisation, and hence might not be always of the latest version. Hence the newer agents should be able to talk older protocols. I can imagine us to deprecate older models if we decide to completely redo the protocol. But currently, we have no plans to make any drastic changes to the Producer API model in any foreseeable future. The model may evolve, of course, but it will most likely be just some backward compatible additions.

[juja76]

much appreciate your help!