Model architecture: "Alias IDs" or shareKeys

[soyarsauce]

Expanding on #3654 (comment),

@nf-s : >It should be easy to add some sort of id map if needed though

@soyarsauce:

yeah id map is easy but we should do it at terriajs level as a trait.

for magda - when i put a v7 catalog into https://github.com/TerriaJS/init-to-magda it will generate UUIDs to avoid name clashes. so in the DEA prod v7 catalog right now, there exists a layer with ID dea_coastlines but it gets created into something that looks like https://dev.saas.terria.io/api/v0/registry-auth/records/b7dc3bcc-0a96-4d76-a190-dec7c42e132f?optionalAspect=terria

maybe just simplifying it down, all we need is a way to store that b7dc3bcc-0a96-4d76-a190-dec7c42e132f used to have IDs (or ID generated from name) that included dea_coastlines, so that share/story links have a way to reference back and it'll "just work"

and via @nf-s

I think that might sort out my user added data nested group id problem also - as I would just need to convert id for the top level group - and it could have some reference to its old id, so all nested items can still refer to it

Let's get a minimal implementation of this, while being careful that it doesn't break actual normal ID usage, ie getModelById should always return a model that actually has a uniqueId foo1 over a model that has uniqueId bar2 but with aliasIds: ["foo1"]

[nf-s]

I think shareKeys from master may be relevant -

terriajs/lib/Models/CatalogMember.js

Lines 128 to 134 in b58c4bd

	/**
	* An array of all possible keys that can be used to match to this catalog member when specified in a share link -
	* used for maintaining backwards compatibility when adding or changing {@link CatalogMember#id}.
	*
	* @type {String[]}
	*/
	this.shareKeys = undefined;

[soyarsauce]

On @steve9164 Sprint64Top10, to reach out to team if run into things

[soyarsauce]

Crispy to start implementation and we take it from there

(brief discussion)
//nfs: problem: going from catalog based maps, to magda based maps.

• magda will add "random IDs" to everything
• IDs do not match the automatically generated ID (from a json based catalog)
• share keys added in catalog-converter - perhaps generate IDs in catalog converter
• init-to-magda to carry the rest of the load of (v8 json catalog -> magda)

[tephenavies]

• Implement shareKeys for JSON catalogs (see Share keys #4975)
  • Add shareKeys when a JSON catalog is loaded
  • Try to match ids in share data by registered share keys (if an id doesn't resolve to a model) for:
    • Share data model strata
    • Share data such as workbench & timeline
• Generalise shareKeys to work with Magda catalogs:
  • Decide whether shareKeys should be registered for a MagdaReference or for it's target
    • Check some things for a catalog made of MagdaReferences:
      • Are ids shared between Reference and target?
      • If not check which id is serialised for models & workbench
      • Is every node in the tree represented with a MagdaReference pointing at an item. And is this guaranteed somehow?
    • Try implementing for targets
    • Add some simple unit tests for this method
    • Test against a real magda
  • Add tests for Magda catalogs that share keys are registered

---

UPDATE 11/11/2020:

• I've fixed the tests for shareKeys and spent ages debugging some other tests that I broke
  • That problem is fixed too
• Test JSON catalogs with http://ci.terria.io/share-keys/#https://gist.githubusercontent.com/steve9164/1cd25818891e37ec959fb2ba7e3b3203/raw/0336847cf8543039ead47cdbe4351b1ec19abf80/new-catalog.json&share=s-gf20jkHmg1D5MUdklkeKPa0kfZY
  • That share link was made with old-catalog.json from https://gist.github.com/steve9164/1cd25818891e37ec959fb2ba7e3b3203 and then it's applied using new-catalog.json, where the CSV has moved in the catalog
• Turns out I forgot about all the other places that getModelById is used from share data
  • Serialised workbench
  • Serialised timeline
• Extra thought needed for MagdaReference. I think I can register shareKeys for MagdaReferences
  • It should be sufficient to do it for MagdaReference targets I think.
  • This problem touches on other problems of the strict ordering of loading parents then members throughout the tree I think
    • For that problem we will need to add an API to magda to fetch a deep catalog item (and probably just enough detail about each of it's ancestors to keep the UI & model layer working)

---

UPDATE 17/11/2020:

• Working on shareKeys in Magda catalog + tests

[nf-s]

Finalising sharekeys effort:

• TerriaJS - Revert magda config ID patch + add shareKey - Revert magda config ID patch + add shareKey #5042
• TerriaJS - add v7 catalog support through v7initializationUrls - add v7 catalog support through v7initializationUrls #5054
• TerriaJS - contains previous two PRs + dynamic group shareKeys fix - Finalise shareKeys #5055
• Terrace - Move v7 catalog handling to terriaJS - https://github.com/TerriaJS/terrace/pull/158
• Catalog converter sharekeys - Sharekeys support catalog-converter#32
• Document SaaS handling of sharekeys in magda records - see ADRs

[nf-s]

Closing!