Model Architecture: Implement SdmxCatalogItem #3524

tephenavies · 2019-06-24T01:52:07Z

No description provided.

soyarsauce · 2020-08-03T07:00:24Z

Ping Rowan or Nanda if the master sdmx stuff is still unclear

AnaBelgun · 2020-08-08T02:58:06Z

ABS has released the latest data api based on SDMX JSON v2.1
Here are the details to test: https://docs.google.com/document/d/1J3con-CBdBGlQLSelZ_eYwesX2cjYI5UnGfSLMNZg9o/edit#
ABS would like for an APIkey to be used with the calls

nf-s · 2020-08-18T06:21:25Z

SDMX learning materials

REST cheat sheets:

https://raw.githubusercontent.com/sdmx-twg/sdmx-rest/master/v2_1/ws/rest/docs/rest_cheat_sheet.pdf
https://sis-cc.gitlab.io/dotstatsuite-documentation/using-api/restful/

Reference

For structure messages see - https://github.com/sdmx-twg/sdmx-json/blob/structure-message-release/structure-message/docs/1-sdmx-json-field-guide.md#agencyscheme

SDMX group structure

Something similar to this - https://ilostat.ilo.org/data/sdmx-query-builder/

https://registry.sdmx.org/webservice/data.html

https://github.com/sdmx-twg/sdmx-json/blob/structure-message-release/structure-message/requirements/browse-by-topic.md

Steps:

Get agencyIds https://stats-nsi-stable.pacificdata.org/rest/agencyscheme/
~~Get categoryschemes https://stats-nsi-stable.pacificdata.org/rest/categoryscheme/SPC~~
- ~~Maps category schemes to categories~~
~~Get categories https://stats-nsi-stable.pacificdata.org/rest/categorisation/SPC/~~
- ~~Maps categories to dataflows~~
~~Get dataflows https://stats-nsi-stable.pacificdata.org/rest/dataflow/SPC/~~
NO! This will return categoryschemes/categories dataflows instead - https://stats-nsi-stable.pacificdata.org/rest/categoryscheme?references=parentsandsiblings
Get datastructure https://stats-nsi-stable.pacificdata.org/rest/datastructure/SPC/DSD_COMMODITY_PRICES
Get data! https://stats-nsi-stable.pacificdata.org/rest/data/DF_SDG_17

APIs

https://sdd-dotstat-api-gateway.portal.azure-api.net/concepts

See https://github.com/sdmx-twg/sdmx-rest for more

Thoughts

Maybe use sdmx-csv instead? - see https://github.com/sdmx-twg/sdmx-csv/blob/master/data-message/docs/sdmx-csv-field-guide.md

Js libs

Started with sdmx-rest, but is only really useful for creating requests.

Master used sdmxjsonlib, but it is quite basic, isn't maintained

dotstatsuite-sdmxjs which is used by .stat suite - looks like the most useful.

At this stage I am not using any libs - as the type definitions do all the hard work. I may reconsider this as I create the SdmxJsonCatalogItem

nf-s · 2020-08-19T12:29:40Z

I have types for all SDMX data messages, and some SDMX structure messages - https://github.com/nf-s/sdmx-json-types

nf-s · 2020-08-21T04:20:32Z

Groups are now working in this order:

Agency
Category scheme
Category
Dataflow

nf-s · 2020-08-21T04:21:23Z

nf-s · 2020-08-24T05:49:27Z

SDMX item structure:

Codelist
Data flow:
- Data structure
  - Concepts
- Content Constraints

All of this information is given by = https://stats-nsi-stable.pacificdata.org/rest/dataflow/SPC/DF_CPI?references=all
It may be more efficient to fetch Codelists and concepts at the SdmxGroup level - but this is easier for the moment

nf-s · 2020-08-24T13:13:50Z

Very basic SDMX item is now working

Some TODOs:

- Add region mapping support
- Deal with measurement dimensions
- Deal with time dimensions - see Region mapping not working with time-dimension #4674
- Improve compatibility with other SDMX-json services
- Test with ABS
- ~~Fix dimension options (currently it shows union of all contentContraints + cubeRegions => so some combinations of values won't return anything)~~ I don't think this matters for the moment, the cubeRegions don't seem 100% for most layers
- Simplify generated type definitions
- Support nested categories

nf-s · 2020-08-25T08:43:01Z

Basic region mapping is working (time selector isn't working yet)

But this isn't a very good visualisation for pacific countries...

nf-s · 2020-08-26T02:30:44Z

Issue with region-mapping #4676 and time-dimension #4674

nf-s · 2020-08-28T10:17:34Z

ABS region mapping now working

nf-s · 2020-09-02T03:09:29Z

It is now working on CI-deployment http://ci.terria.io/sdmx-next/#clean&https://gist.githubusercontent.com/nf-s/24b9288c6cdb6ba41d12f3d269318a0d/raw/c5ed64071c8a651ae458b4c0d3dc79c8618cb86d/sdmx.json

Not all layers will work (especially some ABS ones), but quite a few do.
Note issues with region-mapping #4676 and time-dimension #4674

nf-s · 2020-09-02T03:12:05Z

Some layers will require setting conceptTraits to work correctly - (usually for region-mapping).

Setting up regions

Information about that can be found here - https://github.com/TerriaJS/terriajs/blob/c351cb6c598ce66dc1ed373e7cc811a88766f0c5/lib/Traits/SdmxCommonTraits.ts

RegionType is determined by sdmxJsonDataflowStratum.column in this order:

conceptTraits.regionType

if conceptTraits.type === 'region' and there exists another conceptTraits with type === 'region-type' in the datastructure, therefore region-type will be set from this concept

the dimension id

the concept name

the concept id (the actual string, not URN form)

For example SPC - Option 1 above

The following catalog json specifies that the CS_COMMON(2.0).GEO_PICT concept corresponds to regionType CNT2 (2-digit country code). So whenever this concept is used in a dataflow, the region type will be set to CNT2

 "conceptOverrides": [
        {
          "id": "urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=SPC:CS_COMMON(2.0).GEO_PICT",
          "type": "region",
          "regionType": "CNT2"
        },
...

Another example, ABS - Option 2 above

Some dataflows use REGION_TYPE concept to specify the region type, and REGION concept to specify the region. If the REGION_TYPE concept is found within a dataflow, it's selectedId will be used as the regionType for the REGION concept.

 "conceptOverrides": [
        {
          "id": "urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=ABS:CS_C16_COMMON(1.0.0).REGION_TYPE",
          "type": "region-type",
          "selectedId": "SA3"
        },
        {
          "id": "urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=ABS:CS_C16_COMMON(1.0.0).REGION",
          "type": "region"
        },
...

Options 3 - 5

These are a bit more obvious, and are kind of handled in the same way a column title is used to find the corresponding regionType - for example lga_code_2016 - but it uses the first option which matches a valid regionType

Other `ConceptTraits`

ConceptTraits extends DimensionTraits so you also get the following properties you can override:

@primitiveTrait({
    type: "string",
    name: "Name",
    description: "Dimension name (human-readable)"
  })
  name?: string;

  @objectArrayTrait({
    type: DimensionOptionTraits,
    idProperty: "id",
    name: "Options",
    description: "Dimension options"
  })
  options?: DimensionOptionTraits[];

  @primitiveTrait({
    type: "string",
    name: "Selected ID",
    description: "Selected Option's ID"
  })
  selectedId?: string;

  @primitiveTrait({
    type: "boolean",
    name: "Allow undefined",
    description: "Allow dimension to be undefined"
  })
  allowUndefined?: boolean;

  @primitiveTrait({
    type: "boolean",
    name: "Disable dimension",
    description: "Hides dimension"
  })
  disable?: boolean;

nf-s · 2020-09-02T05:47:25Z

Some TODOs:

Add EEZ regions for pacific region mapping - https://github.com/PacificCommunity/Terria-PacificMap/blob/master/wwwroot/data/regionMapping.json - this might fix antimeridian issue
Look into weird local terriajs-server proxy issue with api.data.abs.gov.au/dataflow/ - it returns HTTP 400 error:

HTTP/1.1 400 Bad Request
X-Powered-By: Express
Access-Control-Allow-Origin: *
content-type: application/json; charset=utf-8
content-length: 0
date: Wed, 02 Sep 2020 05:36:34 GMT
x-amzn-requestid: 43faaed7-84af-458b-8e09-e865338dd2cf
x-amzn-remapped-content-length: 0
x-amzn-remapped-connection: Keep-Alive
x-amz-apigw-id: SORXXGyGSwMFYaw=
x-amzn-remapped-server: BigIP
x-cache: Error from cloudfront
via: 1.1 aee00c65edc39a007fb4a2fe5f4541d6.cloudfront.net (CloudFront)
x-amz-cf-pop: SYD1-C1
x-amz-cf-id: HCKu6JPsdHohx9V71EPRandj5NYujfJtdj9Q_RmJiBvLMvq82JBDaQ==
Vary: Accept-Encoding
Connection: keep-alive

nf-s · 2020-09-04T08:23:07Z

HTTP 400 issue may be related to lack of IPv6 support.

nf-s · 2020-09-07T13:14:53Z

I might add ColumnTraits to ConceptTraits so you can directly add column style/etc... properties to concepts

nf-s · 2020-09-14T02:38:35Z

Update!

I consider this finished (as a first pass anyway) - I just need to write tests.

There are a few things missing from master implementation:

Ability to select multiple values for a given dimension
Aggregate dimension values (eg sum or show percentage of total)

Other possible features

Add ColumnTraits to ConceptTraits

soyarsauce · 2020-09-14T06:03:23Z

rowan: test some of the ways we use in nationalmap for it - and if we get the same results then we are good.

nick: main difference w/ v7 implementation, we have sdmx groups, instead of creating a catalog item for every single data flow
ideally we just 'browse a sdmx'

@nf-s to update post-Wedneday

nf-s · 2020-09-16T01:06:05Z

Feedback from meeting

- Fix feature info template - so we can see more useful info when you select a region
- Click region to see time series
- ^ sync timeseries with region-mapping (click on chart to show time)
- Sum/Proportion for Concepts (eg sex - show proportion of men/total) - be careful to only allow this for certain concepts
- ABS category schemes incoming
- ~~SDMX from CKAN?~~ NO NO
- EZ pacific region vector tiles

nf-s · 2020-09-16T01:22:17Z

Other thoughts

Start using ABS API key
Compare with National Map SDMX
Get into next pacific map release

nf-s · 2020-09-23T04:23:29Z

I'm going to close this as #4659 has been merged. We can create new issue(s) after we have received more feedback - I will make issues on current feedback

nf-s · 2021-02-04T09:22:18Z

I'm going to re-open this for - NationalMap - Beta release (V8) - Feb 2021 TerriaJS/nationalmap#997

I wanted to create separate tickets, but I didn't have time

nf-s · 2021-02-04T09:25:01Z

Add column title traits for feature info

This is not great:

Maybe write a featureInfoTemplate using region feature info from #5211

nf-s · 2021-02-04T09:25:29Z

Make sure loading icon works when selectors change values

nf-s · 2021-02-04T09:26:15Z

Add way to toggle manual region mapping (in tool menu thing)

nf-s · 2021-02-05T02:49:11Z

objectArrayTrait ordering across strata #4675 Object array trait ordering

nf-s · 2021-02-09T13:59:39Z

Update!

"Export" will now return raw SDMX-CSV data
Add TableMixin time-series chart for region mapping
Add SdmxDataflowStratum.featureInfoTemplate to improve feature info section
Improve SDMX timeseries chart
Fix loader/spinner
Renamed conceptOverrides to modelOverrides - and added codelist override support
Added regionTypeReplacements - so manually override detected regionTypes
Added primary-measure-descriptor modelOverride type - this will set the primary measure title based on selected value of primary measure descriptor dimension
Set time column manually
Fix regionType bugs
Add sdmx data error message

nf-s · 2021-02-15T04:39:43Z

URL to test it out!
http://ci.terria.io/sdmx-fixes-loading/#clean&https://gist.githubusercontent.com/nf-s/1e05f005a93d7b9ab6f1723b8b40d1d2/raw/f74e8c68f849db10cccfa96b05943cdb9cdb635b/abs-sdmx.json

Known dataset issues

RIME by region of arrival by departure, SA4 and above (ASGS 2016), 2016-17 onwards - Not sure how to represent
Consumer Price Index - Time period issue (2020-Q1...)
Population Projections... - GCCSA region issue - region mapping type not supported by Terria
Australian Census Longitudinal Dataset (ACLD): Tenure and landlord type, 2006-2011 - using "urn:sdmx:org.sdmx.infomodel.codelist.Codelist=ABS:CL_ASGS_2011(1.0.0)" but doesn't have region_type dimension

nf-s · 2021-02-25T04:24:51Z

TODO

Region mapping for Land Management Practices Survey, 2011-12

Future Improvements

Chart "expanding"

At the moment, when you expand a chart you don't get any information about dimensions

nf-s · 2021-02-25T04:40:12Z

ABS Dataflow issues

Supply Use Table

UNIT_MEASURE is set to OBS_COMMENT concept.
https://api.data.abs.gov.au/dataflow/ABS/ABS_SU_TABLE_2018?references=all

nf-s · 2021-04-13T12:51:27Z

Issues

"Expanding" charts will overwrite existing charts
Hide date/time selector if no region mapping

nf-s · 2021-05-30T07:44:52Z

Closing again!
Due to #5234

tephenavies added the Version 8 formerly MobX label Jun 24, 2019

kring added this to the Prioritized New Model Architecture Issues milestone Jul 22, 2019

AnaBelgun added the P-PacificMap label Feb 28, 2020

nf-s mentioned this issue Aug 3, 2020

V8: sdmx-json support #4609

Closed

nf-s self-assigned this Aug 3, 2020

AnaBelgun added the P-NationalMap label Aug 8, 2020

AnaBelgun mentioned this issue Aug 8, 2020

NationalMap - Migration to v8 Sep 20-Feb 21 TerriaJS/nationalmap#968

Closed

18 tasks

nf-s mentioned this issue Aug 20, 2020

Implement SdmxCatalogGroup + SdmxCatalogItem #4659

Merged

4 tasks

nf-s linked a pull request Aug 31, 2020 that will close this issue

Implement SdmxCatalogGroup + SdmxCatalogItem #4659

Merged

4 tasks

nf-s closed this as completed Sep 23, 2020

nf-s reopened this Feb 4, 2021

nf-s mentioned this issue Feb 4, 2021

[EPIC] NationalMap - Beta release (V8) TerriaJS/nationalmap#997

Closed

79 tasks

nf-s mentioned this issue Feb 8, 2021

SDMX improvements for NatMap v8 #5222

Closed

2 tasks

nf-s mentioned this issue Feb 11, 2021

SDMX improvements for NatMap v8 #5234

Merged

2 tasks

nf-s added the Epic label Mar 9, 2021

nf-s mentioned this issue Mar 14, 2021

Continuous colour maps #1585

Closed

nf-s mentioned this issue Apr 22, 2021

"Expanding" a chart data series sometimes replaces already active series #5443

Closed

AnaBelgun added the High priority label May 25, 2021

nf-s closed this as completed May 30, 2021

nf-s mentioned this issue May 30, 2021

Model architecture: Add support for continuous color ramps #3441

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Model Architecture: Implement SdmxCatalogItem #3524

Model Architecture: Implement SdmxCatalogItem #3524

tephenavies commented Jun 24, 2019

soyarsauce commented Aug 3, 2020

AnaBelgun commented Aug 8, 2020

nf-s commented Aug 18, 2020 •

edited

Loading

nf-s commented Aug 19, 2020

nf-s commented Aug 21, 2020

nf-s commented Aug 21, 2020

nf-s commented Aug 24, 2020

nf-s commented Aug 24, 2020 •

edited

Loading

nf-s commented Aug 25, 2020

nf-s commented Aug 26, 2020 •

edited

Loading

nf-s commented Aug 28, 2020

nf-s commented Sep 2, 2020

nf-s commented Sep 2, 2020 •

edited

Loading

nf-s commented Sep 2, 2020 •

edited

Loading

nf-s commented Sep 4, 2020

nf-s commented Sep 7, 2020

nf-s commented Sep 14, 2020 •

edited

Loading

soyarsauce commented Sep 14, 2020

nf-s commented Sep 16, 2020 •

edited

Loading

nf-s commented Sep 16, 2020 •

edited

Loading

nf-s commented Sep 23, 2020

nf-s commented Feb 4, 2021

nf-s commented Feb 4, 2021 •

edited

Loading

nf-s commented Feb 4, 2021 •

edited

Loading

nf-s commented Feb 4, 2021

nf-s commented Feb 5, 2021

nf-s commented Feb 9, 2021 •

edited

Loading

nf-s commented Feb 15, 2021 •

edited

Loading

nf-s commented Feb 25, 2021 •

edited

Loading

nf-s commented Feb 25, 2021

nf-s commented Apr 13, 2021 •

edited

Loading

nf-s commented May 30, 2021

Model Architecture: Implement SdmxCatalogItem #3524

Model Architecture: Implement SdmxCatalogItem #3524

Comments

tephenavies commented Jun 24, 2019

soyarsauce commented Aug 3, 2020

AnaBelgun commented Aug 8, 2020

nf-s commented Aug 18, 2020 • edited Loading

SDMX learning materials

Reference

SDMX group structure

Steps:

APIs

Thoughts

Js libs

nf-s commented Aug 19, 2020

nf-s commented Aug 21, 2020

nf-s commented Aug 21, 2020

nf-s commented Aug 24, 2020

SDMX item structure:

nf-s commented Aug 24, 2020 • edited Loading

nf-s commented Aug 25, 2020

nf-s commented Aug 26, 2020 • edited Loading

nf-s commented Aug 28, 2020

nf-s commented Sep 2, 2020

nf-s commented Sep 2, 2020 • edited Loading

Setting up regions

For example SPC - Option 1 above

Another example, ABS - Option 2 above

Options 3 - 5

Other ConceptTraits

nf-s commented Sep 2, 2020 • edited Loading

nf-s commented Sep 4, 2020

nf-s commented Sep 7, 2020

nf-s commented Sep 14, 2020 • edited Loading

Update!

There are a few things missing from master implementation:

Other possible features

soyarsauce commented Sep 14, 2020

nf-s commented Sep 16, 2020 • edited Loading

Feedback from meeting

nf-s commented Sep 16, 2020 • edited Loading

Other thoughts

nf-s commented Sep 23, 2020

nf-s commented Feb 4, 2021

nf-s commented Feb 4, 2021 • edited Loading

nf-s commented Feb 4, 2021 • edited Loading

nf-s commented Feb 4, 2021

nf-s commented Feb 5, 2021

nf-s commented Feb 9, 2021 • edited Loading

Update!

nf-s commented Feb 15, 2021 • edited Loading

nf-s commented Feb 25, 2021 • edited Loading

TODO

Future Improvements

Chart "expanding"

nf-s commented Feb 25, 2021

ABS Dataflow issues

Supply Use Table

nf-s commented Apr 13, 2021 • edited Loading

Issues

nf-s commented May 30, 2021

nf-s commented Aug 18, 2020 •

edited

Loading

nf-s commented Aug 24, 2020 •

edited

Loading

nf-s commented Aug 26, 2020 •

edited

Loading

nf-s commented Sep 2, 2020 •

edited

Loading

Other `ConceptTraits`

nf-s commented Sep 2, 2020 •

edited

Loading

nf-s commented Sep 14, 2020 •

edited

Loading

nf-s commented Sep 16, 2020 •

edited

Loading

nf-s commented Sep 16, 2020 •

edited

Loading

nf-s commented Feb 4, 2021 •

edited

Loading

nf-s commented Feb 4, 2021 •

edited

Loading

nf-s commented Feb 9, 2021 •

edited

Loading

nf-s commented Feb 15, 2021 •

edited

Loading

nf-s commented Feb 25, 2021 •

edited

Loading

nf-s commented Apr 13, 2021 •

edited

Loading