Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: first-pass at handling "download/<label>:: <url>" as an array #45

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions manager/metadata/records/sdoh-indices.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"created_by": "",
"last_modified_by": "Admin",
"filled": 20,
"to_fill": 25,
"filled_pct": 80,
"to_fill": 26,
"filled_pct": 76,
"progress_class": "warning"
},
"id": "herop-yvjqrx",
Expand Down Expand Up @@ -83,11 +83,11 @@
"file_size": null,
"references": {
"http://schema.org/url": "https://sdohatlas.github.io/",
"http://schema.org/downloadUrl": "[{'url': 'https://geodacenter.github.io/data-and-lab/data/us-sdoh-2014.zip', 'label': 'ZIP Archive'}]"
"http://schema.org/downloadUrl": "[{\"label\": \"ZIP Archive\", \"url\": \"https://geodacenter.github.io/data-and-lab/data/us-sdoh-2014.zip\"}]"
},
"wxs_indentifier": null,
"identifier": null,
"modified": "2025-02-03T21:05:47Z",
"modified": "2025-02-20T21:26:18Z",
"metadata_version": "SDOH PlaceProject",
"suppressed": false,
"spatial_resolution": [
Expand Down Expand Up @@ -139,6 +139,9 @@
"SDOH_CL - 1:rural affordable, 2:suburban affluent, 3:suburban affordable, 4:extreme poverty, 5:multilingual working, 6:urban core opportunity, 7:sparse areas"
],
"highlight_ids": null,
"featured_variable": "Socioeconomic Advantage Index, Limited Mobility Index, Urban Core Opportunity Index, Mixed Immigrant, Cohesion, & Access Index, SDOH Cluster - 1:rural affordable, 2:suburban affluent, 3:suburban affordable, 4:extreme poverty, 5:multilingual working, 6:urban core opportunity, 7:sparse areas (see https://sdohatlas.github.io/)",
"data_usage_notes": null
"featured_variable": [
"Socioeconomic Advantage Index, Limited Mobility Index, Urban Core Opportunity Index, Mixed Immigrant, Cohesion, & Access Index, SDOH Cluster - 1:rural affordable, 2:suburban affluent, 3:suburban affordable, 4:extreme poverty, 5:multilingual working, 6:urban core opportunity, 7:sparse areas (see https://sdohatlas.github.io/)"
],
"data_usage_notes": null,
"preferred_citation": null
}
29 changes: 27 additions & 2 deletions manager/registry.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import json
import logging
from json import JSONDecodeError
from pathlib import Path
from datetime import datetime
from dotenv import load_dotenv
Expand Down Expand Up @@ -133,11 +135,20 @@ def save_data(self, index=False):

if self.data["references"]:
cleaned_references = {}
download_refs = []
for (
k,
v,
) in self.data["references"].items():
cleaned_references[k.rstrip().lstrip()] = v.rstrip().lstrip()
if k.startswith('download/'):
label = k.rstrip().lstrip()[9:]
url = v.rstrip().lstrip()
download_refs.append({'label': label, 'url': url})
else:
cleaned_references[k.rstrip().lstrip()] = v.rstrip().lstrip()

if len(download_refs) > 0:
cleaned_references['http://schema.org/downloadUrl'] = json.dumps(download_refs)
self.data["references"] = cleaned_references

coverages = (
Expand Down Expand Up @@ -206,6 +217,7 @@ def to_json(self):

return self.data


def to_form(self):
"""Prepares the raw backend data to populate an html form."""
form_data = {}
Expand All @@ -216,7 +228,20 @@ def to_form(self):
if key == "references" and isinstance(value, dict):
lines = ""
for x, y in value.items():
lines += f"{x}:: {y}\n"
logging.warning(f'item - {x}:: {y}')
if x == 'http://schema.org/downloadUrl':
try:
y_parsed = json.loads(y)
if isinstance(y_parsed, list):
# downloadUrl is a list of objects defining label + url
# break it up into multiple lines of download/<label>:: <url>
for u in y_parsed:
lines += f"download/{u['label']}:: {u['url']}\n"
except JSONDecodeError as ex:
# downloadUrl is a single string
lines += f"{x}:: {y}\n"
else:
lines += f"{x}:: {y}\n"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, what if in the except block, we force a transformation to the new pattern? Something like

lines += f"download/Data Download:: {y}"

That way, when a form is opened for an old record, it will auto-populate the old download link into the new format, and then get saved appropriately. We can have students go through all the records then an check the download links, which will fix them.

Does that make sense?

value = lines
if field.multiple and isinstance(value, list):
if field.widget == "text-area.html":
Expand Down
2 changes: 2 additions & 0 deletions manager/templates/crud/widgets/references.html
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

{% block note %}
put multiple references on separate lines, with this format: <code>&lt;reference uri&gt;:: &lt;value&gt;</code> where <code>reference uri</code> is one of <a href="https://opengeometadata.org/reference-uris/" target="_blank">these options &nearr;</a>
<br>
specify multiple download formats using <code>download/&lt;label&gt;:: &lt;url&gt;</code> where <code>label</code> is a unique string, such as <code>CSV</code>, <code>JSON</code>, <code>XML</code>, etc.
{% endblock %}

{% block input %}
Expand Down