Incorproate linkml-runtime, linkml-model/linkml-files.py changes from linkml-runtime #201

sierra-moxon · 2024-07-15T23:56:09Z

back-porting changes in the runtime PR to linkml-model (current SoT for linkml-model/linkml_files.py methods).

this PR was a result of my dive into converting schemaview (linkml-runtime) tests to pytest when I ran into an error in linkml_files.py and test_linkml_files.py in linkml-runtime/linkml-model. It might be that we want to reconsider vendoring.

my biggest concern here with this PR, is that the tests for linkml-model and linkml-runtime that exercise linkml_files.py were different. If I put back in (as I did with this PR), the tests currently in linkml-model, there are slight changes in resulting expected URLs for model files - the new URLs do resolve, so I've adjusted the tests. We likely need @cmungall's review here before merge.

…pdate in linkml-runtime stays the same

…s the change in paths to expect here. noting the 'new' paths do resolve.

sierra-moxon · 2024-07-16T01:00:29Z

tests/test_linkml_files.py

+        GITHUB_PATH_FOR(Source.META, Format.NATIVE_JSONLD, "v0.0.1")
+    current_loc = re.sub(r'linkml-model/[0-9a-f]*/', 'linkml-model/SHA/', GITHUB_PATH_FOR(Source.TYPES, Format.YAML))
+    assert "https://raw.githubusercontent.com/linkml/linkml-model/SHA/model/schema/types.yaml" == current_loc
+    assert 'https://raw.githubusercontent.com/linkml/linkml-model/missing_branch/linkml_model/owl/mappings.owl.ttl' == \


note: change in path here

the thing more worthy of fixing to me here is /SHA/model and missing_branch/linkml_model having different directory structure, but if the URLs resolve and they give us the thing we were after then they are the correct URLs in my book!

sneakers-the-rat

Basically i feel like i have already spent too much time for the three of us overengineering this thing, and am fine with whatever as long as the paths work ;) some minor suggestions on test semantics but take em or leave em

sneakers-the-rat · 2024-07-16T02:46:40Z

tests/test_linkml_files.py

+def test_fixed_meta_url():
+    """
+    One fixed canary value - the METAMODEL_URI as used in linkml main shouldn't change
+    """
+    assert URL_FOR(Source.META, Format.YAML) == 'https://w3id.org/linkml/meta.yaml'
+    assert URL_FOR(Source.META, Format.JSONLD) == 'https://w3id.org/linkml/meta.context.jsonld'
+

 root_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "linkml_model"))

-SKIP_GITHUB_API = False              # True means don't do the github API tests
-
-
-class LinkMLFilesTestCase(unittest.TestCase):
-    """ Test that linkml_model/linkml_files.py work """
-    def test_basic_rules(self):
-        self.assertEqual("https://w3id.org/linkml/annotations.yaml",
-                         URL_FOR(Source.ANNOTATIONS, Format.YAML))
-        self.assertEqual("https://w3id.org/linkml/meta.model.context.jsonld",
-                         URL_FOR(Source.META, Format.NATIVE_JSONLD))
-        self.assertEqual(os.path.join(root_path, "model/schema/meta.yaml"),
-                         LOCAL_PATH_FOR(Source.META, Format.YAML))
-        print(LOCAL_PATH_FOR(Source.META, Format.YAML))
-        self.assertEqual(os.path.join(root_path, "jsonld/types.model.context.jsonld"),
-                         LOCAL_PATH_FOR(Source.TYPES, Format.NATIVE_JSONLD))
-        self.assertEqual("https://linkml.github.io/linkml-model/model/schema/meta.yaml",
-                         GITHUB_IO_PATH_FOR(Source.META, Format.YAML))
-        self.assertEqual("https://linkml.github.io/linkml-model/jsonld/types.model.context.jsonld",
-                         GITHUB_IO_PATH_FOR(Source.TYPES, Format.NATIVE_JSONLD))
-        self.assertEqual("https://raw.githubusercontent.com/linkml/linkml-model/main/jsonld/meta.model.context.jsonld",
-                         GITHUB_PATH_FOR(Source.META, Format.NATIVE_JSONLD, ReleaseTag.LATEST))
-        self.assertEqual("https://raw.githubusercontent.com/linkml/linkml-model/testing_branch/owl/mappings.owl.ttl",
-                         GITHUB_PATH_FOR(Source.MAPPINGS, Format.OWL, branch="testing_branch"))
-
-    @unittest.skipIf(SKIP_GITHUB_API, "Github API tests skipped")
-    def test_github_specific_rules(self):
-        """
-        Test accesses that require github API to access
+
+def test_basic_rules():
+    assert "https://w3id.org/linkml/annotations.yaml" == URL_FOR(Source.ANNOTATIONS, Format.YAML)
+    assert "https://w3id.org/linkml/meta.context.jsonld" == URL_FOR(Source.META, Format.NATIVE_JSONLD)
+    assert os.path.join(root_path, "model/schema/meta.yaml") == LOCAL_PATH_FOR(Source.META, Format.YAML)
+    print(LOCAL_PATH_FOR(Source.META, Format.YAML))
+    assert os.path.join(root_path, "jsonld/types.context.jsonld") == LOCAL_PATH_FOR(Source.TYPES, Format.NATIVE_JSONLD)
+    assert "https://linkml.github.io/linkml-model/latest/linkml_model/model/schema/meta.yaml" == GITHUB_IO_PATH_FOR(
+        Source.META, Format.YAML)
+    assert "https://linkml.github.io/linkml-model/latest/linkml_model/jsonld/types.context.jsonld" == \
+           GITHUB_IO_PATH_FOR(Source.TYPES, Format.NATIVE_JSONLD)
+    assert "https://raw.githubusercontent.com/linkml/linkml-model/main/linkml_model/jsonld/meta.context.jsonld" == \
+           GITHUB_PATH_FOR(Source.META, Format.NATIVE_JSONLD, ReleaseTag.LATEST)
+    assert "https://raw.githubusercontent.com/linkml/linkml-model/testing_branch/linkml_model/owl/mappings.owl.ttl" == \
+           GITHUB_PATH_FOR(Source.MAPPINGS, Format.OWL, branch="testing_branch")


test_basic_rules is basically the same thing as test_fixed_meta_url - in general i only test fixed values when there is a specific reason for it to be fixed forever, so that's why i only tested those two URLs, but i think if we're going to test the rest of these then they should be in a test structure that reflects the meaning of the test :)

test each of these functions (URL_FOR, LOCAL_PATH_FOR, GITHUB_IO_PATH_FOR and GITHUB_PATH_FOR) separately, and some of what we want to test for might already be covered by other tests.

URL_FOR, LOCAL_PATH_FOR, and GITHUB_IO_PATH_FOR are trivial string formatting functions, so they can be covered by unit-style tests.

LOCAL_PATH_FOR has a test_local_paths test for existence for all the files in the expected formats already.

URL_FOR has test_url_for_format that tests the resolution of the generated URLs for the formats that have rewrite rules in the w3id repo.

GITHUB_PATH_FOR is substantially different than the others and should be tested separately for sure,

ideally these would have docstrings about why they should be fixed forever (if they were just fixed forever, why wouldn't we just hardcode the strings in the module itself and call it a day?), but idrc about that ;)

sneakers-the-rat · 2024-07-16T02:47:49Z

tests/test_linkml_files.py

+           GITHUB_PATH_FOR(Source.MAPPINGS, Format.OWL, branch="testing_branch")
+
+
+SKIP_GITHUB_API = False


ideally this would be a pytest option or dependent on some flag, if we want to keep skipping them - i marked them as skip because they kept failing bc the ratelimits on github API were very low, but if we want to rescue them then yes a flag that can be accessed would be good :)

sneakers-the-rat · 2024-07-16T02:50:04Z

tests/test_linkml_files.py

+        GITHUB_PATH_FOR(Source.META, Format.NATIVE_JSONLD, "v0.0.1")
+    current_loc = re.sub(r'linkml-model/[0-9a-f]*/', 'linkml-model/SHA/', GITHUB_PATH_FOR(Source.TYPES, Format.YAML))
+    assert "https://raw.githubusercontent.com/linkml/linkml-model/SHA/model/schema/types.yaml" == current_loc
+    assert 'https://raw.githubusercontent.com/linkml/linkml-model/missing_branch/linkml_model/owl/mappings.owl.ttl' == \


the thing more worthy of fixing to me here is /SHA/model and missing_branch/linkml_model having different directory structure, but if the URLs resolve and they give us the thing we were after then they are the correct URLs in my book!

sneakers-the-rat · 2024-07-16T02:53:49Z

pyproject.toml

@@ -33,11 +33,13 @@ packages = [
 [tool.poetry.dependencies]
 python = "^3.9"
 linkml-runtime = "^1.6.3"
+rdflib = "^7.0.0"


oof if we are gaining rdflib as a runtime dependency just to use the Namespace class to declare a module constant i woudl call that highly undesirable and worthy of thinking about whether we need that

sierra-moxon added 2 commits July 15, 2024 10:58

port over changes in linkml-runtime/linkml-model/linkml_files.py

d6fa9a1

port linkml-runtime test changes to linkml-model so that make model-u…

db2f648

…pdate in linkml-runtime stays the same

sierra-moxon mentioned this pull request Jul 15, 2024

first draft conversion of unittest syntax to pytest syntax for schema… linkml/linkml-runtime#328

Draft

sierra-moxon added 4 commits July 15, 2024 16:57

poetry lock --no-update

1b6598c

reduce aggressive new testing of later python versions

d3f07f2

fixing tests

469b58d

fix tests according to changes in linkml-runtime: for consideration i…

e2b031b

…s the change in paths to expect here. noting the 'new' paths do resolve.

sierra-moxon commented Jul 16, 2024

View reviewed changes

sierra-moxon marked this pull request as ready for review July 16, 2024 01:02

sierra-moxon requested review from cmungall and sneakers-the-rat July 16, 2024 01:02

sneakers-the-rat reviewed Jul 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorproate linkml-runtime, linkml-model/linkml-files.py changes from linkml-runtime #201

Incorproate linkml-runtime, linkml-model/linkml-files.py changes from linkml-runtime #201

sierra-moxon commented Jul 15, 2024 •

edited

Loading

sierra-moxon Jul 16, 2024

sneakers-the-rat Jul 16, 2024

sneakers-the-rat left a comment

sneakers-the-rat Jul 16, 2024

sneakers-the-rat Jul 16, 2024

sneakers-the-rat Jul 16, 2024

sneakers-the-rat Jul 16, 2024

		GITHUB_PATH_FOR(Source.MAPPINGS, Format.OWL, branch="testing_branch")


		SKIP_GITHUB_API = False

Incorproate linkml-runtime, linkml-model/linkml-files.py changes from linkml-runtime #201

Are you sure you want to change the base?

Incorproate linkml-runtime, linkml-model/linkml-files.py changes from linkml-runtime #201

Conversation

sierra-moxon commented Jul 15, 2024 • edited Loading

sierra-moxon Jul 16, 2024

Choose a reason for hiding this comment

sneakers-the-rat Jul 16, 2024

Choose a reason for hiding this comment

sneakers-the-rat left a comment

Choose a reason for hiding this comment

sneakers-the-rat Jul 16, 2024

Choose a reason for hiding this comment

sneakers-the-rat Jul 16, 2024

Choose a reason for hiding this comment

sneakers-the-rat Jul 16, 2024

Choose a reason for hiding this comment

sneakers-the-rat Jul 16, 2024

Choose a reason for hiding this comment

sierra-moxon commented Jul 15, 2024 •

edited

Loading