CMX V4.1.0 (mlcommons#1403)

.github/workflows/test-cmxr-mlperf-inference-rgat.yml → .github/workflows/test-cmlcr-mlperf-inference-rgat.yml

@@ -1,4 +1,4 @@
-name: MLPerf inference R-GAT via CMXR
+name: MLPerf inference R-GAT via CMLCR
 
 on:
   pull_request_target:
@@ -34,10 +34,10 @@ jobs:
         pip install tabulate
     - name: Pull MLOps repository
       run: |
-        cmx run mlc pull repo mlcommons@mlperf-automations --branch=main
+        cmlc pull repo mlcommons@mlperf-automations --branch=main
     - name: Test MLPerf Inference R-GAT using ${{ matrix.backend }} on ${{ matrix.os }}
       run: |
-        cmxr --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --adr.inference-src.tags=_branch.dev --pull_changes=yes --pull_inference_changes=yes  --submitter="MLC" --hw_name=gh_${{ matrix.os }}_x86 --model=rgat --implementation=${{ matrix.implementation }} --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --category=datacenter --quiet  --v --target_qps=1
+        cmlcr run,mlperf,inference,generate-run-cmds,_submission,_short --adr.inference-src.tags=_branch.dev --pull_changes=yes --pull_inference_changes=yes  --submitter="MLC" --hw_name=gh_${{ matrix.os }}_x86 --model=rgat --implementation=${{ matrix.implementation }} --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --category=datacenter --quiet  -v --target_qps=1
   
     - name: Push Test MLPerf Results (only for cTuning dev branches)
       if: github.repository_owner == 'ctuning'
@@ -52,4 +52,4 @@ jobs:
         git config --global credential.https://github.com.helper "!gh auth git-credential"
         git config --global credential.https://gist.github.com.helper ""
         git config --global credential.https://gist.github.com.helper "!gh auth git-credential"
-        cmxr --tags=push,github,mlperf,inference,submission --repo_url="https://github.com/ctuning/test_mlperf_inference_submissions" --repo_branch=main --commit_message="Results from R50 GH action on ${{ matrix.os }}" --quiet
+        cmlcr push,github,mlperf,inference,submission --repo_url="https://github.com/ctuning/test_mlperf_inference_submissions" --repo_branch=main --commit_message="Results from R-Gat GH action on ${{ matrix.os }}" --quiet

README.md

@@ -11,7 +11,7 @@
 [![CMX MLPerf inference r-GAT test](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-rgat.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-rgat.yml)
 [![CMX MLPerf inference BERT deepsparse test](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-bert-deepsparse-tf-onnxruntime-pytorch.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-bert-deepsparse-tf-onnxruntime-pytorch.yml)
 
-## Collective Knowledge
+## Collective Knowledge project (CK)
 
 [Collective Knowledge (CK)](https://cKnowledge.org) 
 is a community-driven project dedicated to supporting open science, enhancing reproducible research, 
@@ -21,13 +21,9 @@ across diverse models, data sets, software and hardware:
 
 It includes the following sub-projects.
 
-### Common Metadata eXchange framework (CMX, 2024+)
+### Collective Mind project (MLCommons CM)
 
-*Starting in February 2025, CMX V4+ serves as drop-in, backward-compatible replacement 
- for the earlier Collective Mind framework (CM) and other MLCommons automation prototypes, 
- while providing a simpler and more robust interface.*
-
-The [Common Metadata eXchange framework (CMX)](https://github.com/mlcommons/ck/tree/master/cmx)
+The [Collective Mind automation framework (CM)](https://github.com/mlcommons/ck/tree/master/cmind)
 was developed to support open science and facilitate
 collaborative, reproducible, and reusable research, development, 
 and experimentation based on [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data).
@@ -42,14 +38,22 @@ and technology-agnostic automation workflows, enabling users to
 rerun, reproduce, and reuse complex experimental setups across diverse and rapidly evolving models, datasets,
 software, and hardware. 
 
-For example, CMX helps to modularize, automate and customize MLPerf benchmarks.
+For example, CM helps to modularize, automate and customize MLPerf benchmarks.
 
-See the [project page](https://github.com/mlcommons/ck/tree/master/cmx) for more details.
+#### Legacy CM API and CLI (2021-2024)
+
+See the [project page](https://github.com/mlcommons/ck/blob/master/cm/README.CM.md) for more details.
 
+#### New CM API and CLI (CMX, 2025+)
 
-### Collective Mind framework (CM, 2021-2024)
+[Collective Mind eXtension or Common Metadata eXchange (CMX)](https://github.com/mlcommons/ck/tree/master/cmx) 
+is the next evolution of the [Collective Mind automation framework (MLCommons CM)](https://github.com/mlcommons/ck/tree/master/cm) 
+designed to enhance simplicity, flexibility, and extensibility of automations 
+based on user feedback. It is backwards compatible with CM, released along with CM 
+in the [cmind package](https://pypi.org/project/cmind/) and can serve as drop-in replacement 
+for CM and legacy MLPerf automations while providing a simpler and more robust interface.
 
-See the [project page](https://github.com/mlcommons/ck/blob/master/cm/README.CM.md) for more details.
+See the [project page](https://github.com/mlcommons/ck/tree/master/cmx) for more details.
 
 ### MLOps and MLPerf automations
 
@@ -73,8 +77,8 @@ See the [online MLPerf documentation](https://docs.mlcommons.org/inference)
 at MLCommons to run MLPerf inference benchmarks across diverse systems using CMX.
 Just install `pip install cmind` and substitute the following commands and flags:
 * `cm` -> `cmx`
-* `mlc` -> `cmx run mlc`
-* `mlcr` -> `cmxr`
+* `mlc` -> `cmlc`
+* `mlcr` -> `cmlcr`
 * `-v` -> `--v`
 
 ### Collective Knowledge Playground
@@ -119,7 +123,7 @@ Copyright (c) 2014-2021 cTuning foundation
 
 ## Maintainers
 
-* CM, CM4MLOps and MLPerf automations: [MLCommons infra WG](https://mlcommons.org)
+* Legacy CM, CM4MLOps and MLPerf automations: [MLCommons infra WG](https://mlcommons.org)
 * CMX (the next generation of CM since 2025): [Grigori Fursin](https://cKnowledge.org/gfursin)
 
 ## Concepts
@@ -144,7 +148,7 @@ and [cTuning](https://cTuning.org) for supporting this project,
 as well as our dedicated [volunteers and collaborators](https://github.com/mlcommons/ck/blob/master/CONTRIBUTORS.md)
 for their feedback and contributions!
 
-If you found the CM automations helpful, kindly reference this article:
+If you found the CM, CMX and MLPerf automations helpful, kindly reference this article:
 [ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].
 
 You are welcome to contact the [author](https://cKnowledge.org/gfursin) to discuss long-term plans and potential collaboration.

cm/CHANGES.md

@@ -1,5 +1,7 @@
-## CMX V4.0.2.1
-   - dev version
+## mlc-scripts
+   - added -v flag to print version
+   - improve help
+   - added support for legacy CM front-end for MLPerf (mlc, mlcr, mlcflow)
 
 ## CMX V4.0.2
    - bug fixes

cm/README.CM.md

@@ -22,18 +22,6 @@ an educational community project to learn how to run AI, ML and other emerging w
 in the most efficient and cost-effective way across diverse 
 and ever-evolving systems using the MLPerf benchmarking methodology.
 
-#### CMX automation framework
-
-[CMX](https://github.com/mlcommons/ck/tree/master/cmx) - the next evolution 
-of the [Collective Mind framework](https://github.com/mlcommons/ck/tree/master/cm) 
-designed to enhance simplicity, flexibility, and extensibility of automations 
-based on user feedback. It is backwards compatible with CM, released along with CM 
-in the [cmind package](https://pypi.org/project/cmind/) and can serve as drop-in replacement for CM.
-
-The CM/CMX architecture diagram is available for viewing 
-[here](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png).
-
-
 ## CM internal architecture
 
 The diagram below illustrates the primary classes, functions, and internal automations within the Collective Mind framework:

cm/README.CMX.md

@@ -11,9 +11,9 @@
 [![CMX MLPerf inference r-GAT test](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-rgat.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-rgat.yml)
 [![CMX MLPerf inference BERT deepsparse test](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-bert-deepsparse-tf-onnxruntime-pytorch.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-bert-deepsparse-tf-onnxruntime-pytorch.yml)
 
-# Common Metadata eXchange (CMX)
+# Collective Mind eXtension or Common Metadata eXchange (CMX)
 
-The [Common Metadata eXchange framework (CMX)](https://github.com/mlcommons/ck/tree/master/cmx)
+The [Collective Mind eXtension or Common Metadata eXchange framework (CMX)](https://github.com/mlcommons/ck/tree/master/cmx)
 was developed to support open science and facilitate
 collaborative, reproducible, and reusable research, development, 
 and experimentation based on [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data)
@@ -43,7 +43,7 @@ cmx run script "run-mlperf inference _performance-only _short" --model=resnet50
 cmx show cache
 ```
 
-CMX extends the [Collective Mind (CM) framework](https://zenodo.org/records/8105339),
+CMX extends the [legacy Collective Mind framework (MLCommons CM)](https://zenodo.org/records/8105339),
 which have been successfully validated to 
 [modularize, automate, and modernize MLPerf benchmarks](https://arxiv.org/abs/2406.16791).
 

cm/README.md

@@ -4,9 +4,11 @@
 [![Downloads](https://static.pepy.tech/badge/cmind)](https://pepy.tech/project/cmind)
 [![arXiv](https://img.shields.io/badge/arXiv-2406.16791-b31b1b.svg)](https://arxiv.org/abs/2406.16791)
 
-This Python package contains 2 projects:
-* [Common Metadata eXchange framework (CMX, 2024+)](https://github.com/mlcommons/ck/blob/master/cm/README.CMX.md)
-* [Collective Mind framework (CM, 2021-2024)](https://github.com/mlcommons/ck/blob/master/cm/README.CM.md)
+# Collective Mind workflow automation framework (MLCommons CM)
+
+This Python package contains 2 front-ends:
+* [Collective Mind eXtension or Common Metadata eXchange (CMX, 2024+)](https://github.com/mlcommons/ck/blob/master/cm/README.CMX.md)
+* [Legacy Collective Mind (CM, 2021-2024)](https://github.com/mlcommons/ck/blob/master/cm/README.CM.md)
 
 ## License
 
@@ -41,7 +43,7 @@ To learn more about the concepts and motivation behind this project, please expl
 
 ## Citation
 
-If you found the CM and CMX automations helpful, kindly reference this article:
+If you found the CM, CMX and MLPerf automations helpful, kindly reference this article:
 [ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].
 
 You are welcome to contact the [author](https://cKnowledge.org/gfursin) to discuss long-term plans and potential collaboration.

cm/cmind/__init__.py

@@ -1,4 +1,4 @@
-# Common Metadata eXchange and Collective Mind init
+# Common Metadata eXchange (CMX) and Collective Mind (CM) init
 #
 # Author(s): Grigori Fursin
 # Contributor(s):
@@ -9,7 +9,7 @@
 # White paper: https://arxiv.org/abs/2406.16791
 # Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md
 
-__version__ = "4.0.2.1"
+__version__ = "4.1.0"
 
 from cmind.core import access
 from cmind.core import x

cm/cmind/cli.py

@@ -123,30 +123,49 @@ def run_script(argv = None):
     return run(['run', 'script'] + argv)
 
 ############################################################
-def runx_script(argv = None):
+def run_legacy_mlc(argv = None):
     """
-    Shortcut to "cmx run script ..." or "mlcr ..."
+    Shortcut to legacy CM front-end (mlc, mlcflow)
 
-    CMX command line format:
+    TBD: unify output for CMX
 
-    Args:
-        argv (list | string): command line arguments
+    """
 
-    Returns: 
-        (CM return dict):
+    return run_legacy_mlcflow('mlc', argv)
 
-        * return (int): return code == 0 if no error and >0 if error
-        * (error) (str): error string if return>0
+############################################################
+def run_legacy_mlcr(argv = None):
+    """
+    Shortcut to legacy CM front-end (mlcr, mlcflow)
 
-        * Output from a CM automation action
+    TBD: unify output for CMX
+
+    """
+
+    return run_legacy_mlcflow('mlcr', argv)
+
+############################################################
+def run_legacy_mlcflow(legacy_frontend, argv = None):
+    """
+    Shortcut to legacy CM front-end (mlcr, mlcflow)
+
+    TBD: unify output for CMX
 
     """
 
-    # Access CMX
     if argv is None:
         argv = sys.argv[1:]
 
-    return runx(['run', 'mlcr'] + argv)
+    import subprocess
+
+    rx = subprocess.run([legacy_frontend] + argv)
+
+    return_code = rx.returncode
+
+    r = {'return':0} if return_code == 0 else {'return':return_code, 'error':'legacy CM front-end failed (mlcflow)'}
+
+    return r
+
 
 ############################################################
 def docker_script(argv = None):

cm/cmind/core.py

@@ -1079,7 +1079,7 @@ def _x(self, i, control):
         # Check if asked for "version" and no automation
         if action == 'version' and automation == '':
             automation = 'core'
-        elif action == '' and automation == '' and control.get('version', False):
+        elif action == '' and automation == '' and (control.get('version', False) or control.get('v', False)):
             action = 'version'
             automation = 'core'
         elif action == 'init' and automation == '':
@@ -1111,7 +1111,7 @@ def _x(self, i, control):
                    print ('Control flags:')
                    print ('')
                    print ('  -h | -help - print this help')
-                   print ('  -version - print version')
+                   print ('  -v | -version - print version')
                    print ('  -out (default) - output to console')
                    print ('  -out=con (default) - output to console')
                    print ('  -j | -json - print output of the automation action to console as JSON')
@@ -1455,23 +1455,33 @@ def _x(self, i, control):
         # Process artifacts for this automation action
         if len(artifacts)>0:
             parsed_artifacts = []
+            unparsed_artifacts = []
 
             for extra_artifact in artifacts:
                 # Parse artifact
                 r = parse_cm_object_and_check_current_dir(self, extra_artifact)
-                if r['return'] >0 : return r
-
-                parsed_artifacts.append(r['cm_object'])
-
+# Adding compatibility for the legacy cmlcr
+#                if r['return'] >0 : return r
+                if r['return'] == 0 : 
+                    parsed_artifacts.append(r['cm_object'])
+                else:
+                    unparsed_artifacts.append(extra_artifact)
+
             control['_parsed_artifacts'] = parsed_artifacts
 
+            if len(unparsed_artifacts)>0:
+                control['_unparsed_artifacts'] = unparsed_artifacts
+
         # Check artifact and artifacts
         if artifact != '':
             # Parse artifact
             r = parse_cm_object_and_check_current_dir(self, artifact)
-            if r['return'] >0 : return r
-
-            control['_parsed_artifact'] = r['cm_object']
+# Adding compatibility for the legacy cmlcr
+#            if r['return'] >0 : return r
+            if r['return'] == 0 : 
+                control['_parsed_artifact'] = r['cm_object']
+            else:
+                control['_unparsed_artifact'] = artifact
 
         # Check min CM version requirement
         min_cm_version = automation_meta.get('min_cm_version','').strip()

cm/cmind/repo/automation/automation/_cm.json

@@ -3,7 +3,7 @@
   "automation_alias": "automation",
   "automation_uid": "bbeb15d8f0a944a4",
   "desc": "Managing CM automations",
-  "developers": "[Grigori Fursin](https://cKnowledge.org/gfursin)",
+  "authors": "[Grigori Fursin](https://cKnowledge.org/gfursin)",
   "sort": -2000,
   "tags": [
     "automation"

cm/cmind/repo/automation/ckx/_cm.json

@@ -3,7 +3,7 @@
   "automation_alias": "automation",
   "automation_uid": "bbeb15d8f0a944a4",
   "desc": "Accessing legacy CK automations",
-  "developers": "[Grigori Fursin](https://cKnowledge.org/gfursin)",
+  "authors": "[Grigori Fursin](https://cKnowledge.org/gfursin)",
   "sort": -1000,
   "tags": [
     "automation",

cm/cmind/repo/automation/core/_cm.json

@@ -2,8 +2,8 @@
   "alias": "core",
   "automation_alias": "automation",
   "automation_uid": "bbeb15d8f0a944a4",
-  "desc": "Accessing some core CM functions",
-  "developers": "[Grigori Fursin](https://cKnowledge.org/gfursin)",
+  "desc": "Core CM functions",
+  "authors": "[Grigori Fursin](https://cKnowledge.org/gfursin)",
   "sort": 500,
   "tags": [
     "automation",

cm/cmind/repo/automation/mlc/_cm.json

@@ -2,11 +2,13 @@
   "alias": "mlc",
   "automation_alias": "automation",
   "automation_uid": "bbeb15d8f0a944a4",
-  "desc": "CMX interface for mlcflow",
+  "desc": "CMX interface for legacy CM front-ends (mlc, mlcr, mlcflow)",
+  "authors": "[Grigori Fursin](https://cKnowledge.org/gfursin)",
   "sort": -1000,
   "tags": [
     "automation",
-    "mlc"
+    "mlc",
+    "mlcflow"
   ],
   "uid": "26193c384fae476c"
 }

cm/cmind/repo/automation/mlc/module.py

@@ -1,4 +1,13 @@
-# CMX interface for mlcflow
+# CMX interface for legacy CM front-ends (mlc, mlcr, mlcflow)
+
+# Authors: Grigori Fursin
+# Contributors:
+#
+# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt
+# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md
+# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md
+# White paper: https://arxiv.org/abs/2406.16791
+# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md
 
 import os
 
@@ -8,7 +17,7 @@
 
 class CAutomation(Automation):
     """
-    CMX interface for mlcflow
+    CMX interface for legacy CM front-ends (mlc, mlcr, mlcflow)
     """
 
     ############################################################
@@ -18,7 +27,7 @@ def __init__(self, cmind, automation_file):
     ############################################################
     def run(self, i):
         """
-        CMX interface for mlcflow
+        CMX interface for legacy CM front-ends (mlc, mlcr, mlcflow)
         
         """