From 8636fa50da3be7e67d513460d6066570a0d74867 Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Thu, 13 Feb 2025 18:13:10 +0100 Subject: [PATCH] Improving documentation --- README.md | 19 ++ cm/README.CMX.md | 16 +- cmx/README.md | 57 +++-- cmx/commands.md | 7 + cmx/getting-started.md | 3 - cmx/mlperf-inference/v4.1/README.md | 1 + cmx/mlperf-inference/v5.0/README.md | 1 + cmx/motivation.md | 81 +------ cmx/understanding-cmx.md | 204 ++++++++++++++++++ cmx4mlops/README.md | 4 +- cmx4mlops/cmx4mlops/README.md | 0 cmx4mlops/cmx4mlops/cmr.yaml | 4 +- cmx4mlops/cmx4mlops/repo/README.md | 0 .../_cm.yaml | 10 +- mkdocs.yml | 6 +- 15 files changed, 293 insertions(+), 120 deletions(-) create mode 100644 cmx/commands.md delete mode 100644 cmx/getting-started.md create mode 100644 cmx/mlperf-inference/v4.1/README.md create mode 100644 cmx/mlperf-inference/v5.0/README.md create mode 100644 cmx/understanding-cmx.md delete mode 100644 cmx4mlops/cmx4mlops/README.md delete mode 100644 cmx4mlops/cmx4mlops/repo/README.md diff --git a/README.md b/README.md index 6760da50a6..d3067691c2 100755 --- a/README.md +++ b/README.md @@ -23,6 +23,23 @@ It includes the following sub-projects. ### Common Metadata eXchange framework (CMX, 2024+) +The [Common Metadata eXchange framework (CMX)](https://github.com/mlcommons/ck/tree/master/cmx) +was developed to support open science and facilitate +collaborative, reproducible, and reusable research, development, +and experimentation based on [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data). + +It helps users non-intrusively convert their software projects +into file-based repositories of portable and reusable artifacts +(code, data, models, scripts) with extensible metadata, +a unified command-line interface, and a simple Python API. + +Such artifacts can be easily chained together into portable +and technology-agnostic automation workflows, enabling users to +rerun, reproduce, and reuse complex experimental setups across diverse and rapidly evolving models, datasets, +software, and hardware. + +For example, CMX helps to modularize, automate and customize MLPerf benchmarks. + See the [project page](https://github.com/mlcommons/ck/tree/master/cmx) for more details. ### Collective Mind framework (CM, 2021-2024) @@ -124,3 +141,5 @@ for their feedback and contributions! If you found the CM automations helpful, kindly reference this article: [ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. + +You are welcome to contact the [author](https://cKnowledge.org/gfursin) to discuss long-term plans and potential collaboration. diff --git a/cm/README.CMX.md b/cm/README.CMX.md index 1151bc6354..1daba53cce 100644 --- a/cm/README.CMX.md +++ b/cm/README.CMX.md @@ -16,7 +16,8 @@ The [Common Metadata eXchange framework (CMX)](https://github.com/mlcommons/ck/tree/master/cmx) was developed to support open science and facilitate collaborative, reproducible, and reusable research, development, -and experimentation based on [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data). +and experimentation based on [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data) +and the [Collective Knowledge concept](https://learning.acm.org/techtalks/reproducibility). It helps users non-intrusively convert their software projects, directories, and Git(Hub) repositories into file-based repositories @@ -24,10 +25,12 @@ of portable and reusable artifacts (code, data, models, scripts) with extensible metadata, a unified command-line interface, and a simple Python API. -Such artifacts can be easily chained together into portable automation -workflows, enabling users to rerun, reproduce, and reuse complex -experimental setups across diverse and rapidly evolving models, datasets, -software, and hardware. +Such artifacts can be easily chained together into portable and technology-agnostic automation workflows, +enabling users to rerun, reproduce, and reuse complex experimental setups across diverse and rapidly +evolving models, datasets, software, and hardware. + +Such workflows, in turn, can be easily integrated with CI/CD pipelines and GitHub Actions +and used to create powerful, portable, modular and GUI-based applications. For example, you can run image classification and the MLPerf inference benchmark on Linux, macOS, and Windows using a few CMX commands as follows: @@ -40,8 +43,7 @@ cmx run script "run-mlperf inference _performance-only _short" --model=resnet50 cmx show cache ``` -CMX extends the [Collective Knowledge (CK)](https://learning.acm.org/techtalks/reproducibility) -and [Collective Mind (CM)](https://zenodo.org/records/8105339) concepts, +CMX extends the [Collective Mind (CM) framework](https://zenodo.org/records/8105339), which have been successfully validated to [modularize, automate, and modernize MLPerf benchmarks](https://arxiv.org/abs/2406.16791). diff --git a/cmx/README.md b/cmx/README.md index e622f71d53..7bf801211a 100644 --- a/cmx/README.md +++ b/cmx/README.md @@ -1,20 +1,22 @@ # Common Metadata eXchange (CMX) -The [Common Metadata eXchange framework (CMX)](https://github.com/mlcommons/ck/tree/master/cmx) -was developed to support open science and facilitate +We are developing the [Common Metadata eXchange framework (CMX)](https://github.com/mlcommons/ck/tree/master/cmx) +to support open science and facilitate collaborative, reproducible, and reusable research, development, -and experimentation based on [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data). +and experimentation based on [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data) +and the [Collective Knowledge concept](https://learning.acm.org/techtalks/reproducibility). -It helps users non-intrusively convert their software projects, -directories, and Git(Hub) repositories into file-based repositories -of portable and reusable artifacts (code, data, models, scripts) -with extensible metadata, a unified command-line interface, -and a simple Python API. +It helps users non-intrusively convert their software projects +into file-based repositories of portable and reusable artifacts +(code, data, models, scripts) with extensible metadata, +a unified command-line interface, and a simple Python API. -Such artifacts can be easily chained together into portable automation -workflows, enabling users to rerun, reproduce, and reuse complex -experimental setups across diverse and rapidly evolving models, datasets, -software, and hardware. +Such artifacts can be easily chained together into portable and technology-agnostic automation workflows, +enabling users to rerun, reproduce, and reuse complex experimental setups across diverse and rapidly +evolving models, datasets, software, and hardware. + +Such workflows, in turn, can be easily integrated with CI/CD pipelines and GitHub Actions +and used to create powerful, portable, modular and GUI-based applications. For example, you can run image classification and the MLPerf inference benchmark on Linux, macOS, and Windows using a few CMX commands as follows: @@ -27,8 +29,7 @@ cmx run script "run-mlperf inference _performance-only _short" --model=resnet50 cmx show cache ``` -CMX extends the [Collective Knowledge (CK)](https://learning.acm.org/techtalks/reproducibility) -and [Collective Mind (CM)](https://zenodo.org/records/8105339) concepts, +CMX extends the [Collective Mind (CM) framework](https://zenodo.org/records/8105339), which have been successfully validated to [modularize, automate, and modernize MLPerf benchmarks](https://arxiv.org/abs/2406.16791). @@ -67,16 +68,16 @@ Collective Mind (CM) in the Python cmind package: *Under preparation* * [Installation (Linux, Windows, MacOS)](install.md) -* [Getting Started Guide](getting-started.md) -* [MLOps, DevOps and MLPerf automations](https://access.cknowledge.org/playground/?action=scripts) -* [High-level architecture](architecture-4.0.0.png) -* [Python API](https://cknowledge.org/docs/cmx) +* CMX Guide: + * [Understanding CMX](understanding-cmx.md) + * [CMX commands to share and reuse artifacts](commands.md) + * [CMX automation commands](cmx-automations.md) + * [Reusing CMX automations and artifacts for MLOps, DevOps and MLPerf](cmx4mlops.md) +* [CMX Python API](https://cknowledge.org/docs/cmx) +* [CMX internal architecture](architecture-4.0.0.png) * [Motivation](motivation.md) - - - ## Author [Grigori Fursin](https://cKnowledge.org/gfursin). @@ -94,17 +95,9 @@ Copyright (c) 2024-2025 MLCommons Grigori Fursin and the cTuning foundation donated this project to MLCommons to benefit everyone. -## Concepts - -To learn more about the motivation behind this project, please explore the following articles and presentations: - -* HPCA'25 article "MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI": [ [Arxiv](https://arxiv.org/abs/2410.12032) ], [ [tutorial to reproduce results using CM/CMX](https://github.com/aryatschand/MLPerf-Power-HPCA-2025/blob/main/measurement_tutorial.md) ] -* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] -* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] -* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] -* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ] - ## Citation -If you found the CMX automations helpful, kindly reference this article: +If you found the CM/CMX automations for MLOps, DevOps and MLPerf helpful, kindly reference this article: [ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. + +You are welcome to contact the [author](https://cKnowledge.org/gfursin) to discuss long-term plans and potential collaboration. diff --git a/cmx/commands.md b/cmx/commands.md new file mode 100644 index 0000000000..804aded33d --- /dev/null +++ b/cmx/commands.md @@ -0,0 +1,7 @@ +[ [Back to documentation](README.md) ] + +# CMX commands + +## Command Line + +## Python API diff --git a/cmx/getting-started.md b/cmx/getting-started.md deleted file mode 100644 index 332c1ecb3e..0000000000 --- a/cmx/getting-started.md +++ /dev/null @@ -1,3 +0,0 @@ -[ [Back to documentation](README.md) ] - -TBD diff --git a/cmx/mlperf-inference/v4.1/README.md b/cmx/mlperf-inference/v4.1/README.md new file mode 100644 index 0000000000..a0990367ef --- /dev/null +++ b/cmx/mlperf-inference/v4.1/README.md @@ -0,0 +1 @@ +TBD diff --git a/cmx/mlperf-inference/v5.0/README.md b/cmx/mlperf-inference/v5.0/README.md new file mode 100644 index 0000000000..a0990367ef --- /dev/null +++ b/cmx/mlperf-inference/v5.0/README.md @@ -0,0 +1 @@ +TBD diff --git a/cmx/motivation.md b/cmx/motivation.md index e265360ccc..0227ad38f5 100644 --- a/cmx/motivation.md +++ b/cmx/motivation.md @@ -1,77 +1,14 @@ -[ [Back to index](README.md) ] +[ [Back to documentation](README.md) ] -Introduction to the MLCommons Collective Mind (CM) workflow automation framework and its new version, Common Metadata eXchange (CMX). +# CK/CM/CMX motivation -## Introduction +To learn more about the concepts and motivation behind this project, please explore the following articles and presentations: -During the past 10 years, the community has considerably improved -the reproducibility of experimental results from research projects and published papers -by introducing the [artifact evaluation process](https://cTuning.org/ae) -with a [unified artifact appendix and reproducibility checklists](https://github.com/mlcommons/ck/blob/master/docs/artifact-evaluation/checklist.md), -Jupyter notebooks, containers, and Git repositories. +* HPCA'25 article "MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI": [ [Arxiv](https://arxiv.org/abs/2410.12032) ], [ [tutorial to reproduce results using CM/CMX](https://github.com/aryatschand/MLPerf-Power-HPCA-2025/blob/main/measurement_tutorial.md) ] +* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] +* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] +* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] +* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ] -On the other hand, [our experience reproducing more than 150 papers](https://www.youtube.com/watch?v=7zpeIVwICa4) -revealed that it still takes weeks and months of painful and -repetitive interactions between researchers and evaluators to reproduce experimental results. +You are welcome to contact the [author](https://cKnowledge.org/gfursin) to discuss long-term plans and potential collaboration. -This effort includes decrypting numerous README files, examining ad-hoc artifacts -and containers, and figuring out how to reproduce computational results. -Furthermore, snapshot containers pose a challenge to optimize algorithms' performance, -accuracy, power consumption and operational costs across diverse -and rapidly evolving software, hardware, and data used in the real world. - -![](https://raw.githubusercontent.com/ctuning/ck-guide-images/master/cm-ad-hoc-projects.png) - -This practical experience and the feedback from the community motivated -us to establish the [MLCommons Task Force on Automation and Reproducibility](taskforce.md) -and develop a light-weight, technology agnostic, and English-like -workflow automation language called Collective Mind (MLCommons CM). - -This language provides a common, non-intrusive and human-readable interface to any software project -transforming it into a collection of [reusable automation recipes (CM scripts)]( https://github.com/mlcommons/ck/tree/master/cm-mlops/script ). -Following [FAIR principles](https://www.go-fair.org/fair-principles), CM automation actions and scripts -are simple wrappers around existing user scripts and artifacts to make them -* findable via human-readable tags, aliases and unique IDs; -* accessible via a unified CM CLI and Python API with JSON/YAML meta descriptions; -* interoperable and portable across any software, hardware, models and data; -* reusable across all projects. - -CM is written in simple Python and uses JSON and/or YAML meta descriptions with a unified CLI -to minimize the learning curve and help researchers and practitioners describe, share, and reproduce experimental results -in a unified, portable, and automated way across any rapidly evolving software, hardware, and data -while solving the "dependency hell" and automatically generating unified README files and modular containers. - -![](https://raw.githubusercontent.com/ctuning/ck-guide-images/master/cm-unified-projects.png) - -Our ultimate goal is to use CM language to facilitate reproducible research for AI, ML and systems projects, -minimize manual and repetitive benchmarking and optimization efforts, -and reduce time and costs when transferring technology to production -across continuously changing software, hardware, models, and data. - - -## Some projects supported by CM - -* [A unified way to run MLPerf inference benchmarks with different models, software and hardware](mlperf/inference). See [current coverage](https://github.com/mlcommons/ck/issues/1052). -* [A unitied way to run MLPerf training benchmarks](tutorials/reproduce-mlperf-training.md) *(prototyping phase)* -* [A unified way to run MLPerf tiny benchmarks](tutorials/reproduce-mlperf-tiny.md) *(prototyping phase)* -* A unified CM to run automotive benchmarks *(prototyping phase)* -* [An open-source platform to aggregate, visualize and compare MLPerf results](https://access.cknowledge.org/playground/?action=experiments) - * [Leaderboard for community contributions](https://access.cknowledge.org/playground/?action=contributors) -* [Artifact Evaluation and reproducibility initiatives](https://cTuning.org/ae) at ACM/IEEE/NeurIPS conferences: - * [A unified way to run experiments and reproduce results from ACM/IEEE MICRO'23 and ASPLOS papers](https://github.com/ctuning/cm4research) - * [Student Cluster Competition at SuperComputing'23](https://github.com/mlcommons/ck/blob/master/docs/tutorials/scc23-mlperf-inference-bert.md) - * [CM automation to reproduce IPOL paper](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/reproduce-ipol-paper-2022-439/README-extra.md) -* [Auto-generated READMEs to reproduce official MLPerf BERT inference benchmark v3.0 submission with a model from the Hugging Face Zoo](https://github.com/mlcommons/submissions_inference_3.0/tree/main/open/cTuning/code/huggingface-bert/README.md) -* [Auto-generated Docker containers to run and reproduce MLPerf inference benchmark](../cm-mlops/script/app-mlperf-inference/dockerfiles/retinanet) - -## Presentations - -* [CK vision (ACM Tech Talk at YouTube)](https://www.youtube.com/watch?v=7zpeIVwICa4) -* [CK concepts (Philosophical Transactions of the Royal Society)](https://doi.org/10.1098/rsta.2020.0211) -* [CM workflow automation introduction (slides from ACM REP'23 keynote)](https://doi.org/10.5281/zenodo.8105339) -* [MLPerf inference submitter orientation (slides)](https://doi.org/10.5281/zenodo.8144274) - -## Common Metadata eXchange automation framework (CMX) - -Since 2025, we have been developing a new backward-compatible version of CM with simpler -and more intuitive interfaces for automation recipes in MLOps, DevOps, and MLPerf. diff --git a/cmx/understanding-cmx.md b/cmx/understanding-cmx.md new file mode 100644 index 0000000000..794b298c36 --- /dev/null +++ b/cmx/understanding-cmx.md @@ -0,0 +1,204 @@ +[ [Back to documentation](README.md) ] + +# Getting Started with CMX + +## Understanding Common Metadata eXchange (CMX) + +CMX allows users to embed any metadata into any artifact—such as programs, datasets, models, and scripts—within their projects +using `_cm.yaml` and/or `_cm.json` files (Common Metadata). + +This common metadata includes extensible tags, a user-friendly alias, and an automatically generated unique ID (16 lowercase hexadecimal characters), +enabling everyone to find and reuse both public and private artifacts in accordance with [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data) +(findable, accessible, interoperable, and reusable). + +At the same time, CMX allows users to share and reuse common automations +with a unified CLI and Python API, applying them to related artifacts +based on their common metadata. + +## Understanding CMX repositories + +A [typical CMX-based GitHub repository](https://github.com/mlcommons/ck/tree/master/cmx4mlops/cmx4mlops), +which includes common metadata and automations in the CMX format, is structured as follows: + +```bash +│ cmr.yaml +│ +├───automation +│ └───script +│ COPYRIGHT.md +│ module.py +│ README.md +│ _cm.json +│ +│ +└───script + ├───app-image-classification-torch-py + │ │ COPYRIGHT.md + │ │ README.md + │ │ requirements.txt + │ │ run.bat + │ │ run.sh + │ │ _cm.yaml + │ │ + │ └───src + │ pytorch_classify_preprocessed.py + │ + ├───app-mlperf-inference + │ COPYRIGHT.md + │ customize.py + │ README.md + │ run.sh + │ _cm.yaml + │ + ├───compile-program + │ COPYRIGHT.md + │ customize.py + │ run.bat + │ run.sh + │ _cm.yaml + │ + └───detect-os + COPYRIGHT.md + customize.py + _cm.yaml + +``` + +All CMX repositories follow a file-based structure with a two-level directory hierarchy. +Each repository includes a `cmr.yaml` file (Common Metadata Repository), +which contains a unique ID and a user-friendly alias for easy identification. + +All CMX repositories are structured as a file-based system with a two-level +directory hierarchy and a `cmr.yaml` (Common Metadata Repository) +with a unique ID and user-friendly alias for this repository . + +The first-level directories categorize artifacts and +include their relevant automation, while the second-level directories +house the specific artifacts within each category. + +In the above example, we have common CMX artifacts called `script`, along +with a related CMX `automation` of the same name. + +Each subdirectory within the `script` directory always contains either +a `_cm.yaml` file (typically manually generated), a `_cm.json` file +(usually automatically generated), or both. +If both are present, CMX first reads `_cm.yaml` and then merges it with `_cm.json`. +These files describe a given artifact and include all related user files associated with it. + +For CMX scripts, we typically have native scripts and an optional `customize.py` file. +The `customize.py` enables the unification of environment variables and APIs before executing +a given script via CMX automation, ensuring it runs consistently across different operating +systems through a standardized CMX interface. + +After installing CMX and cloning this repository from GitHub, you can find all shared `script` artifacts +as follows: + +```bash +fursin@laptop:~$ pip install cmind +fursin@laptop:~$ cmx pull repo mlcommons@ck --dir=cmx4mlops/cmx4mlops +fursin@laptop:~$ cmx find script + +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/script/app-image-classification-torch-py +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/script/app-mlperf-inference +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/script/compile-program +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/script/detect-os +... +``` + +You can also find all categories (automations) for shared artifacts as follows: +```bash +fursin@laptop:~$ cmx find automation + +/home/fursin/cmx/lib/python3.12/site-packages/cmind/repo/automation/automation +/home/fursin/cmx/lib/python3.12/site-packages/cmind/repo/automation/ckx +/home/fursin/cmx/lib/python3.12/site-packages/cmind/repo/automation/core +/home/fursin/cmx/lib/python3.12/site-packages/cmind/repo/automation/repo +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/cache +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/cfg +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/challenge +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/data +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/docker +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/experiment +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/report +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/script +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/automation/utils +... +``` + +By default, when you pull repositories via CMX, they are stored in the `$HOME/CM` directory. +This default location helps CMX efficiently search for all shared artifacts and automations. + +You can change this directory by setting the `CM_REPOS` environment variable. + +Additionally, you can pull multiple public and private repositories with CMX artifacts, +allowing you to reuse artifacts and automations across different projects. + +## Understanding CMX metadata + +All CMX artifacts contain a minimal set of keys in their metadata files. + +For example, the CMX script artifact `app-image-classification-torch-py`, +which includes PyTorch code for classifying images using the reference ResNet50 model, +has the following `_cm.yaml` format: + + +```yaml +uid: e3986ae887b84ca8 +alias: app-image-classification-torch-py + +automation_alias: script +automation_uid: 5b4e0237da074764 + +tags: +- app +- image-classification +- torch +- python + +... +``` + +This metadata allows users to find this artifact from command line using the following CMX commands: +```bash +$ cmx find script app-image-classification-torch-py +$ cmx find script e3986ae887b84ca8 +$ cmx find script app-image-classification-torch-py,e3986ae887b84ca8 +$ cmx find script *image-classification-torch* +$ cmx find script --tags=app,image-classification,torch +$ cmx find script "python app image-classification torch" + +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/script/app-image-classification-torch-py +``` + +You can also use a simple Python API to locate these artifacts as follows: + +```bash +$ python + +import cmind + +r = cmind.x({'action':'find', 'automation':'script', 'artifact':'app-image-classification-torch-py,e3986ae887b84ca8'}) +if r['return']>0: cmind.errorx(r) + +artifacts = r['list'] + +for artifact in artifacts: + print (artifact.path) + print (artifact.meta) +``` + +```bash +/home/fursin/CM/repos/mlcommons@ck/cmx4mlops/cmx4mlops/repo/script/app-image-classification-torch-py +{'alias': 'app-image-classification-torch-py', 'automation_alias': 'script', 'automation_uid': '5b4e0237da074764', ... + +``` + +## Understanding CMX automations + + + + + + + +To be continued... diff --git a/cmx4mlops/README.md b/cmx4mlops/README.md index 2d1bb694dd..0833bbdce2 100644 --- a/cmx4mlops/README.md +++ b/cmx4mlops/README.md @@ -3,9 +3,9 @@ [![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md) [![Powered by CM/CMX](https://img.shields.io/badge/Powered_by-MLCommons%20CM-blue)](https://pypi.org/project/cmind). -This repository is powered by the [Collective Mind workflow automation framework](https://github.com/mlcommons/ck/tree/master/cm). +This repository is powered by the [CM and/or CMX framework](https://github.com/mlcommons/ck/tree/master/cm). -Two key automations developed using CM are **Script** and **Cache**, which streamline machine learning (ML) workflows, +Two key automations developed using CM are **script** and **cache**, which streamline machine learning (ML) workflows, including managing Docker runs. Both Script and Cache automations are part of the **cmx4mlops** repository. The [CM scripts](https://access.cknowledge.org/playground/?action=scripts), diff --git a/cmx4mlops/cmx4mlops/README.md b/cmx4mlops/cmx4mlops/README.md deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/cmx4mlops/cmx4mlops/cmr.yaml b/cmx4mlops/cmx4mlops/cmr.yaml index 4e2dbf480a..850541f6b6 100644 --- a/cmx4mlops/cmx4mlops/cmr.yaml +++ b/cmx4mlops/cmx4mlops/cmr.yaml @@ -3,12 +3,12 @@ uid: 428611a6db02407f git: true -version: "0.5.1" +version: "1.0.1" author: "Grigori Fursin and contributors" install_python_requirements: false -min_cm_version: "3.5.2" +min_cm_version: "4.0.2" prefix: repo diff --git a/cmx4mlops/cmx4mlops/repo/README.md b/cmx4mlops/cmx4mlops/repo/README.md deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/cmx4mlops/cmx4mlops/repo/script/app-image-classification-torch-py/_cm.yaml b/cmx4mlops/cmx4mlops/repo/script/app-image-classification-torch-py/_cm.yaml index 6684bb7370..cd48334d15 100644 --- a/cmx4mlops/cmx4mlops/repo/script/app-image-classification-torch-py/_cm.yaml +++ b/cmx4mlops/cmx4mlops/repo/script/app-image-classification-torch-py/_cm.yaml @@ -1,10 +1,15 @@ +uid: e3986ae887b84ca8 alias: app-image-classification-torch-py + automation_alias: script automation_uid: 5b4e0237da074764 + category: Modular AI/ML application pipeline + default_env: CM_BATCH_COUNT: '1' CM_BATCH_SIZE: '1' + deps: - tags: detect,os - names: @@ -31,13 +36,16 @@ deps: USE_CUDA: - 'yes' tags: get,generic-python-lib,_torchvision_cuda + tags: - app - image-classification - torch - python + tags_help: app image-classification python torch -uid: e3986ae887b84ca8 + + variations: cuda: deps: diff --git a/mkdocs.yml b/mkdocs.yml index 892cd4de55..8fac8798e5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -25,11 +25,15 @@ nav: - CMX/CM: - Summary: cmx/README.md - Installation: cmx/install.md - - Getting Started Guide: cmx/getting-started.md + - Understanding CMX: cmx/understanding-cmx.md + - CMX commands: cmx/commands.md - MLOps, DevOps and MLPerf automations: https://access.cknowledge.org/playground/?action=scripts - High-level architecture: https://cKnowledge.org/images/cmx-architecture-4.0.0.png - Python API: https://cknowledge.org/docs/cmx - Motivation: cmx/motivation.md + - MLPerf + - MLPerf inference benchmark <= v4.1: cmx/mlperf-inference/4.1/README.md + - MLPerf inference benchmark v5.0: cmx/mlperf-inference/5.0/README.md - CK Playground: https://access.cKnowledge.org - Releases: https://github.com/mlcommons/ck/releases