diff --git a/README.md b/README.md index 330dea6768..8a27a7f8f1 100755 --- a/README.md +++ b/README.md @@ -11,53 +11,36 @@ ### About -Collective Knowledge (CK) in a community project to develop open-source tools, platforms and automation recipes -that can help researchers and engineers automate their repetitive, tedious and time-consuming tasks +[Collective Knowledge (CK)](https://cKnowledge.org) in an educational project +to help researchers and engineers automate their repetitive, tedious and time-consuming tasks to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data, software and hardware. -CK consists of several ongoing sub-projects: +CK consists of several sub-projects: * [Collective Mind framework (CM)](cm) - a very light-weight Python-based framework with minimal dependencies to help users implement, share and reuse cross-platform automation recipes to build, benchmark and optimize applications on any platform - with any software and hardware. CM attempts to extends the `cmake` concept - with reusable automation recipes and workflows written in plain Python or native OS scripts, - accessible via a human readable interface with simple tags, - and shareable in public and private repositories in a decentralized way. - Furthermore, in comparison with cmake, these automation recipes can not only detect missing code - but also download artifacts (models, data sets), preprocess them, build missing - dependencies, install them and run the final code on diverse platforms in a unified and automated way. - You can learn more about the CM concept from this [white paper](https://arxiv.org/abs/2406.16791) - and the [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339). + with any software and hardware. + * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference) * [CM4MLOPS](https://github.com/mlcommons/cm4mlops) - a collection of portable, extensible and technology-agnostic automation recipes with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications - on diverse platforms with any software and hardware: see [online cKnowledge catalog](https://access.cknowledge.org/playground/?action=scripts), + on diverse platforms with any software and hardware: see [online catalog at CK playground](https://access.cknowledge.org/playground/?action=scripts), [online MLCommons catalog](https://docs.mlcommons.org/cm4mlops/scripts) - and [source code](https://github.com/mlcommons/cm4mlops/blob/master/script). * [CM4ABTF](https://github.com/mlcommons/cm4abtf) - a unified CM interface and automation recipes to run automotive benchmark across different models, data sets, software and hardware from different vendors. -* [Collective Knowledge Playground](https://access.cKnowledge.org) - an external platform being developed by [cKnowledge](https://cKnowledge.org) +* [Collective Knowledge Playground](https://access.cKnowledge.org) - a unified platform to list CM scripts similar to PYPI, aggregate AI/ML Systems benchmarking results in a reproducible format with CM workflows, and organize [public optimization challenges and reproducibility initiatives](https://access.cknowledge.org/playground/?action=challenges) - to find the most performance and cost-effective AI/ML Systems. + to co-design more efficient and cost-effiective software and hardware for emerging workloads. - * [GUI to run modular benchmarks](https://access.cknowledge.org/playground/?action=howtorun) - such benchmarks - are composed from [CM scripts](https://access.cknowledge.org/playground/?action=scripts) - and can run via a unified CM interface. +* [Artifact Evaluation](https://cTuning.org/ae) - automating artifact evaluation and reproducibility initiatives at ML and systems conferences. - * [MLCommons docs to run MLPerf inference benchmarks from command line via CM](https://docs.mlcommons.org/inference) - -### Incubator - -We are preparing new projects based on user feedback - please contact [Grigori Fursin](https://cKnowledge.org/gfursin) for more details: -* [The next generation of CM](_incubator/cm-next-gen) *(prototyping stage)* -* [Collaborative testing of MLPerf benchmarks](_incubator/cm4mlops-testing) *(brainstorming stage)* ### License @@ -69,8 +52,15 @@ We are preparing new projects based on user feedback - please contact [Grigori F * Copyright (c) 2021-2024 MLCommons * Copyright (c) 2014-2021 cTuning foundation +### Motivation and long-term vision + +You can learn more about the motivation behind these projects from the following articles and presentations: + +* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] +* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] +* ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] -### Documentation +### CM Documentation * [CM installation GUI](https://access.cknowledge.org/playground/?action=install) * [CM Getting Started Guide and FAQ](docs/getting-started.md) @@ -83,17 +73,6 @@ We are preparing new projects based on user feedback - please contact [Grigori F * [CM and CK history](docs/history.md) -### Citing CM - -If you found CM useful, please cite this article: -[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. - -You can learn more about the motivation behind these projects from the following articles and presentations: - -* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] -* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] -* ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] - ### Acknowledgments Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin), diff --git a/_incubator/README.md b/_incubator/README.md deleted file mode 100644 index a0990367ef..0000000000 --- a/_incubator/README.md +++ /dev/null @@ -1 +0,0 @@ -TBD diff --git a/_incubator/cm-next-gen/README.md b/_incubator/cm-next-gen/README.md deleted file mode 100644 index 9bcaad2582..0000000000 --- a/_incubator/cm-next-gen/README.md +++ /dev/null @@ -1 +0,0 @@ -# Prototyping the next generation of CM diff --git a/_incubator/cm4mlops-testing/README.md b/_incubator/cm4mlops-testing/README.md deleted file mode 100644 index 74b682783e..0000000000 --- a/_incubator/cm4mlops-testing/README.md +++ /dev/null @@ -1 +0,0 @@ -Prototyping infrastructure to crowd-test [CM4MLOps scripts](https://access.cknowledge.org/playground/?action=scripts) diff --git a/_incubator/cm4mlperf/README.md b/_incubator/cm4mlperf/README.md deleted file mode 100644 index 72f21b16cc..0000000000 --- a/_incubator/cm4mlperf/README.md +++ /dev/null @@ -1,4 +0,0 @@ -# CM4MLPerf: CM automation for MLPerf benchmarks - -The idea is to provide a standard PYPI package to run MLPerf via CM. - diff --git a/cm/CHANGES.md b/cm/CHANGES.md index b4060a5129..23b4dd21f1 100644 --- a/cm/CHANGES.md +++ b/cm/CHANGES.md @@ -1,3 +1,17 @@ +## V3.0.1 + - fixed minor bug in CM core + +## V3.0.0 + - added `min_cm_version` to the CM repo description (cmr.yaml) + to check compatibility of repositories with CM. + It is needed to implement new features in CM core and repositories + requested by our users that may not work with previous CM versions. + Hence, starting a new version. + + - started prototyping a simpler and cleaner version of `cmind.access()`. + it should not influence existing automations and workflows + and will co-exist in the future. + ## V2.4.0 - added `install_python_requirements` to the CM repo description (cmr.yaml) to install requirements to a current python with CM installation if needed diff --git a/cm/README.md b/cm/README.md index 44f33b7f83..3536e37140 100644 --- a/cm/README.md +++ b/cm/README.md @@ -9,18 +9,26 @@ ### About -Collective Mind (CM) is a collection of portable, extensible, technology-agnostic and ready-to-use automation recipes -with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications -on any platform with any software and hardware: see [CM4MLOps online catalog](https://access.cknowledge.org/playground/?action=scripts), -[source code](https://github.com/mlcommons/ck/blob/master/cm-mlops/script), [ArXiv white paper]( https://arxiv.org/abs/2406.16791 ). - -CM scripts require Python 3.7+ with minimal dependencies and are +Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework +with a human-friendly interface to make it easier to build, run, benchmark and optimize applications +across diverse models, data sets, software and hardware. + +CM is a part of [Collective Knowledge (CK)](https://github.com/mlcommons/ck) - +an educational community project to learn how to run emerging workloads +in the most efficient and cost-effictive way across diverse +and continuously changing systems. + +CM includes a collection of portable, extensible and technology-agnostic automation recipes +with a common API and CLI (aka CM scripts) to unify and automate different steps +required to compose, run, benchmark and optimize complex ML/AI applications +on any platform with any software and hardware. + +CM scripts extend the concept of `cmake` with simple Python automations, native scripts +and JSON/YAML meta descriptions. They require Python 3.7+ with minimal dependencies and are [continuously extended by the community and MLCommons members](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux and any other operating system, in a cloud or inside automatically generated containers -while keeping backward compatibility - please don't hesitate -to report encountered issues [here](https://github.com/mlcommons/ck/issues) -to help this collaborative engineering effort! +while keeping backward compatibility. CM scripts were originally developed based on the following requirements from the [MLCommons members](https://mlcommons.org) @@ -36,210 +44,24 @@ from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors: and simple JSON/YAML descriptions instead of inventing new workflow languages; * must have the same interface to run all automations natively, in a cloud or inside containers. -[CM scripts](https://access.cknowledge.org/playground/?action=scripts) -are used by MLCommons, cTuning.org and cKnowledge.org to modularize MLPerf inference benchmarks -(see [this white paper](https://arxiv.org/abs/2406.16791)) -and help anyone run them across different models, datasets, software and hardware: -https://docs.mlcommons.org/inference . - -For example, you should be able to run the MLPerf inference benchmark on Linux, Windows and MacOS -using a few CM commands: - -```bash - -pip install cmind -U - -cm pull repo mlcommons@cm4mlops --branch=dev - -cm run script "run-mlperf-inference _r4.1 _accuracy-only _short" \ - --device=cpu \ - --model=resnet50 \ - --precision=float32 \ - --implementation=reference \ - --backend=onnxruntime \ - --scenario=Offline \ - --clean \ - --quiet \ - --time - -cm run script "run-mlperf-inference _r4.1 _submission _short" \ - --device=cpu \ - --model=resnet50 \ - --precision=float32 \ - --implementation=reference \ - --backend=onnxruntime \ - --scenario=Offline \ - --clean \ - --quiet \ - --time - -... - - 0 -Organization CTuning -Availability available -Division open -SystemType edge -SystemName ip_172_31_87_92 -Platform ip_172_31_87_92-reference-cpu-onnxruntime-v1.1... -Model resnet50 -MlperfModel resnet -Scenario Offline -Result 14.3978 -Accuracy 80.0 -number_of_nodes 1 -host_processor_model_name Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz -host_processors_per_node 1 -host_processor_core_count 2 -accelerator_model_name NaN -accelerators_per_node 0 -Location open/CTuning/results/ip_172_31_87_92-reference... -framework onnxruntime v1.18.1 -operating_system Ubuntu 24.04 (linux-6.8.0-1009-aws-glibc2.39) -notes Automated by MLCommons CM v2.3.2. -compliance 1 -errors 0 -version v4.1 -inferred 0 -has_power False -Units Samples/s - - -``` - -You can also run the same commands using a unified CM Python API: - -```python -import cmind -output=cmind.access({ - 'action':'run', 'automation':'script', - 'tags':'run-mlperf-inference,_r4.1,_performance-only,_short', - 'device':'cpu', - 'model':'resnet50', - 'precision':'float32', - 'implementation':'reference', - 'backend':'onnxruntime', - 'scenario':'Offline', - 'clean':True, - 'quiet':True, - 'time':True, - 'out':'con' -}) -if output['return']==0: print (output) -``` - - -We suggest you to use this [online CM interface](https://access.cknowledge.org/playground/?action=howtorun) -to generate CM commands that will prepare and run MLPerf benchmarks and AI applications across different platforms. - - -See more examples of CM scripts and workflows to download Stable Diffusion, GPT-J and LLAMA2 models, process datasets, install tools and compose AI applications: - - -```bash -pip install cmind -U - -cm pull repo mlcommons@cm4mlops --branch=dev - -cm show repo - -cm run script "python app image-classification onnx" -cmr "python app image-classification onnx" - -cmr "download file _wget" --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e -cmr "python app image-classification onnx" --input=computer_mouse.jpg -cmr "python app image-classification onnx" --input=computer_mouse.jpg --debug - -cm find script "python app image-classification onnx" -cm load script "python app image-classification onnx" --yaml - -cmr "get python" --version_min=3.8.0 --name=mlperf-experiments -cmr "install python-venv" --version_max=3.10.11 --name=mlperf - -cmr "get ml-model stable-diffusion" -cmr "get ml-model sdxl _fp16 _rclone" -cmr "get ml-model huggingface zoo _model-stub.alpindale/Llama-2-13b-ONNX" --model_filename=FP32/LlamaV2_13B_float32.onnx --skip_cache -cmr "get dataset coco _val _2014" -cmr "get dataset openimages" -j - -cm show cache -cm show cache "get ml-model stable-diffusion" - -cmr "get generic-python-lib _package.onnxruntime" --version_min=1.16.0 -cmr "python app image-classification onnx" --input=computer_mouse.jpg - -cm rm cache -f -cmr "python app image-classification onnx" --input=computer_mouse.jpg --adr.onnxruntime.version_max=1.16.0 - -cmr "get cuda" --version_min=12.0.0 --version_max=12.3.1 -cmr "python app image-classification onnx _cuda" --input=computer_mouse.jpg - -cm gui script "python app image-classification onnx" - -cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -j -docker_it - -cm docker script "get coco dataset _val _2017" --to=d:\Downloads\COCO-2017-val --store=d:\Downloads --docker_cm_repo=ctuning@mlcommons-ck - -cmr "run common mlperf inference" --implementation=nvidia --model=bert-99 --category=datacenter --division=closed -cm find script "run common mlperf inference" - -cmr "get generic-python-lib _package.torch" --version=2.1.2 -cmr "get generic-python-lib _package.torchvision" --version=0.16.2 -cmr "python app image-classification torch" --input=computer_mouse.jpg - - -cm rm repo mlcommons@cm4mlops -cm pull repo --url=https://zenodo.org/records/12528908/files/cm4mlops-20240625.zip - -cmr "install llvm prebuilt" --version=17.0.6 -cmr "app image corner-detection" - -cm run experiment --tags=tuning,experiment,batch_size -- echo --batch_size={{ print_str("VAR1{range(1,8)}") }} - -cm replay experiment --tags=tuning,experiment,batch_size - -cmr "get conda" - -cm pull repo ctuning@cm4research -cmr "reproduce paper micro-2023 victima _install_deps" -cmr "reproduce paper micro-2023 victima _run" - -``` - - -See a few examples of modular containers and GitHub actions with CM commands: - -* [GitHub action with CM commands to test MLPerf inference benchmark](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-bert.yml) -* [Dockerfile to run MLPerf inference benchmark via CM](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-mlperf-inference/dockerfiles/bert-99.9/ubuntu_22.04_python_onnxruntime_cpu.Dockerfile) - - -Please check the [**Getting Started Guide**](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md) -to understand how CM automation recipes work, how to use them to automate your own projects, -and how to implement and share new automations in your public or private projects. - - -### Documentation - -**MLCommons is updating the CM documentation based on user feedback - please stay tuned for more details**. - -* [Getting Started Guide and FAQ](https://github.com/mlcommons/ck/tree/master/docs/getting-started.md) - * [Common CM interface to run MLPerf inference benchmarks](https://github.com/mlcommons/ck/tree/master/docs/mlperf/inference) - * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](https://github.com/mlcommons/ck/tree/master/docs/tutorials/common-interface-to-reproduce-research-projects.md) - * [Other CM tutorials](https://github.com/mlcommons/ck/tree/master/docs/tutorials) -* [Full documentation](https://github.com/mlcommons/ck/tree/master/docs/tutorials/README.md) - -### Projects modularized and automated by CM +### Resources -* [cm4research](https://github.com/ctuning/cm4research) -* [cm4mlops](https://github.com/mlcommons/cm4mlops) -* [cm4abtf](https://github.com/mlcommons/cm4abtf) +* CM v2.x (stable version 2022-cur): [installation on Linux, Windows, MacOS](https://access.cknowledge.org/playground/?action=install) ; + [docs](https://docs.mlcommons.org/ck) ; [popular commands](https://github.com/mlcommons/ck/tree/master/cm/docs/demos/some-cm-commands.md) ; + [getting started guide](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md) +* CM v3.x (prototype 2024-cur): [docs](https://github.com/mlcommons/ck/tree/master/cm/docs/cmx) +* MLPerf inference benchmark automated via CM + * [Run MLPerf for submissions](https://docs.mlcommons.org/inference) + * [Run MLPerf at the Student Cluster Competition'24](https://docs.mlcommons.org/inference/benchmarks/text_to_image/reproducibility/scc24) +* Examples of modular containers and GitHub actions with CM commands: + * [GitHub action with CM commands to test MLPerf inference benchmark](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-bert.yml) + * [Dockerfile to run MLPerf inference benchmark via CM](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-mlperf-inference/dockerfiles/bert-99.9/ubuntu_22.04_python_onnxruntime_cpu.Dockerfile) ### License [Apache 2.0](LICENSE.md) -### Citing CM +### Citing CM and CM4MLOps If you found CM useful, please cite this article: [ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. @@ -248,11 +70,11 @@ You can learn more about the motivation behind these projects from the following * "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] -* ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] +* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] ### Acknowledgments -Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin), +The Collective Mind framework (CM) was created by [Grigori Fursin](https://cKnowledge.org/gfursin), sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. Since then, this open-source technology (CM, CM4MLOps, CM4MLPerf, CM4ABTF, CM4Research, etc) is being developed as a community effort thanks to all our diff --git a/cm/cmind/__init__.py b/cm/cmind/__init__.py index a5018228f4..0fdfad1bc8 100644 --- a/cm/cmind/__init__.py +++ b/cm/cmind/__init__.py @@ -2,9 +2,10 @@ # # Written by Grigori Fursin -__version__ = "2.4.0" +__version__ = "3.0.1" from cmind.core import access +from cmind.core import x from cmind.core import error from cmind.core import halt from cmind.core import CM diff --git a/cm/cmind/cli.py b/cm/cmind/cli.py index 5449fd8e49..d62fbc90e3 100644 --- a/cm/cmind/cli.py +++ b/cm/cmind/cli.py @@ -51,6 +51,44 @@ def run(argv = None): sys.exit(r['return']) +############################################################ +def runx(argv = None): + """ + Run CM automation actions from the new command line. + + CM command line format: + + cm {action} {automation} (artifacts) (-params) (--inputs) (@input.yaml) (@input.json) + + Args: + argv (list | string): command line arguments + + Returns: + (CM return dict): + + * return (int): return code == 0 if no error and >0 if error + * (error) (str): error string if return>0 + + * Output from a CM automation action + + """ + + # Access CM + from cmind.core import CM + + cm = CM() + + if argv is None: + argv = sys.argv[1:] + + r = cm.x(argv, out='con') + + if r['return']>0 and (cm.output is None or cm.output == 'con'): + cm.error(r) + + sys.exit(r['return']) + + ############################################################ def run_script(argv = None): """ @@ -275,5 +313,169 @@ def parse(cmd): return {'return':0, 'cm_input':cm_input} +############################################################ +def parsex(cmd): + """ + Parse CM command line into CM input dictionary. + + Args: + cmd (str | list) : arguments as a string or list + + Returns: + (CM return dict): + + * return (int): return code == 0 if no error and >0 if error + * (error) (str): error string if return>0 + + * cm_input (dict): CM unified input to the CM access function + + """ + + # Init variables + cmx_input = {'control':{}} + + # If input is string, convert to argv + # We use shlex to properly convert "" + if cmd is None: + argv = [] + + elif type(cmd) == str: + import shlex + argv = shlex.split(cmd) + + else: + argv = cmd + + cmx_input['control']['_cmd'] = argv.copy() + + # First positional arguments + for key in ['action', 'automation']: + if len(argv) > 0: + x = argv[0].strip() + if x != '' and x[0] not in ['-', '@']: + cmx_input[key] = argv.pop(0) + + # Check if just one artifact or multiple ones + artifact ='' + artifacts = [] # Only added if more than 1 artifact! + + for index in range(0, len(argv)): + a = argv[index] + + if a == '--': + unparsed = [] if index>len(argv) else argv[index+1:] + cmx_input['control']['_unparsed_cmd'] = unparsed + break + + elif a.startswith('@'): + # Load JSON or YAML file + from cmind import utils + r = utils.load_json_or_yaml(file_name = a[1:], check_if_exists=True) + if r['return'] >0 : return r + + meta = r['meta'] + + control = meta.pop('control', {}) + + for k in control: + cmx_input['control'][k] = control[k] + + cmx_input.update(meta) + + # Check flags + elif a.startswith('---'): + return {'return':1, 'error': f'flag "{a}" has unknown prefix'} + + elif a.startswith('--'): + a = a[2:] + split_flag(a, cmx_input) + + elif a.startswith('-'): + a = a[1:] + split_flag(a, cmx_input['control']) + + else: + # artifact + if '=' in a: + split_flag(a, cmx_input) + + elif artifact == '': + artifact = a + cmx_input['artifact'] = a + + else: + artifacts.append(a) + + + # Add extra artifacts if specified + if len(artifacts) > 0: + cmx_input['artifacts'] = artifacts + + return {'return':0, 'cmx_input':cmx_input} + +########################################################################### +def split_flag(flag, array, value=None): + """ + Split flag key=value and add to array + + Args: + flag (str): flag of format "key=value" + array (dict): array where to accumulate flags + value (str): if not None use it (for "-key value" flags) + + Returns: + key (str): key + value (str): value + array (dict): updated array + + """ + + # flags + j = flag.find('=') # find first = + if j>0: + key = flag[:j].strip() + value = flag[j+1:].strip() + else: + key = flag + if value == None: + if key.endswith('-'): + key = key[:-1] + value=False + else: + value=True + + # Decided not to do that (20241006) + # Force '-' to '_' in keys in CLI + # key = key.replace('-', '_') + + if key.endswith(','): + key = key[:-1] + if value in [True, False]: + value = [value] + elif value !='': + value = value.split(',') + else: + value = [] + + if '.' in key: + keys = key.split('.') + + first = True + + for key in keys[:-1]: + if first: + first = False + + if key not in array: + array[key] = {} + array = array[key] + + array[keys[-1]] = value + else: + array[key] = value + + return key, value, array + +########################################################################### if __name__ == "__main__": run() diff --git a/cm/cmind/config.py b/cm/cmind/config.py index db750ce70d..36255c138a 100644 --- a/cm/cmind/config.py +++ b/cm/cmind/config.py @@ -37,7 +37,8 @@ def __init__(self, config_file = None): "flag_help2": "help", "error_prefix": "CM error:", - "info_cli": "cm {action} {automation} {artifact(s)} {--flags} @input.yaml @input.json", + "info_cli": "cm {action} {automation} {artifact(s)} {flags} @input.yaml @input.json", + "info_clix": "cmx {action} {automation} {artifact(s)} {flags} @input.yaml @input.json", "default_home_dir": "CM", @@ -50,6 +51,7 @@ def __init__(self, config_file = None): "file_meta_repo": "cmr", "common_automation_module_name": "module", + "common_automation_module_namex": "modulex", "action_substitutions": { "ls":"search", @@ -62,7 +64,7 @@ def __init__(self, config_file = None): "cp":"copy" }, - "new_repo_requirements": "cmind >= 2.0.4\n", + "new_repo_requirements": "cmind >= 3.0.0\n", "cmind_automation":"automation", diff --git a/cm/cmind/core.py b/cm/cmind/core.py index afd04e6dfe..d4831d5ac8 100644 --- a/cm/cmind/core.py +++ b/cm/cmind/core.py @@ -6,6 +6,7 @@ from cmind.repos import Repos from cmind.index import Index from cmind.automation import Automation +#from cmind.automation import AutomationDummy from cmind import utils import sys @@ -96,8 +97,9 @@ def __init__(self, repos_path = '', debug = False): # Check Python version self.python_version = list(sys.version_info) - # Save output to json (only from CLI) + # Save output to json or yaml(only from CLI) self.save_to_json = '' + self.save_to_yaml = '' # Logging self.logger = None @@ -110,6 +112,8 @@ def __init__(self, repos_path = '', debug = False): if os.environ.get(self.cfg['env_index'],'').strip().lower() in ['no','off','false']: self.use_index = False + self.x_was_called = False + ############################################################ def error(self, r): """ @@ -622,6 +626,520 @@ def access(self, i, out = None): return r + ############################################################ + def x(self, i, out = None): + """ + New unified access to CM automation actions + + Args: + i (dict | str | argv): unified CM input + + (action) (str): automation action + (automation (CM object): CM automation in format (alias | UID | alias,UID) + or (repo alias | repo UID | repo alias,UID):(alias | UID | alias,UID) + (artifact) (CM object): CM artifact + (artifacts) (list of CM objects): extra CM artifacts + + (common) (bool): if True, use common automation action from Automation class + + (help) (bool): if True, print CM automation action API + + (ignore_inheritance) (bool): if True, ignore inheritance when searching for artifacts and automations + + (out) (str): if 'con', tell automations and CM to output extra information to console + + Returns: + (CM return dict): + + * return (int): return code == 0 if no error and >0 if error + * (error) (str): error string if return>0 + + * Output from a CM automation action + """ + + # Check if first access call + x_was_called = self.x_was_called + self.x_was_called = True + + # Check the type of input + if i is None: + i = {} + + # Attempt to detect debug flag early (though suggest to use environment) + # If error in parse_cli, it will raise error + if self.cfg['flag_debug'] in i: + self.debug = True + + # Check if log + if self.logger is None: + self.logger = logging.getLogger("cm") + + # Parse as command line if string or list + if type(i) == str or type(i) == list: + import cmind.cli + + r = cmind.cli.parsex(i) + if r['return'] >0 : return r + + i = r['cmx_input'] + + # Assemble input flags for extra checks in automations + if 'control' not in i: + i['control'] = {} + + i['control']['_input'] = {} + + for k in i: + if k not in ['control', 'action', 'automation', 'artifact', 'artifacts']: + i['control']['_input'][k] = i[k] + + control = i['control'] + + # Check if force out programmatically (such as from CLI) + if 'out' not in control and out is not None: + control['out'] = out + + output = control.get('out', '') + + # Check and force json console output + if control.get('j', False) or control.get('json', False): + output = 'json' + + # Set self.output to the output of the very first access + # to print error in the end if needed + if self.output is None: + self.output = output + + # Check if console + console = (output == 'con') + + # Check if has help flag + cm_help = control.get(self.cfg['flag_help'], False) or control.get(self.cfg['flag_help2'], False) + + # Initialized common automation with collective database actions + if self.common_automation == None: + self.common_automation = Automation(self, __file__) + + # Check automation action + action = i.get('action','') + + # Check automation + automation = i.get('automation','') + + # Check if asked for "version" and no automation + if action == 'version' and automation == '': + automation = 'core' + elif action == '' and automation == '' and control.get('version', False): + action = 'version' + automation = 'core' + elif action == 'init' and automation == '': + automation = 'core' + + + # Print basic help if action == '' + extra_help = True if action == 'help' and automation == '' else False + + if action == '' or extra_help: + if console: + print (self.cfg['info_clix']) + + if cm_help or extra_help: + print_db_actions(self.common_automation, self.cfg['action_substitutions'], '', cmx = True) + + return {'return':0, 'warning':'no action specified'} + + # Load info about all CM repositories (to enable search for automations and artifacts) + if self.repos == None: + repos = Repos(path = self.repos_path, cfg = self.cfg, + path_to_internal_repo = self.path_to_cmind_repo) + + r = repos.load() + if r['return'] >0 : return r + + # Set only after all initializations + self.repos = repos + + # Load index + if self.index is None: + self.index = Index(self.repos_path, self.cfg) + + if self.use_index: + r = self.index.load() + if r['return']>0: return r + + if not r['exists']: + # First time + if console: + print ('Warning: CM index is used for the first time. CM will reindex all artifacts now - it may take some time ...') + + r = self.access({'action':'reindex', + 'automation':'repo,55c3e27e8a140e48'}) + if r['return']>0: return r + + # Check if forced common automation + use_common_automation = True if control.get('common', False) else False + + automation_lst = [] + use_any_action = False + + artifact = i.get('artifact', '').strip() + artifacts = i.get('artifacts', []) # Only if more than 1 artifact + + # Check if automation is "." - then attempt to detect repo, automation and artifact from the current directory + if automation == '.' or artifact == '.': + r = self.access({'action':'detect', + 'automation':'repo,55c3e27e8a140e48'}) + if r['return']>0: return r + + # Check and substitute automation + if automation == '.': + automation = '' + if r.get('artifact_found', False): + if not r.get('found_in_current_path',False): + # If not in the root directory (otherwise search through all automations) + automation = r['cm_automation'] + + # Check and make an artifact (only if artifacts are not specified) + if artifact == '.' or artifact == '': + artifact = '' + if r.get('cm_artifact','')!='': + artifact = r['cm_artifact'] + + if r.get('registered', False): + cm_repo = r['cm_repo'] + + if ':' not in artifact: + artifact = cm_repo + ':' + artifact + + for ia in range(0,len(artifacts)): + a = artifacts[ia] + if ':' not in a: + a = cm_repo + ':' + a + artifacts[ia] = a + + # If automation!='', attempt to find it and load + # Otherwise use the common automation + if automation != '': + # Parse automation potentially with a repository + # and convert it into CM object [(artifact,UID) (,(repo,UID))] + r = utils.parse_cm_object(automation) + if r['return'] >0 : return r + + parsed_automation = r['cm_object'] + control['_parsed_automation'] = parsed_automation + + if use_common_automation: + # Check that UID is set otherwise don't know how to add + xuid = parsed_automation[0][1] + if xuid == '': + return {'return':1, 'error':'you must add `,CM UID` for automation {} when using --common'.format(parsed_automation[0][0])} + elif not utils.is_cm_uid(xuid): + return {'return':1, 'error':'you must use CM UID after automation {} when using --common'.format(parsed_automation[0][0])} + + automation_meta = {} + automation_use_x = True + automation_found = False + + if automation != '' and not use_common_automation: + # If wildcards in automation, use the common one (usually for search across different automations) + # However, still need above "parse_automation" for proper search + if '*' in automation or '?' in automation: + use_common_automation = True + else: + # First object in a list is an automation + # Second optional object in a list is a repo + auto_name = parsed_automation[0] if len(parsed_automation)>0 else ('','') + auto_repo = parsed_automation[1] if len(parsed_automation)>1 else None + + # Search for automations in repos (local, internal, other) TBD: maybe should be local, other, internal? + ii={'parsed_automation':[('automation','bbeb15d8f0a944a4')], + 'parsed_artifact':parsed_automation} + + # Ignore inheritance when called recursively + if control.get('ignore_inheritance', False): + ii['ignore_inheritance'] = True + + r = self.common_automation.search(ii) + if r['return']>0: return r + +# TBD: Fill in alias automatically from the name in search ... + + automation_lst = r['list'] + + if len(automation_lst)==1: + automation = automation_lst[0] + + automation_path = automation.path + automation_meta = automation.meta + + use_any_action = automation_meta.get('use_any_action',False) + + # Update parsed_automation with UID and alias + parsed_automation[0] = (automation_meta.get('alias',''), + automation_meta.get('uid','')) + + # Find Python module for this automation: should also work with 3.12+ + found_module = False + for automation_name in [self.cfg['common_automation_module_namex'], self.cfg['common_automation_module_name']]: + automation_full_path = os.path.join(automation_path, automation_name + '.py') + + if os.path.isfile(automation_full_path): + found_module = True + break + + automation_use_x = False + + if not found_module: + return {'return': 1, 'error': f"can\'t find CM Python module file in \"{automation_path}\""} + + found_automation_spec = importlib.util.spec_from_file_location(automation_name, automation_full_path) + if found_automation_spec == None: + return {'return': 1, 'error': 'can\'t find Python module file {}'.format(automation_full_path)} + + try: + loaded_automation = importlib.util.module_from_spec(found_automation_spec) + found_automation_spec.loader.exec_module(loaded_automation) + except Exception as e: # pragma: no cover + return {'return': 1, 'error': 'can\'t load Python module code (path={}, name={}, err={})'.format(automation_path, automation_name, format(e))} + + loaded_automation_class = loaded_automation.CAutomation + + # Try to load meta description + automation_path_meta = os.path.join(automation_path, self.cfg['file_cmeta']) + + r = utils.is_file_json_or_yaml(file_name = automation_path_meta) + if r['return']>0: return r + + if not r['is_file']: + return {'return':4, 'error':'automation meta not found in {}'.format(automation_path)} + + # Load artifact class + r=utils.load_yaml_and_json(automation_path_meta) + if r['return']>0: return r + + automation_meta = r['meta'] + automation_found = True + + elif len(automation_lst)>1: + return {'return':2, 'error':'ambiguity because several automations were found for {}'.format(auto_name)} + + # Report an error if a repo is specified for a given automation action but it's not found there + if len(automation_lst)==0 and auto_repo is not None: + return {'return':3, 'error':'automation was not found in a specified repo {}'.format(auto_repo)} + + + + # Convert action into function (substitute internal words) + original_action = action + action = action.replace('-','_') + + if action in self.cfg['action_substitutions']: + action = self.cfg['action_substitutions'][action] + elif action in automation_meta.get('action_substitutions',{}): + action = automation_meta['action_substitutions'][action] + + # Check if common automation and --help + if (use_common_automation or automation == '') and cm_help: + return print_action_help(self.common_automation, + self.common_automation, + 'common', + action, + original_action) + + # If no automation was found we do not force common automation, check if should fail or continue + if not use_common_automation and len(automation_lst)==0: + if self.cfg['fail_if_automation_not_found']: + # Quit with error + if automation=='': + return {'return':4, 'error':'automation was not specified'} + else: + return {'return':4, 'error':f'automation {automation} not found'} + + # If no automation was found or we force common automation + loaded_common_automation = False + if use_common_automation or len(automation_lst)==0: + auto=('automation','bbeb15d8f0a944a4') + from . import automation as loaded_automation + + loaded_automation_class = loaded_automation.Automation + + automation_full_path = loaded_automation.self_path + + automation_meta = { + 'alias':'automation', + 'uid':'bbeb15d8f0a944a4' + } + + loaded_common_automation = True + + # Finalize automation class initialization + initialized_automation = loaded_automation_class(self, automation_full_path) + initialized_automation.meta = automation_meta + initialized_automation.full_path = automation_full_path + + # Check if action is not present in the class (inheritance) + if automation_use_x and automation_found: + # In such case, use old CM API <3+ + v = vars(initialized_automation.__class__) + if action not in v or not inspect.isroutine(v[action]): + automation_use_x = False + + # Check if action exists + print_automation = automation_meta.get('alias','') + ',' + automation_meta.get('uid','') + initialized_automation.artifact = print_automation + + if not hasattr(initialized_automation, action): + return {'return':4, 'error':f'action "{action}" not found in automation "{print_automation}"'} + else: + # Check if has _cmx extension + if loaded_common_automation: + # By default don't use CMX in the common automation unless has _cmx extension + # (checked later) + automation_use_x = False + + if not automation_use_x and hasattr(initialized_automation, action + '_cmx'): + action_addr = getattr(initialized_automation, action + '_cmx') + automation_use_x = True + else: + action_addr = getattr(initialized_automation, action) + + # Check action in a class when importing + if use_any_action: + action = 'any' + + # Check if help about automation actions + if action == 'help': + import types + + print (self.cfg['info_clix']) + + print ('') + print ('Automation python module: {}'.format(automation_full_path)) + + r = print_db_actions(self.common_automation, self.cfg['action_substitutions'], automation_meta.get('alias',''), cmx = True) + if r['return']>0: return r + + db_actions = r['db_actions'] + + actions = [] + for d in sorted(dir(initialized_automation)): + if d not in db_actions and type(getattr(initialized_automation, d))==types.MethodType and not d.startswith('_'): + actions.append(d) + + if len(actions)>0: + print ('') + print ('Automation actions:') + print ('') + + for d in actions: + print (' * cmx ' + d + ' ' + automation_meta.get('alias','')) + + return {'return':0, 'warning':'no automation action'} + + + + # Check if help for a given automation action + delayed_help = False + delayed_help_api = '' + delayed_help_api_prefix = '' + delayed_help_api_prefix_0 = '' + + if cm_help: + # Find path to automation + rr = print_action_help(initialized_automation, + self.common_automation, + print_automation, + action, + original_action) + + if rr['return']>0: return rr + + if not rr.get('delayed_help', False): + return rr + + delayed_help = True + delayed_help_api = rr['help'] + delayed_help_api_prefix = rr['help_prefix'] + delayed_help_api_prefix_0 = rr['help_prefix_0'] + + # Process artifacts for this automation action + if len(artifacts)>0: + parsed_artifacts = [] + + for extra_artifact in artifacts: + # Parse artifact + r = parse_cm_object_and_check_current_dir(self, extra_artifact) + if r['return'] >0 : return r + + parsed_artifacts.append(r['cm_object']) + + control['_parsed_artifacts'] = parsed_artifacts + + # Check artifact and artifacts + if artifact != '': + # Parse artifact + r = parse_cm_object_and_check_current_dir(self, artifact) + if r['return'] >0 : return r + + control['_parsed_artifact'] = r['cm_object'] + + # Check min CM version requirement + min_cm_version = automation_meta.get('min_cm_version','').strip() + if min_cm_version != '': + from cmind import __version__ as current_cm_version + comparison = utils.compare_versions(current_cm_version, min_cm_version) + if comparison < 0: + return {'return':1, 'error':'CM automation requires CM version >= {} while current CM version is {} - please update using "pip install cmind -U"'.format(min_cm_version, current_cm_version)} + + + + # Roll back to older input for older CM versions < 3 + ii = i + if not automation_use_x: + for k in ['_parsed_automation', '_parsed_artifact', '_parsed_artifacts', '_cmd', '_unparsed_cmd']: + if k in control: + ii[k[1:]] = control[k] + + for k in control: + if not k.startswith('_'): + ii[k] = control[k] + + # Call automation action + r = action_addr(i) + + # Check if need to save index + if self.use_index and self.index.updated: + rx = self.index.save() + # Ignore output for now to continue working even if issues ... + + self.index.updated = False + + # If delayed help + if delayed_help and not r.get('skip_delayed_help', False): + print ('') + print (delayed_help_api_prefix_0) + print ('') + print (delayed_help_api_prefix) + print ('') + print (delayed_help_api) + + if not x_was_called: + # Very first call (not recursive) + # Check if output to json and save file + + if self.output == 'json': + utils.dump_safe_json(r) + + # Check if save to json + if control.get('save_to_json_file', '') != '': + utils.save_json(control['save_to_json_file'], meta = r) + + if control.get('save_to_yaml_file', '') != '': + utils.save_yaml(control['save_to_yaml_file'], meta = r) + + return r + + ############################################################ def parse_cm_object_and_check_current_dir(cmind, artifact): """ @@ -644,7 +1162,7 @@ def parse_cm_object_and_check_current_dir(cmind, artifact): return utils.parse_cm_object(artifact) ############################################################ -def print_db_actions(automation, equivalent_actions, automation_name): +def print_db_actions(automation, equivalent_actions, automation_name, cmx = False): """ Internal function: prints CM database actions. @@ -682,7 +1200,9 @@ def print_db_actions(automation, equivalent_actions, automation_name): x = ' ' + automation_name if automation_name!='' else '' - print (' * cm ' + s + x) + postfix = 'x' if cmx else '' + + print (f' * cm{postfix} ' + s + x) return {'return':0, 'db_actions':db_actions} @@ -750,7 +1270,7 @@ def access(i): without the need to initialize and customize CM class. Useful for Python automation scripts. - See CM.access function for more details. + See cmind.CM.access function for more details. """ global cm @@ -760,6 +1280,23 @@ def access(i): return cm.access(i) +############################################################ +def x(i): + """ + Automatically initialize CM and run automations + without the need to initialize and customize CM class. + Useful for Python automation scripts. + + See cmind.CM.x function for more details. + """ + + global cm + + if cm is None: + cm=CM() + + return cm.x(i) + ############################################################ def error(i): """ diff --git a/cm/cmind/repo/automation/automation/module.py b/cm/cmind/repo/automation/automation/module.py index cecb0964cc..66fe9bb780 100644 --- a/cm/cmind/repo/automation/automation/module.py +++ b/cm/cmind/repo/automation/automation/module.py @@ -115,6 +115,77 @@ def add(self, i): return r_obj + ############################################################ + def add_cmx(self, i): + """ + Add new automation in CMX format + + Args: + (CMX input dict): + + Returns: + (CM return dict): + + * return (int): return code == 0 if no error and >0 if error + * (error) (str): error string if return>0 + + """ + + import shutil + + console = i['control'].get('out', '') == 'con' + + # Prepare to call common function + r = utils.process_input(i) + if r['return']>0: return r + + # Take only out from original control + i['control']={'out':i['control']['out'], + 'common':True} + + tags_list = utils.convert_tags_to_list(i) + if 'automation' not in tags_list: tags_list.append('automation') + + i['meta']={'automation_alias':self.meta['alias'], + 'automation_uid':self.meta['uid'], + 'tags':tags_list} + + if 'tags' in i: del(i['tags']) + + # Use yaml by default + if 'yaml' not in i: + i['yaml'] = True + + # Pass to common action + r_obj = self.cmind.x(i) + if r_obj['return']>0: return r_obj + + new_automation_path = r_obj['path'] + + if console: + print ('Created automation in {}'.format(new_automation_path)) + + module_name = 'modulex.py' + + # Create Python module holder + module_holder_path = new_automation_path + + # Copy support files + original_path = os.path.dirname(self.path) + + # Copy module files + for f in ['modulex_dummy.py']: + f1 = os.path.join(self.path, f) + f2 = os.path.join(new_automation_path, f.replace('_dummy','')) + + if console: + print (' * Copying {} to {}'.format(f1, f2)) + + shutil.copyfile(f1,f2) + + return r_obj + + ############################################################ def doc(self, i): diff --git a/cm/cmind/repo/automation/automation/modulex_dummy.py b/cm/cmind/repo/automation/automation/modulex_dummy.py new file mode 100644 index 0000000000..6dc0028a00 --- /dev/null +++ b/cm/cmind/repo/automation/automation/modulex_dummy.py @@ -0,0 +1,71 @@ +# Developer(s): Grigori Fursin + +import os + +from cmind.automation import Automation +from cmind import utils + +class CAutomation(Automation): + """ + Automation actions + """ + + ############################################################ + def __init__(self, cmind, automation_file): + super().__init__(cmind, __file__) + + ############################################################ + def test(self, i): + """ + Test automation + + Args: + i (dict): CM input dict + + action (str): CM action + automation (str): CM automation + artifact (str): CM artifact + artifacts (list): list of extra CM artifacts + + control: (dict): various CM control + (out) (str): if 'con', output to console + ... + + (flags) + ... + + Returns: + (CM return dict): + + * return (int): return code == 0 if no error and >0 if error + * (error) (str): error string if return>0 + + * Output from this automation action + + """ + + # Access CM + print (self.cmind) + + # Print self path + print (self.path) + + # Print self meta + print (self.meta) + + # Print self automation module path + print (self.automation_file_path) + + # Print self automation module path + print (self.full_path) + + # Print self artifact + print (self.artifact) + + # Print input + + import json + print (json.dumps(i, indent=2)) + + return {'return':0} + diff --git a/cm/cmind/repos.py b/cm/cmind/repos.py index 8175613e25..4300f5f6fa 100644 --- a/cm/cmind/repos.py +++ b/cm/cmind/repos.py @@ -537,6 +537,14 @@ def pull(self, alias, url = '', branch = '', checkout = '', console = False, des if not os.path.isdir(path_to_repo_with_prefix): os.makedirs(path_to_repo_with_prefix) + # Check min CM version requirement + min_cm_version = meta.get('min_cm_version','').strip() + if min_cm_version != '': + from cmind import __version__ as current_cm_version + comparison = utils.compare_versions(current_cm_version, min_cm_version) + if comparison < 0: + return {'return':1, 'error':'This repository requires CM version >= {} while current CM version is {} - please update using "pip install cmind -U"'.format(min_cm_version, current_cm_version)} + # Get final alias alias = meta.get('alias', '') @@ -660,11 +668,12 @@ def init(self, alias, uid, path = '', console = False, desc = '', prefix = '', o if r['return']>0: return r # Check requirements.txt - path_to_requirements = os.path.join(path_to_repo, 'requirements.txt') - - if not os.path.isfile(path_to_requirements): - r = utils.save_txt(file_name = path_to_requirements, string = self.cfg['new_repo_requirements']) - if r['return']>0: return r + # 20241006: Moved the check for min cmind version to _cmr.yaml +# path_to_requirements = os.path.join(path_to_repo, 'requirements.txt') +# +# if not os.path.isfile(path_to_requirements): +# r = utils.save_txt(file_name = path_to_requirements, string = self.cfg['new_repo_requirements']) +# if r['return']>0: return r else: r=utils.load_yaml_and_json(file_name_without_ext=path_to_repo_desc) diff --git a/cm/cmind/utils.py b/cm/cmind/utils.py index 36929fd813..51944e8291 100644 --- a/cm/cmind/utils.py +++ b/cm/cmind/utils.py @@ -1142,6 +1142,64 @@ def assemble_cm_object2(cm_obj): return assemble_cm_object(cm_obj[0], cm_obj[1]) +########################################################################### +def process_input(i, update_input = True): + """ + Process automation, artifact and artifacts from i['control'] + + Args: + control['_unparsed_automation'] + control['_unparsed_artifact'] + control['_unparsed_artifacts'] + + Returns: + (dict) + name + alias + + """ + + control = i['control'] + + for k in ['_parsed_automation', '_parsed_artifact']: + if k in control: + a = process_input_helper(control[k]) + control[k[7:]] = a + + if '_parsed_artifacts' in control: + control['_artifacts'] = [] + for artifact in control['_parsed_artifacts']: + control['_artifacts'].append(process_input_helper(artifact)) + + if update_input: + for k in ['_automation', '_artifact', '_artifacts']: + if k in control: + i[k[1:]] = control[k] + + + return {'return':0} + +def process_input_helper(cm_obj): + cm_artifact = '' + + if len(cm_obj)>0: + cm_obj_artifact = cm_obj.pop(0) + + cm_artifact_alias = cm_obj_artifact[0] + cm_artifact_uid = cm_obj_artifact[1] + + cm_artifact = assemble_cm_object(cm_artifact_alias, cm_artifact_uid) + + if len(cm_obj)>0: + cm_obj_repo = cm_obj.pop(0) + + cm_repo_alias = cm_obj_repo[0] + cm_repo_uid = cm_obj_repo[1] + + cm_artifact = assemble_cm_object(cm_repo_alias, cm_repo_uid) + ':' + cm_artifact + + return cm_artifact + ########################################################################### def dump_safe_json(i): """ diff --git a/cm/dev/cmx/README.md b/cm/dev/cmx/README.md new file mode 100644 index 0000000000..09e623a353 --- /dev/null +++ b/cm/dev/cmx/README.md @@ -0,0 +1,6 @@ +# Resolving tech. issues + +## Can't run CM console scripts + +`python setup.py install` is gradually deprecated and may result in failing console scripts. +Use `pip install .` instead. diff --git a/cm/docs/cmx/README.md b/cm/docs/cmx/README.md new file mode 100644 index 0000000000..d3e6e686c3 --- /dev/null +++ b/cm/docs/cmx/README.md @@ -0,0 +1,3 @@ +# Collective Mind v3 (prototype) + +* [Installation (Linux, Windows, MacOS)](install.md) diff --git a/cm/docs/cmx/install.md b/cm/docs/cmx/install.md new file mode 100644 index 0000000000..d64071cff0 --- /dev/null +++ b/cm/docs/cmx/install.md @@ -0,0 +1,124 @@ +[ [Back to index](README.md) ] + +# CM Installation + +CM framework requires minimal dependencies to run on any platform: `python 3.7+, pip, venv, git, git-lfs, wget, curl`. + +By default, CM will pull Git repositories and cache installations and downloaded files in your `$HOME/CM` directory on Linux and MacOS +or `%userprofile%\CM` directory on Windows. +You can change it to any another directory using the `CM_REPOS` environment variable, for example `export CM_REPOS=/scratch/CM`. + +*Feel free to use the [online installation GUI](https://access.cknowledge.org/playground/?action=install-cmx)*. + + +## Ubuntu, Debian + +*We have successfully tested CMX with the following system dependencies on Ubuntu 20.x, 22.x , 24.x:* + +```bash +sudo apt update +sudo apt install python3 python3-pip python3-venv git git-lfs wget curl + +python3 -m venv cm +source cm/bin/activate + +python3 -m pip install cmind +``` + +Note that you may need to restart your shell to update PATH to the "cmx" binary. + +Alternatively you can run + +```bash +source $HOME/.profile +``` + +You can check that CMX works using the following command: +```bash +cmx test core +``` + + + +## Red Hat + +*We have successfully tested CM on Red Hat 9 and CentOS 8* + +```bash +sudo dnf update + +sudo dnf install python3 python-pip git git-lfs wget curl + +python3 -m pip install cmind --user + +``` + + +## MacOS + +*Note that CM currently does not work with Python installed from the Apple Store. + Please install Python via brew as described below.* + +If `brew` package manager is not installed, please install it as follows (see details [here](https://brew.sh/)): +```bash +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" +``` + +Don't forget to add brew to PATH environment as described in the end. + +Then install python, pip, git and wget: + +```bash +brew install python3 git git-lfs wget curl + +python3 -m pip install cmind +``` + +*Sometimes python does not add `cm` and `cmr` binaries to the `PATH` environment variable. + You may need to find these files and add their path to `PATH` variable. + We plan to simplify this installation in the future.* + + +## Windows + +* Configure Windows 10+ to [support long paths](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later) from command line as admin: + + + + ```bash + reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /v LongPathsEnabled /t REG_DWORD /d 1 /f + ``` + + + +* Download and install Git from [git-for-windows.github.io](https://git-for-windows.github.io). + * Configure Git to accept long file names: `git config --system core.longpaths true` +* Download and install Python 3+ from [www.python.org/downloads/windows](https://www.python.org/downloads/windows). + * Don't forget to select option to add Python binaries to PATH environment! + * Configure Windows to accept long fie names during Python installation! + +* Install CM via PIP: + +```bash +python -m pip install cmind +``` + +*Note that we [have reports](https://github.com/mlcommons/ck/issues/844) + that CM does not work when Python was first installed from the Microsoft Store. + If CM fails to run, you can find a fix [here](https://stackoverflow.com/questions/57485491/python-python3-executes-in-command-prompt-but-does-not-run-correctly)*. + + +*We plan to provide a self-sustained package in the future to simplify CM installation on Windows.* + + + +# CM CLI testing + +If the installation is successful, you can run the CM CLI as follows: + +```bash +gfursin@cmind:~$ cmx + +cmx {action} {automation} {artifact(s)} {flags} @input.yaml @input.json +``` + diff --git a/cm/docs/demos/some-cm-commands.md b/cm/docs/demos/some-cm-commands.md new file mode 100644 index 0000000000..7aa09cea14 --- /dev/null +++ b/cm/docs/demos/some-cm-commands.md @@ -0,0 +1,175 @@ +Some CM commands: + + +```bash + +pip install cmind -U + +cm init + +cm run script "run-mlperf-inference _r4.1 _accuracy-only _short" \ + --device=cpu \ + --model=resnet50 \ + --precision=float32 \ + --implementation=reference \ + --backend=onnxruntime \ + --scenario=Offline \ + --clean \ + --quiet \ + --time + +cm run script "run-mlperf-inference _r4.1 _submission _short" \ + --device=cpu \ + --model=resnet50 \ + --precision=float32 \ + --implementation=reference \ + --backend=onnxruntime \ + --scenario=Offline \ + --clean \ + --quiet \ + --time + +... + + 0 +Organization CTuning +Availability available +Division open +SystemType edge +SystemName ip_172_31_87_92 +Platform ip_172_31_87_92-reference-cpu-onnxruntime-v1.1... +Model resnet50 +MlperfModel resnet +Scenario Offline +Result 14.3978 +Accuracy 80.0 +number_of_nodes 1 +host_processor_model_name Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz +host_processors_per_node 1 +host_processor_core_count 2 +accelerator_model_name NaN +accelerators_per_node 0 +Location open/CTuning/results/ip_172_31_87_92-reference... +framework onnxruntime v1.18.1 +operating_system Ubuntu 24.04 (linux-6.8.0-1009-aws-glibc2.39) +notes Automated by MLCommons CM v2.3.2. +compliance 1 +errors 0 +version v4.1 +inferred 0 +has_power False +Units Samples/s + + +``` + +You can also run the same commands using a unified CM Python API: + +```python +import cmind +output=cmind.access({ + 'action':'run', 'automation':'script', + 'tags':'run-mlperf-inference,_r4.1,_performance-only,_short', + 'device':'cpu', + 'model':'resnet50', + 'precision':'float32', + 'implementation':'reference', + 'backend':'onnxruntime', + 'scenario':'Offline', + 'clean':True, + 'quiet':True, + 'time':True, + 'out':'con' +}) +if output['return']==0: print (output) +``` + + +We suggest you to use this [online CM interface](https://access.cknowledge.org/playground/?action=howtorun) +to generate CM commands that will prepare and run MLPerf benchmarks and AI applications across different platforms. + + +See more examples of CM scripts and workflows to download Stable Diffusion, GPT-J and LLAMA2 models, process datasets, install tools and compose AI applications: + + +```bash +pip install cmind -U + +cm pull repo mlcommons@cm4mlops --branch=dev + +cm show repo + +cm run script "python app image-classification onnx" +cmr "python app image-classification onnx" + +cmr "download file _wget" --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e +cmr "python app image-classification onnx" --input=computer_mouse.jpg +cmr "python app image-classification onnx" --input=computer_mouse.jpg --debug + +cm find script "python app image-classification onnx" +cm load script "python app image-classification onnx" --yaml + +cmr "get python" --version_min=3.8.0 --name=mlperf-experiments +cmr "install python-venv" --version_max=3.10.11 --name=mlperf + +cmr "get ml-model stable-diffusion" +cmr "get ml-model sdxl _fp16 _rclone" +cmr "get ml-model huggingface zoo _model-stub.alpindale/Llama-2-13b-ONNX" --model_filename=FP32/LlamaV2_13B_float32.onnx --skip_cache +cmr "get dataset coco _val _2014" +cmr "get dataset openimages" -j + +cm show cache +cm show cache "get ml-model stable-diffusion" + +cmr "get generic-python-lib _package.onnxruntime" --version_min=1.16.0 +cmr "python app image-classification onnx" --input=computer_mouse.jpg + +cm rm cache -f +cmr "python app image-classification onnx" --input=computer_mouse.jpg --adr.onnxruntime.version_max=1.16.0 + +cmr "get cuda" --version_min=12.0.0 --version_max=12.3.1 +cmr "python app image-classification onnx _cuda" --input=computer_mouse.jpg + +cm gui script "python app image-classification onnx" + +cm docker script "python app image-classification onnx" --input=computer_mouse.jpg +cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -j -docker_it + +cm docker script "get coco dataset _val _2017" --to=d:\Downloads\COCO-2017-val --store=d:\Downloads --docker_cm_repo=ctuning@mlcommons-ck + +cmr "run common mlperf inference" --implementation=nvidia --model=bert-99 --category=datacenter --division=closed +cm find script "run common mlperf inference" + +cmr "get generic-python-lib _package.torch" --version=2.1.2 +cmr "get generic-python-lib _package.torchvision" --version=0.16.2 +cmr "python app image-classification torch" --input=computer_mouse.jpg + + +cm rm repo mlcommons@cm4mlops +cm pull repo --url=https://zenodo.org/records/12528908/files/cm4mlops-20240625.zip + +cmr "install llvm prebuilt" --version=17.0.6 +cmr "app image corner-detection" + +cm run experiment --tags=tuning,experiment,batch_size -- echo --batch_size={{ print_str("VAR1{range(1,8)}") }} + +cm replay experiment --tags=tuning,experiment,batch_size + +cmr "get conda" + +cm pull repo ctuning@cm4research +cmr "reproduce paper micro-2023 victima _install_deps" +cmr "reproduce paper micro-2023 victima _run" + +``` + + +See a few examples of modular containers and GitHub actions with CM commands: + +* [GitHub action with CM commands to test MLPerf inference benchmark](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-bert.yml) +* [Dockerfile to run MLPerf inference benchmark via CM](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-mlperf-inference/dockerfiles/bert-99.9/ubuntu_22.04_python_onnxruntime_cpu.Dockerfile) + + +Please check the [**Getting Started Guide**](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md) +to understand how CM automation recipes work, how to use them to automate your own projects, +and how to implement and share new automations in your public or private projects. diff --git a/cm/setup.py b/cm/setup.py index e3a47ed5cb..f48f654958 100644 --- a/cm/setup.py +++ b/cm/setup.py @@ -58,7 +58,7 @@ def run(self): for artifact in repo: directory=os.path.join(artifact[0], '*') ignore=False - for ignore_dir in ['__pycache__', 'build', 'egg-info', 'dist']: + for ignore_dir in ['__pycache__', 'build', 'egg-info', 'dist', 'dev', 'docs']: if ignore_dir in directory: ignore=True break @@ -99,6 +99,7 @@ def run(self): entry_points={"console_scripts": [ "cmind = cmind.cli:run", "cm = cmind.cli:run", + "cmx = cmind.cli:runx", "cmr = cmind.cli:run_script", "cmrd = cmind.cli:docker_script", "cmg = cmind.cli:gui_script",