feat: add the model processor proposal #257

chlins · 2025-01-15T09:36:03Z

vivek

Thanks @chlins for authoring this proposal. Overall it looks good. The key aspects of the proposal make it appealing. My takeaways:

This proposal brings AI models as first class citizen in Harbor. The user stories to use this CNAI model spec which follows OCI spec to surface AI model
Uses existing OCI spec (artifactType and mediaType ) along with following CNAI model format https://github.com/CloudNativeAI/model-spec/blob/main/docs/spec.md
No additional APIs needed as it leverages additions API - This is great

I left a comment regarding future model format changes and how new additions can be handled in generic way without requiring code changes. PTAL, I might be off as new to Harbor at code level, as long as we can handle such use case, its all looking good.

proposals/new/AI-model-processor.md

steven-zou · 2025-02-17T11:46:16Z

proposals/new/AI-model-processor.md

+
+## Architecture
+
+There will be a new AI model processor added to the artifact processors in Harbor core.


It's better to give details about how the new processor processes the data following the model spec into the Harbor artifact object which supports the related APIs shown in the API sections.
|---->[CONFIG]-
[model spec] ----> (processor)----> [artifact object] ------>（API)
|---->[LAYERS]-

proposals/new/AI-model-processor.md

s3rj1k · 2025-02-20T14:59:31Z

Just for reference, I am using below Modelfile (https://huggingface.co/google/gemma-2-2b), Modelfile is inside artifact directory (gemma-2-2b/Modelfile)

# https://huggingface.co/google/gemma-2-2b/tree/main

NAME gemma-2-2b
ARCH transformer
FAMILY gemma
FORMAT safetensors
PARAMSIZE 2
PRECISION bf16
QUANTIZATION none

CONFIG config.json
CONFIG generation_config.json
CONFIG special_tokens_map.json
CONFIG tokenizer_config.json
CONFIG tokenizer.json
CONFIG model.safetensors.index.json

MODEL tokenizer.model
MODEL *.safetensors

Below how oras sees above model after it is pushed into registry

# oras manifest fetch domain.net/library/gemma-2-2b:latest | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "artifactType": "application/vnd.cnai.model.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.cnai.model.config.v1+json",
    "digest": "sha256:c9f3f1a5131634d8a8c59f2085bf6745a6b2f1f13bd80019d5635f74b57e2372",
    "size": 264
  },
  "layers": [
    {
      "mediaType": "application/vnd.cnai.model.doc.v1.tar",
      "digest": "sha256:f2bd54bd4695332a5c9fffb5bd736aa3a47a933c03bd4fa1fca508c9f607a1a1",
      "size": 27648,
      "annotations": {
        "org.cnai.model.filepath": "README.md"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.config.v1.tar",
      "digest": "sha256:0ad46a0e2c2bfb03aeacef30f5cfdf08fc57e1e2109902594a955d1501d6f998",
      "size": 2048,
      "annotations": {
        "org.cnai.model.filepath": "generation_config.json"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.config.v1.tar",
      "digest": "sha256:4c09b0c73a63882fbbd689a085ec52e83f42f97a2c07ffaaeecdab14ca922aaf",
      "size": 2560,
      "annotations": {
        "org.cnai.model.filepath": "special_tokens_map.json"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.config.v1.tar",
      "digest": "sha256:821088b4b19de1ee9564acc8983345a68150839da75755f50fbb78c8013ad0ea",
      "size": 48128,
      "annotations": {
        "org.cnai.model.filepath": "tokenizer_config.json"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.config.v1.tar",
      "digest": "sha256:3e86b2521cd1e5516a4c7aa09efeb94af5cfb6d842bcb6f4b84ff798ede8a4ea",
      "size": 17527296,
      "annotations": {
        "org.cnai.model.filepath": "tokenizer.json"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.config.v1.tar",
      "digest": "sha256:d85985abcd1e05389abb9088251ca703c3307f946aa693d549f0967558f709f4",
      "size": 26112,
      "annotations": {
        "org.cnai.model.filepath": "model.safetensors.index.json"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.config.v1.tar",
      "digest": "sha256:dbc9cf744f8ff3e596924765d8a91c7bed4b7939c73b18a373905b1b31d81b85",
      "size": 2560,
      "annotations": {
        "org.cnai.model.filepath": "config.json"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.v1.tar",
      "digest": "sha256:746911d5850fdd7bc058de4f294c02d282f4e99422df18d31dfb83beb7aa8eed",
      "size": 4242944,
      "annotations": {
        "org.cnai.model.filepath": "tokenizer.model"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.v1.tar",
      "digest": "sha256:2d1be53f694599fe05d2f57ba6b9028e766f8a585225d87643ad984c351dd7b3",
      "size": 4992578048,
      "annotations": {
        "org.cnai.model.filepath": "model-00001-of-00003.safetensors"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.v1.tar",
      "digest": "sha256:5fa6ff435737b221e75b771450c9c09b0a3695c73815776450adab71b24ab4a1",
      "size": 4983444992,
      "annotations": {
        "org.cnai.model.filepath": "model-00002-of-00003.safetensors"
      }
    },
    {
      "mediaType": "application/vnd.cnai.model.weight.v1.tar",
      "digest": "sha256:703f96fc5ba4fee31dd14b537da139da015bb4d3351a212209050c2e68acfc75",
      "size": 481383424,
      "annotations": {
        "org.cnai.model.filepath": "model-00003-of-00003.safetensors"
      }
    }
  ]
}

as opposed to Kitfile from KitOps

# oras manifest fetch domain.net/library/gemma-2-2b:latest | jq
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.kitops.modelkit.config.v1+json",
    "digest": "sha256:4b58400f17b967152ab8ac022946896cd55575f6076e2f79781f1e6e21ffbd0a",
    "size": 2170
  },
  "layers": [
    {
      "mediaType": "application/vnd.kitops.modelkit.model.v1.tar",
      "digest": "sha256:f3b5a4b1c6065dbdeac2afa977d60b110bb84a10455f6e0d6c94c33b7a97cdeb",
      "size": 4992578048
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.modelpart.v1.tar",
      "digest": "sha256:b22bcdad944d6588dd92314753482cc2c2265e5231425e2846529a51a9b2393a",
      "size": 4983444992
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.modelpart.v1.tar",
      "digest": "sha256:d8554286602a841a31bd9a88d3fa2ec85d6aa8b1eb9b585efb10681d22beae35",
      "size": 481383424
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.modelpart.v1.tar",
      "digest": "sha256:d41f170f157a76a8e6fe223726bace7d1b3eb865532488297d96bb5a37adfdcc",
      "size": 2560
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.modelpart.v1.tar",
      "digest": "sha256:d77ba08b9e8b7c7cd4dfe49f0df71f17d7f9173280ea23b3e8ad35e4a2eef1bd",
      "size": 2048
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.modelpart.v1.tar",
      "digest": "sha256:fa1f3ea472adaaf49d62295483bc7423cccdc62e67064b0acc3a2103866a5057",
      "size": 26112
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.modelpart.v1.tar",
      "digest": "sha256:2428f162d376dd61b077463d15ba0a04300697ba888686515d93afdd6fdeb403",
      "size": 2560
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.modelpart.v1.tar",
      "digest": "sha256:5d6bc03de874683603f9bae07c7c79195cad9821e3ed5484361ec59c7191e5ae",
      "size": 17527296
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.modelpart.v1.tar",
      "digest": "sha256:d922ba9526e31d91f7d757c4634ab5d861ed433c19a9906faf7715825a025713",
      "size": 48128
    },
    {
      "mediaType": "application/vnd.kitops.modelkit.code.v1.tar",
      "digest": "sha256:79495fe9531fb5f2eb8181a64ffe6659613f10254d1ad7bc5ced38f098550453",
      "size": 4245504
    }
  ],
  "annotations": {
    "ml.kitops.modelkit.cli-version": "1.1.0"
  }
}

Signed-off-by: chlins <chlins.zhang@gmail.com>

reasonerjt

lgtm

vivek

lgtm

s3rj1k · 2025-02-22T12:39:29Z

proposals/new/AI-model-processor.md

+| 404 | layer with README characteristics not found |
+| 500 | internal server error(other runtime error or unexpected error) |
+
+`GET /api/v2.0/projects/{project_name}/repositories/{repository_name}/artifacts/{reference}/additions/readme`


should endpoint end in readme.md, like for other processors?

yes, it should be readme.md.

s3rj1k · 2025-02-24T09:46:10Z

We also should consider changing model name, reference here https://github.com/goharbor/harbor/pull/21663/files#r1967296025

"MODEL" -> "CNAI" for ArtifactTypeCNAI

Harbor can have multiple ML model processors for different specs, having this "MODEL" is so general, it will look strange when other models specs processors will be added.

chlins requested review from a team as code owners January 15, 2025 09:36

github-actions bot assigned MinerYang, stonezdj and wy65701436 Jan 15, 2025

vivek suggested changes Jan 24, 2025

View reviewed changes

proposals/new/AI-model-processor.md Show resolved Hide resolved

proposals/new/AI-model-processor.md Show resolved Hide resolved

reasonerjt reviewed Jan 26, 2025

View reviewed changes

chlins force-pushed the feat/ai-model-processor branch 2 times, most recently from fc4e42c to 6c82d47 Compare February 17, 2025 08:58

steven-zou reviewed Feb 17, 2025

View reviewed changes

feat: add the model processor proposal

11106e9

Signed-off-by: chlins <chlins.zhang@gmail.com>

chlins force-pushed the feat/ai-model-processor branch from 6c82d47 to 11106e9 Compare February 21, 2025 02:54

reasonerjt approved these changes Feb 21, 2025

View reviewed changes

s3rj1k mentioned this pull request Feb 21, 2025

Add CNAI processor goharbor/harbor#21659

Open

vivek approved these changes Feb 21, 2025

View reviewed changes

s3rj1k reviewed Feb 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add the model processor proposal #257

feat: add the model processor proposal #257

chlins commented Jan 15, 2025 •

edited

Loading

vivek left a comment •

edited

Loading

steven-zou Feb 17, 2025

s3rj1k commented Feb 20, 2025

reasonerjt left a comment

vivek left a comment

s3rj1k Feb 22, 2025

chlins Feb 24, 2025

s3rj1k commented Feb 24, 2025


		## Architecture

		There will be a new AI model processor added to the artifact processors in Harbor core.

feat: add the model processor proposal #257

Are you sure you want to change the base?

feat: add the model processor proposal #257

Conversation

chlins commented Jan 15, 2025 • edited Loading

vivek left a comment • edited Loading

Choose a reason for hiding this comment

steven-zou Feb 17, 2025

Choose a reason for hiding this comment

s3rj1k commented Feb 20, 2025

reasonerjt left a comment

Choose a reason for hiding this comment

vivek left a comment

Choose a reason for hiding this comment

s3rj1k Feb 22, 2025

Choose a reason for hiding this comment

chlins Feb 24, 2025

Choose a reason for hiding this comment

s3rj1k commented Feb 24, 2025

chlins commented Jan 15, 2025 •

edited

Loading

vivek left a comment •

edited

Loading