VE-3102: DMH vector search: AION updates #590

ndthang15 · 2024-06-24T10:00:03Z

Overview

AION needs to be updated with additional properties that could be used by GLC and DMH vector search.

Description

Added referenceId property to objectResult and seriesItem definitions.
Added embedding object (with vector, tags, and referenceId properties) to a new top-level in the AION schema.
Added tags property to the top level of objectResult.

{
    "series": [{
        "start": 0,
        "stop": 1000,
        "referenceId": "ABC" // this is a new property added to the seriesItem definition.
    },
    {
        "start": 2000,
        "stop": 3000,
        "referenceId": "ABC"
    }],
    "embedding": [{  // "embedding" is on the same level as "object" and "series" in the AION schema.
        "vector": [0.0, 0.1]
        "tags": [],
         "referenceId": "ABC"
    },
    {
        "vector": [2.0, 2.1]
        "tags": [],
         "referenceId": "ABC"
    }],
}

Updated AION schema with GLC prototype.

{
    "sourceEngineId": "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", // existing property in the PREAMBLE definition.
    "internalTaskId": "eb20f1b3-20b4-471d-8dd9-76e077969af4", // this is a new property added to the PREAMBLE definition.
    "generatedDateUTC": "2024-05-15T15:27:33.848266649Z",
    "validationContracts": [
        "text"
    ],
	"object" : 
	[
		{
			"fingerprintVector" : [ // this is a new property added to the objectResult definition.
				// ...
			],
			"label" : "person",
			// this used to link timeSeries to a vector
			"referenceId" : "f46f029a-d42c-411c-ad01-8cffe3f87a63", // this is a new property added to the objectResult definition. 
			"tags" :       //  this is a new property added to the objectResult definition.
			[
				{
					"key" : "Accessory",
					"value" : "BagAny"
				}				
			],
			"type" : "fingerprint", //  existing property in the objectResult definition. 
			"vendor" : 
			{
				"label" : "fe_person_reid_512_v17_0_m34"
			}
		}
	],
	"series" : 
	[
		{
			"object" : 
			{
				"boundingPoly" : [
					// ...
				],
				"label" : "",
				"referenceId" : "f46f029a-d42c-411c-ad01-8cffe3f87a63", // this is a new property added to the objectResult definition.
				"type" : "object"
			},
			"startTimeMs" : 0,
			"stopTimeMs" : 0
		},
		{
			"object" : 
			{
				"boundingPoly" : [
					// ...
				],
				"label" : "",				
				"referenceId" : "f46f029a-d42c-411c-ad01-8cffe3f87a63", // this is a new property added to the objectResult definition.
				"type" : "object"
			},
			"startTimeMs" : 40,
			"stopTimeMs" : 40
		}				
	]
}

Added valid and invalid tests to AION schema.

Related Issue

https://veritone.atlassian.net/browse/VE-3102

How Has This Been Tested

Run npm i && npm run test in /packages/veritone-json-schemas folder.
Ensure that all tests are passed.

…eId to vtn-standard schema

orca-security-us

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0 0 0 0	View in Orca
Passed	Secrets	0 0 0 0	View in Orca
Passed	Vulnerabilities	0 0 0 0	View in Orca

mgiasiVeri · 2024-06-24T17:27:39Z

@ndthang15 we also need to support GLC's new AION format which is similar but introduces a few more fields like fingerprintVector. The ticket links to an example in the search draft doc, but the example is collapsed so you have to expand it.

Open up https://veritone.atlassian.net/wiki/spaces/ENG/pages/3330900143/Vector+Search+Draft#AION-updates and search for "GLC prototype" to expand and see it. Note that the object object does not need to have start and stop times. Here is another view of GLCs needed AION changes. https://veritone.atlassian.net/wiki/spaces/VP/pages/2859532299/Next+generation+tracker+AION+format#Accepted-Solution

ndthang15 · 2024-06-25T13:22:59Z

@ndthang15 we also need to support GLC's new AION format which is similar but introduces a few more fields like fingerprintVector. The ticket links to an example in the search draft doc, but the example is collapsed so you have to expand it.

Open up https://veritone.atlassian.net/wiki/spaces/ENG/pages/3330900143/Vector+Search+Draft#AION-updates and search for "GLC prototype" to expand and see it. Note that the object object does not need to have start and stop times. Here is another view of GLCs needed AION changes. https://veritone.atlassian.net/wiki/spaces/VP/pages/2859532299/Next+generation+tracker+AION+format#Accepted-Solution

@mgiasiVeri Today I have updated the schema to support GLC's new AION format base on your information. And this PR is ready for review again. Thank you so much for reviewing!

mgiasiVeri · 2024-06-25T20:22:01Z

@ndthang15 i think we need "tags" added at the objectResult level too. If you look at the GLC example, they leverage tags in the top level object, which is outside of the seriesItem. Let me know if you have any questions. And to be clear, we do not need the start and stop time values added at the objectResult level, that is a mistake in the examples provided.

ndthang15 · 2024-06-26T02:11:09Z

@ndthang15 i think we need "tags" added at the objectResult level too. If you look at the GLC example, they leverage tags in the top level object, which is outside of the seriesItem. Let me know if you have any questions. And to be clear, we do not need the start and stop time values added at the objectResult level, that is a mistake in the examples provided.

@mgiasiVeri You're right that I'm missing the tags in objectResult, so I've just added it in the latest commit.
About the start and stop time values at the objectResult level, we didn't add it to the schema. It is also a mistake when I copied the sample schema in document to PR's description.
Thank you so much for your notes!

crondonveritone · 2024-07-01T17:07:22Z

Based on the information of GLC prototype https://veritone.atlassian.net/wiki/spaces/ENG/pages/3330900143/Vector+Search+Draft#GLC-prototype%3A

I see that fingerPrintVector has the properties label, referenceId, and type but in the example, these properties are outer of fingerPrintVector (at #/definitions/objectResult level). If the second case is correct then that part LGTM:

crondonveritone · 2024-07-01T17:08:47Z

packages/veritone-json-schemas/schemas/vtn-standard/master.json

@@ -767,6 +785,30 @@
    "vendor": {
      "description": "Custom data that doesn't conform to any other field. You can add any arbitrary data inside this object, but it will not be indexed, searchable, or have any impact on the system. However it will be returned when reading the data back out.",
      "type": "object"
+    },
+    "fingerprintVector": {


shouldn't we define the type and range of the items in the fingerprintVector array? like

"items": { "type": "number", "minimum": 0, "maximum": 1, }

I defined the type of items in the fingerprintVector array in e534527. I think we don't need to define range for it, like the KNN example in the ES document:

curl -X POST "localhost:9200/image-index/_bulk?refresh=true&pretty" -H 'Content-Type: application/json' -d' { "index": { "_id": "1" } } { "image-vector": [1, 5, -20], "title-vector": [12, 50, -10, 0, 1], "title": "moose family", "file-type": "jpg" } { "index": { "_id": "2" } } { "image-vector": [42, 8, -15], "title-vector": [25, 1, 4, -12, 2], "title": "alpine lake", "file-type": "png" } { "index": { "_id": "3" } } { "image-vector": [15, 11, 23], "title-vector": [1, 5, 25, 50, 20], "title": "full moon", "file-type": "jpg" } ... '

Could you please review again? Thanks you!

mgiasiVeri · 2024-07-01T23:34:50Z

Based on the information of GLC prototype https://veritone.atlassian.net/wiki/spaces/ENG/pages/3330900143/Vector+Search+Draft#GLC-prototype%3A

I see that fingerPrintVector has the properties label, referenceId, and type but in the example, these properties are outer of fingerPrintVector (at #/definitions/objectResult level). If the second case is correct then that part LGTM:

@crondonveritone I think that was a typo. label, referenced, and type should all be outside of fingerprintVector. And fingerprintVector should only be an array of numbers. @frankayars @alex-oleksiiuk could you confirm.

crondonveritone

LGTM

frankayars

Suggested two minor description changes.

frankayars · 2024-07-03T20:28:41Z

packages/veritone-json-schemas/schemas/vtn-standard/master.json

@@ -767,6 +785,33 @@
    "vendor": {
      "description": "Custom data that doesn't conform to any other field. You can add any arbitrary data inside this object, but it will not be indexed, searchable, or have any impact on the system. However it will be returned when reading the data back out.",
      "type": "object"
+    },
+    "fingerprintVector": {
+      "description": "An array of vectors related to the vectorized data",


Suggestion:
An array of floats representing objects in vector space.

frankayars · 2024-07-03T20:28:53Z

packages/veritone-json-schemas/schemas/vtn-standard/master.json

+      }
+    },
+    "embedding": {
+      "description": "The embedding engine result was generated",


Suggestion:
An array of floats representing objects in vector space.

sonarqubecloud · 2024-07-04T02:38:54Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

ndthang15 · 2024-07-04T02:51:27Z

Suggested two minor description changes.

@frankayars Thanks for your suggestions. I changed the descriptions of these definitions. Also, the fingerprintVector definition was renamed 'vector'. Could you please review again?

frankayars

👍

add embedding object,seriesItem.referenceId and objectResult.referenc…

85d3d84

…eId to vtn-standard schema

ndthang15 requested a review from alex-oleksiiuk June 24, 2024 10:00

orca-security-us bot reviewed Jun 24, 2024

View reviewed changes

update AION schema with GLC prototype

42237e5

ndthang15 requested review from frankayars, stefanminkov, cantv6605-seta, tfedoris-vt, mgiasiVeri, lucaspiscitello and crondonveritone June 25, 2024 16:08

Add "tags" for the objectResult

0408acb

add valid and invalid tests to AION schema

3808bb5

crondonveritone reviewed Jul 1, 2024

View reviewed changes

define the type of the items in the fingerprintVector array

e534527

crondonveritone approved these changes Jul 2, 2024

View reviewed changes

frankayars reviewed Jul 3, 2024

View reviewed changes

rename fingerprintVector definition to vector and update descriptions

41b5991

ndthang15 requested a review from frankayars July 4, 2024 02:38

frankayars approved these changes Jul 8, 2024

View reviewed changes

mgiasiVeri approved these changes Jul 9, 2024

View reviewed changes

mgiasiVeri merged commit d44114b into master Jul 9, 2024
5 checks passed

mgiasiVeri deleted the dev/VE-3102 branch July 9, 2024 20:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VE-3102: DMH vector search: AION updates #590

VE-3102: DMH vector search: AION updates #590

ndthang15 commented Jun 24, 2024 •

edited

Loading

orca-security-us bot left a comment •

edited

Loading

mgiasiVeri commented Jun 24, 2024

ndthang15 commented Jun 25, 2024

mgiasiVeri commented Jun 25, 2024 •

edited

Loading

ndthang15 commented Jun 26, 2024

crondonveritone commented Jul 1, 2024 •

edited

Loading

crondonveritone Jul 1, 2024 •

edited

Loading

ndthang15 Jul 2, 2024 •

edited

Loading

mgiasiVeri commented Jul 1, 2024

crondonveritone left a comment

frankayars left a comment

frankayars Jul 3, 2024

frankayars Jul 3, 2024

sonarqubecloud bot commented Jul 4, 2024

ndthang15 commented Jul 4, 2024 •

edited

Loading

frankayars left a comment

VE-3102: DMH vector search: AION updates #590

VE-3102: DMH vector search: AION updates #590

Conversation

ndthang15 commented Jun 24, 2024 • edited Loading

Overview

Description

Related Issue

How Has This Been Tested

orca-security-us bot left a comment • edited Loading

Choose a reason for hiding this comment

Orca Security Scan Summary

mgiasiVeri commented Jun 24, 2024

ndthang15 commented Jun 25, 2024

mgiasiVeri commented Jun 25, 2024 • edited Loading

ndthang15 commented Jun 26, 2024

crondonveritone commented Jul 1, 2024 • edited Loading

crondonveritone Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

ndthang15 Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

mgiasiVeri commented Jul 1, 2024

crondonveritone left a comment

Choose a reason for hiding this comment

frankayars left a comment

Choose a reason for hiding this comment

frankayars Jul 3, 2024

Choose a reason for hiding this comment

frankayars Jul 3, 2024

Choose a reason for hiding this comment

sonarqubecloud bot commented Jul 4, 2024

Quality Gate passed

ndthang15 commented Jul 4, 2024 • edited Loading

frankayars left a comment

Choose a reason for hiding this comment

ndthang15 commented Jun 24, 2024 •

edited

Loading

orca-security-us bot left a comment •

edited

Loading

mgiasiVeri commented Jun 25, 2024 •

edited

Loading

crondonveritone commented Jul 1, 2024 •

edited

Loading

crondonveritone Jul 1, 2024 •

edited

Loading

ndthang15 Jul 2, 2024 •

edited

Loading

ndthang15 commented Jul 4, 2024 •

edited

Loading