QDM 1.3.0 #24

query-jeremy · 2024-08-23T17:42:17Z

Query Data Model v1.3.0

This PR upgrades QDM's OCSF dependency to OCSF v1.3.0.

It also cherry picks the following upstream OCSF issues that are slated for OCSF 1.4 (anticipated in November):

OSINT Inventory Info discovery event Create OSINT Inventory Info Discovery Event ocsf/ocsf-schema#1154
user Object expansion and related Observables creation user Object expansion and related Observables creation ocsf/ocsf-schema#1155
Expand applicability and direction of cloud.account and cloud.org & deprecate project_uid Expand applicability and direction of cloud.account and cloud.org & deprecate project_uid ocsf/ocsf-schema#1166

Included in the PR is the output of qdm-curator: a pickled Schema Patch, an updated ordering.pkl, and a new exported schema.json file.

It passes qdm-check:

 QDM Export Validator
======================

╔════════════════════╗
║ Must be valid JSON ║
╚════════════════════╝
The file must be parsable as JSON using json.loads().

  [SUCCESS] No findings

╔═══════════════════════════════════╗
║ Must match the expected structure ║
╚═══════════════════════════════════╝
There must be both events and objects, and each must match the expected
structure of the OCSF schema export. This means that required fields including
requirement, type, and caption must be present for each object, event, and
attribute. It also means that include directives should not be present, and any
extends directives should be a string rather than a list of strings.

  [SUCCESS] No findings

╔═════════════════════════════╗
║ Events must have Class UIDs ║
╚═════════════════════════════╝
Each event class must have a Class UID.

  [SUCCESS] No findings

╔════════════════════════════╗
║ Events must have Type UIDs ║
╚════════════════════════════╝
Each event class must have a valid Type UID enum.

  [SUCCESS] No findings

╔═════════╗
║ Summary ║
╚═════════╝
  [ ✓ PASS ] Must be valid JSON
  [ ✓ PASS ] Must match the expected structure
  [ ✓ PASS ] Events must have Class UIDs
  [ ✓ PASS ] Events must have Type UIDs

Signed-off-by: Michael Radka <mradka@splunk.com>

Update ext versions, add siblings

This addresses use-cases where the the activity was an Update but the changes were unable to be captured such as a new file.owner, file.size, file.name, etc.

Signed-off-by: Michael Radka <mradka@splunk.com>

Signed-off-by: Rajas Panat <rajaspa@amazon.com>

[bugfix]: Adding "unknown" enum value to query_result_id

Fix grammar in status_detail desc/caption

Signed-off-by: Michael Radka <mradka@splunk.com>

[nonbreaking] Minor indentation fix for readability

Signed-off-by: Alan Pinkert <apinkert@cisco.com>

Clarify how to reference profiles in metadata

Signed-off-by: Rajas Panat <rajaspa@amazon.com>

… object Signed-off-by: Rajas Panat <rajaspa@amazon.com>

…section Signed-off-by: Rajas Panat <rajaspa@amazon.com>

Signed-off-by: Michael Radka <mradka@splunk.com>

…ction. Signed-off-by: Dave McCormack <dmccorma@cisco.com>

Signed-off-by: Dave McCormack <dmccorma@cisco.com>

Closes ocsf#1043

Address gaps in Process Activity and Memory Activity wrt process injection.

#### Related Issue: ocsf#1061 _Support linting of enum and sibling conventions_ #### Description of changes: * Adds a `suppress_checks` option to the metaschema to configure turning off certain linting rules * Turns off those linting checks for places where we have violated the conventions (there are about 3) * Fixes `data_lifecycle_state_id` to follow the enum convention by adding a 99 (Other) enumerand and articulating that _it_ should be used for "other" --------- Signed-off-by: Donovan Kolbly <donovan@rscheme.org> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

…sf#1137) #### Related Issue: n/a #### Description of changes: 1. Adding `credential_uid` as an observable type, type_id 19. 2. Misc Changelog fixes --------- Signed-off-by: Rajas Panat <rajaspa@amazon.com> Signed-off-by: Rajas <89877409+floydtree@users.noreply.github.com>

Add USG-1 extension reservation @ 990. --------- Signed-off-by: k2niner <120660286+k2niner@users.noreply.github.com>

#### Related Issue: [ocsf#1124](ocsf#1124) #### Description of changes: - Added the pre-existing `job` attribute to the `Evidence Artifacts` object. - Adjusted the `at_least_one` constraint in the object to include `job`. Note that this approach is the same as that taken to fix other gaps in the `Evidence Artifacts` object, e.g. PR ocsf#1078. Signed-off-by: Dave McCormack <dmccorma@cisco.com> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

@dkolbly

…csf#1144) #### Related Issue: surfaced by @dkolbly in slack #### Description of changes: 1. Fixing event class names in the Remediation category to avoid name collision within the framework 2. No need for a changelog entry, this is fixing a new item added in 1.3.0-dev Signed-off-by: Rajas Panat <rajaspa@amazon.com>

…t 4662 (ocsf#1114) Adjust Entity Management class (3004) to be aligned with fields exist in Windows event 4662 - “An operation was performed on an object”. https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4662 #### Related Issue: ocsf#1090 #### Description of changes: We add the attributes access_list, access_mask. ![Screenshot 2024-06-04 at 15 50 27](https://github.com/ocsf/ocsf-schema/assets/100218904/5417d9a9-5956-441c-b173-437183875f49) Signed-off-by: Eliraz Levi [eliraz.levi@hunters.ai](mailto:eliraz.levi@hunters.ai) Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

…nt of `src_endpoint` (ocsf#1147) This adds support for representing a network endpoint listening for new network connections on a network. The listening network endpoint will always be the `dst_endpoint`, and there is no `src_endpoint` because no network connection has been established yet. Therefore `src_endpoint` is changed from required to recommended. The rational behind using `Network Activity` is that when a data consumer asks their data set for `Network Activity`, they will probably be interested in things that are listening on the network even if there wasn't a connection established yet. This is in contrast to needing to ask the data set for a different event type. Signed-off-by: Mitchell Wasson <miwasson@cisco.com> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

@pagbabian-splunk

…ee specific usage' in the description (ocsf#1146) This change was a suggestion from @pagbabian-splunk in ocsf#1111 After this PR, all enums that are defined in `dictionary.json` with only the nominal values (0/99 or 99) also have 'See specific usage' in the description. The effect is that a warning will be generated by the OCSF server if these attributes are used without overriding the description. The process for these changes was: - Decide if the attribute name is general enough to have other uses or not. - If the name is general enough, the definition in `dictionary.json` only has 0/99 and 'See specific usage' is in description. The use of the enum has the additional enum values and a description override. - If the name is very specific, then the whole enum definition is now in `dictionary.json` and the use of it only specifies optionality. Signed-off-by: Mitchell Wasson <miwasson@cisco.com> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

Add two additional attributes for the Software Package object. In both instances the pre-existing attribute is used to uniquely identify the package. This can be useful when looking at Software Bill of Materials (SBOM) inventory data. #### Description of changes: Add cpe_name attribute. Common Platform Enumeration CPE is used to uniquely identity software packages. Add hash attribute. A cryptographic hash is another common way to identify software packages. These changes pass local validation testing. --------- Signed-off-by: Jason Reimer <jason.reimer@tanium.com> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

#### Related Issue: N/A #### Description of changes: Added a new object that encapsulates the `duration_avg_xx` attributes with a `type_id` discriminator and a `just_one` constraint. --------- Signed-off-by: Paul Agbabian <pagbabian@splunk.com> Signed-off-by: pagbabian-splunk <pagbabian@symantec.com> Co-authored-by: pagbabian-splunk <pagbabian@symantec.com>

#### Related Issue: N/A (from Slack) #### Description of changes: Added ip to the load_balancer object. Added Load Balancer to the endpoint type_id enum list. --------- Signed-off-by: Paul Agbabian <pagbabian@splunk.com> Signed-off-by: pagbabian-splunk <pagbabian@symantec.com> Signed-off-by: Rajas <89877409+floydtree@users.noreply.github.com> Co-authored-by: pagbabian-splunk <pagbabian@symantec.com> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

#### Related Issue: 1148 ocsf#1148 #### Description of changes: extending - enrichment by: time, recommended - The timestamp when the enrichment data was generated. desc optional - A long description of the enrichment data. reputation optional - The reputation of the enrichment data short_desc, recommended - A short description of the enrichment data. url_string, recommended - The URL of the source of the enrichment data --------- Signed-off-by: Pavel Jurka <pavel.jurka@sentinelone.com> Signed-off-by: Rajas <89877409+floydtree@users.noreply.github.com> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

ocsf#1150) #### Related Issue: n/a #### Description of changes: 1. Making `resources` available in Vuln Finding and Compliance Finding event classes. 2. Deprecating `resource` attribute. 3. Fixing desc in Data Security Finding event class --------- Signed-off-by: Rajas Panat <rajaspa@amazon.com>

Related Issue: Missing enable/disable state Ids Description of changes: added state id's to Device Config State Change Class. Signed-off-by: Sasha Selin (Cyrebro) (sasha.selin@cyrebro.io) Following closed PR ocsf#1076 (ocsf#1076), Ive created new PR to create disable/enable state to "device_config_state_change" class. state “disable/enable” is very common when it comes to FortiGate logs, especially where the subtype=”system” and action=”add”. The “status” field on this type of logs are represent the “cfgattr” (Configuration value changed) status. Raw log for example: <118>date=2024-05-01 time=11:43:38 devname="Test for OCSF" devid="FG11256985563" eventtime=1714553018203018280 tz="+0300" logid="0100044547" type="event" subtype="system" level="information" vd="North" logdesc="Object attribute configured" user="SashaS" ui="GUI(192.168.190.54)" action="Add" cfgtid=10691505 cfgpath="firewall.policy" cfgobj="136" cfgattr="status[disable]srcintf[OCSF-Test]dstintf[OCSF-Test]srcaddr[Sasha-selin-ocsf-test]dstaddr[Sasha-selin]srcaddr6[]dstaddr6[]src-vendor-mac[]action[accept]schedule[always]service[RDP]groups[]users[]fsso-groups[]comments[ (Copy of 148)]custom-log-fields[]" msg="Add firewall.policy 136" ![image](https://github.com/user-attachments/assets/fcd7991a-aec8-4fe1-b511-3cc2173da6a8) --------- Signed-off-by: SashaSelin <145011693+SashaSelin@users.noreply.github.com> Signed-off-by: Rajas <89877409+floydtree@users.noreply.github.com> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

#### Related Issue: n/a Releasing 1.3.0 #### Description of changes: 1. Updating core and platform extension versions. 2. Updating Changelog. --------- Signed-off-by: Rajas Panat <rajaspa@amazon.com>

Adds a `OSINT Inventory Info` event to the Discovery category to represent retrieval of OSINT, CTI, and other enrichment data from TIPs, XDRs, and other sources of OSINT/CTI --------- Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

Expands the `user` object to add relevant data that comes from various Identity Providers or Directories while keep relevance with LDAP and MITRE D3FEND. - Add Observable `type_id` 31-35 for User UID, Group Name, Group UID, Account Name, Account UID - Add `phone_number` to `user` and to `ldap_person` - this attribute can be assigned to both or one or the other depending on the upstream system. For instance Entra ID or Okta - ~~Add `state_id` and `state` to `user` to represent the various states of a user record in a directory or IDP such as their provisioning status, (de)activation. This is 1:1 with Okta with an extra `Deleted` enum added for Google Workspace~~ Removed as ocsf#1136 already has a solution - Add `has_mfa` Boolean to Dictionary and `user` object as a quick way to tell if a `user` has MFA/2FA enabled/assigned to them --------- Signed-off-by: Jonathan Rau <139361268+jonrau-at-queryai@users.noreply.github.com> Co-authored-by: Rajas <89877409+floydtree@users.noreply.github.com>

@floydtree

… & deprecate `project_uid` (ocsf#1166) #### Related Issue: #### Description of changes: As per conversations with @floydtree and @zschmerber, some better guidance and applicability of existing `org` and `account` objects was required to account for the various ways that logical compartmentalization are defined in various public cloud and SaaS tools. For instance, GCP has Org -> Folder -> Project, OCI has Domain -> Tenancy -> Compartment, AWS has Org -> OU -> Account, and various SaaS tools have high level compartmentalization such as Servicenow Instances, M365 Tenants, Salesforce Accounts, etc. - Deprecate `project_uid` as it was hyper-specific to GCP and doesn't fit other CSPs or SaaS, removed `project_uid` from `cloud`. - Update all descriptions within `org` and `account` to reflect the applicability to CSP and SaaS platforms with more examples for mappers. - Added several new `account.type_id` to reflect AWS Account-like equivalents for Azure, GCP, OCI, Salesforce, M365, and Servicenow.

dictionary.json

srotsinha · 2024-08-26T07:15:55Z

dictionary.json

@@ -113,14 +108,14 @@
    "affected_code": {
      "caption": "Affected Code",
      "description": "List of Affected Code objects that describe details about code blocks identified as vulnerable.",
-      "is_array": true,
-      "type": "affected_code"
+      "type": "affected_code",


I wish all keys were alphabetically sorted. The change have alphabetical sorting for event names but not attributes.

srotsinha

Looks good. There are no additions in archive which makes this upgrade easier. I have not run codegen with export but this PR can be merged irrespectively.

srotsinha · 2024-08-26T07:37:01Z

migrations/curate-qdm-1.2.3-qdm-1.3.0.toml

+#   Any matching paths below will be deprecated in the schema patch. They will
+#   be preserved in the new schema, but with an @deprecated annotation.
+deprecate = [
+    #"events.email_delivery_activity.*",


We can safely remove email_delivery_activity since It has been updated to EmailAcitivity in all the drivers.

mikeradka and others added 30 commits April 12, 2024 17:50

Remove protocol_id from dict and tunnel class

52ca091

Signed-off-by: Michael Radka <mradka@splunk.com>

Update desc with removal of protocol_id

d760de6

Signed-off-by: Michael Radka <mradka@splunk.com>

Merge pull request ocsf#1039 from mikeradka/main

ee35ed9

Update ext versions, add siblings

feat: add file_result to File Hosting Activity

c3adb85

This addresses use-cases where the the activity was an Update but the changes were unable to be captured such as a new file.owner, file.size, file.name, etc.

Fix grammar in status_details desc/caption

137d3cd

Signed-off-by: Michael Radka <mradka@splunk.com>

adding unknown enum value to query_result_id

150db6d

Signed-off-by: Rajas Panat <rajaspa@amazon.com>

Merge pull request ocsf#1050 from floydtree/enum_fix

ed3eb6a

[bugfix]: Adding "unknown" enum value to query_result_id

Merge pull request ocsf#1049 from mikeradka/fix_grammar_in_dict

9567e4c

Fix grammar in status_detail desc/caption

[nonbreaking] Minor indentation fix

357b652

Signed-off-by: Michael Radka <mradka@splunk.com>

Merge pull request ocsf#1052 from mikeradka/fix_indentation

beaa9bc

[nonbreaking] Minor indentation fix for readability

attempt colorized output

eaa9512

try 2

55c8d50

try 3

e17e743

try 4

df05ea1

changelog

7070e7f

clarify how to reference profiles

b06a773

changelog

7f8a174

Merge branch 'main' into improve-profiles-description

987839d

Signed-off-by: Alan Pinkert <apinkert@cisco.com>

Merge pull request ocsf#1056 from alanisaac/improve-profiles-description

d11a59e

Clarify how to reference profiles in metadata

bugfix: logon_type_id enum correction

6bb4f64

Signed-off-by: Rajas Panat <rajaspa@amazon.com>

bugfix: adding type_name, deprecating type attribute in related_event…

841f659

… object Signed-off-by: Rajas Panat <rajaspa@amazon.com>

Porting over 1.2.0 changelog to main, moving new items to unreleased …

b975332

…section Signed-off-by: Rajas Panat <rajaspa@amazon.com>

Merge pull request ocsf#1058 from floydtree/bugfixes

e542cd0

1057 Update linux_users caption

7b31e89

Signed-off-by: Michael Radka <mradka@splunk.com>

Address gaps in Process Activity and Memory Activity wrt process inje…

acc023e

…ction. Signed-off-by: Dave McCormack <dmccorma@cisco.com>

Update CHANGELOG.md per submission guidelines.

597d7fa

Signed-off-by: Dave McCormack <dmccorma@cisco.com>

Fixed CHANGELOG.md update following feedback.

6f60bd0

Signed-off-by: Dave McCormack <dmccorma@cisco.com>

feat: adds ext attribute to File object

772beb2

Closes ocsf#1043

1

795e08b

Merge pull request ocsf#1060 from davemcatcisco/main

e093f0c

Address gaps in Process Activity and Memory Activity wrt process injection.

dkolbly and others added 22 commits July 9, 2024 14:50

Update extensions.md to include US Gov (ocsf#1140)

969e9f0

Add USG-1 extension reservation @ 990. --------- Signed-off-by: k2niner <120660286+k2niner@users.noreply.github.com>

v1.3.0 Release (ocsf#1152)

c8bde8c

#### Related Issue: n/a Releasing 1.3.0 #### Description of changes: 1. Updating core and platform extension versions. 2. Updating Changelog. --------- Signed-off-by: Rajas Panat <rajaspa@amazon.com>

Merge remote-tracking branch 'upstream/v1.3.0' into qdm_1.3.0

133b84e

version bump

42cf4e8

Removing old and conflicting reg key and value objects

499951f

Running curator

de6d2a7

query-jeremy requested review from srotsinha and jonrau-at-queryai August 23, 2024 17:52

jonrau-at-queryai approved these changes Aug 23, 2024

View reviewed changes

srotsinha reviewed Aug 26, 2024

View reviewed changes

dictionary.json Show resolved Hide resolved

srotsinha reviewed Aug 26, 2024

View reviewed changes

srotsinha approved these changes Aug 26, 2024

View reviewed changes

query-jeremy merged commit a371867 into main Aug 27, 2024
2 checks passed

query-jeremy deleted the qdm_1.3.0 branch August 30, 2024 18:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QDM 1.3.0 #24

QDM 1.3.0 #24

query-jeremy commented Aug 23, 2024 •

edited

Loading

srotsinha Aug 26, 2024

srotsinha left a comment

srotsinha Aug 26, 2024

QDM 1.3.0 #24

QDM 1.3.0 #24

Conversation

query-jeremy commented Aug 23, 2024 • edited Loading

Query Data Model v1.3.0

srotsinha Aug 26, 2024

Choose a reason for hiding this comment

srotsinha left a comment

Choose a reason for hiding this comment

srotsinha Aug 26, 2024

Choose a reason for hiding this comment

query-jeremy commented Aug 23, 2024 •

edited

Loading