Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Go: Allow grouping import paths for models-as-data #16941

Merged
merged 12 commits into from
Jul 11, 2024

Conversation

owen-mc
Copy link
Contributor

@owen-mc owen-mc commented Jul 9, 2024

This PR removes the need for duplicate models for different package paths. It's quite common in Go that we want our models to apply to multiple package import paths, e.g. Beego used to be at github.com/astaxie/beego, then it moved to github.com/beego/beego, and then it released v2 and moved some folders around, so that package is now at github.com/beego/beego/v2/server/web (which we model as github.com/beego/beego/server/web so that we get all future versions as well). Currently we have to have three copies of all models, e.g.

  - addsTo:
      pack: codeql/go-all
      extensible: sourceModel
    data:
      - ["github.com/astaxie/beego", "", False, "HTML2str", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]
      - ["github.com/beego/beego", "", False, "HTML2str", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]
      - ["github.com/beego/beego/server/web", "", False, "HTML2str", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]

This duplication makes it hard to be sure that we have all the models for all the import paths. It would be much better to introduce a way of specifying which package paths to use separately. With this PR, that becomes

  - addsTo:
      pack: codeql/go-all
      extensible: packageGrouping
    data:
      - ["beego", "github.com/astaxie/beego"]
      - ["beego", "github.com/beego/beego"]
      - ["beego", "github.com/beego/beego/server/web"]
  - addsTo:
      pack: codeql/go-all
      extensible: sourceModel
    data:
      - ["group:beego", "", False, "HTML2str", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]

@owen-mc owen-mc added the no-change-note-required This PR does not need a change note label Jul 9, 2024
@owen-mc owen-mc requested a review from a team as a code owner July 9, 2024 15:23
@github-actions github-actions bot added the Go label Jul 9, 2024
Copy link
Contributor

github-actions bot commented Jul 9, 2024

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

go

Generated file changes for go

  • Changes to framework-coverage-go.rst:
-    `beego <https://beego.me/>`_,"``github.com/astaxie/beego*``, ``github.com/beego/beego*``",63,49,
+    `beego <https://beego.me/>`_,"``github.com/astaxie/beego*``, ``github.com/beego/beego*``",63,63,
-    Totals,,83,884,24
+    Totals,,83,898,24
  • Changes to framework-coverage-go.csv:
- github.com/beego/beego/context,,15,,,,15,,
+ github.com/beego/beego/context,,15,1,,,15,1,
+ github.com/beego/beego/utils,,,13,,,,13,

@owen-mc
Copy link
Contributor Author

owen-mc commented Jul 10, 2024

All test failures are now changes in provenance information, which I will fix at the last minute, so this is ready to review.

FlowExtensions::summaryModel(p, type, subtypes, name, signature, ext, input, output, kind,
provenance, madId)
|
not exists(string s | p = groupPrefix() + s) and package = p
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just not p.matches(groupPrefix() + "%")?

Copy link
Contributor Author

@owen-mc owen-mc Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's very hard to know what's faster. I'm pretty sure that what I currently have is dealt with well by the optimizer. But I remember now that QL strings have a prefix() method, which surely must be performant, so I'll use that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The packageOrGroup.prefix(groupPrefix().length()) = groupPrefix() should actually be expected to be worse than both of those other options, since it forces the evaluator to manifest all those packageOrGroup-prefixes in the string pool. The matches-version is fine, but I expect exists(string s | p = groupPrefix() + s) to be the best version. We really need a builtin string.startsWith predicate to make it such that the nicest version is also the best version for performance.

string groupPrefix() { result = "group:" }

/** Gets a group that `package` is in, according to `packageGrouping`. */
private string getGroup(string package) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest call this getPackage(string packageOrGroup), and push the "if is group then resolve, else result = input" logic in here (with a bindingset, so it gets inlined at use sites).

/**
* Holds if a source model exists for the given parameters.
*
* Note that we consider all packages in the same group.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Note that we consider all packages in the same group.
* Note that `group:` references are expanded into one or more actual packages by this predicate.

(here and elsewhere)

@owen-mc owen-mc force-pushed the go/mad-package-alias branch from f1d818b to 2c7fbda Compare July 10, 2024 15:48
smowton
smowton previously approved these changes Jul 10, 2024
@owen-mc owen-mc merged commit 5bdef38 into github:main Jul 11, 2024
13 checks passed
@owen-mc owen-mc deleted the go/mad-package-alias branch July 11, 2024 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Go no-change-note-required This PR does not need a change note
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants