Feature request for plugin architecture to allow for dynamic policy evaluation

[cbusbey]

I have a use case that I do not think is handled by existing configuration.

Context

• A file exists in a repo containing structured data (ie JSON, prototext, etc)
• Within the structured data are stanzas affecting some set of stakeholders
• The file contains many stanzas with many different stakeholder groups

Requirement

• When a change is requested for some group of stanzas, only require reviews from those affected stakeholders
• Do not require reviews from stakeholders not affected by the change (as opposed to spamming all stakeholders for review)
• policy-bot can automatically request review from affected stakeholders
• Users can view affected stakeholders that need provide a review in the policy-bot ui

Proposed implementation

• Allow a plugin architecture (webhook?) that can determine policy using the PR change-set as context

As a work around I have broken the structured data into individual files for each stanza with assigned reviewers, but this has added complexity to downstream systems that depend on this structured data. In my particular use case, it would be trivial to identify affected stakeholders based on the delta of the structured data, I just need the hook from the central policy-bot service to install.

This run-time plugin could open other use cases. For example, consider a policy that leverages an IAM outside of GitHub teams.

Is this a feature that has been previously identified?

[bluekeyes]

This sounds similar to #184 and is something that has also come up in our internal discussions. Part of the challenge is designing a flexible way to write the relevant content rules. I don't think we considered a plugin/webhook approach in the past, so that is worth thinking about. In either case, we need to carefully consider how to implement and support something like this before committing to adding any extension points.

One alternative that works today is splitting up the files, as you are doing. Another option (that is kind of similar to a webhook) is to deploy your own application that posts status checks or check runs based on the content of the modified files. You can then write Policy Bot rules that are conditional on the presence of these different status checks. For example:

- name: stakeholder-one approval
  if:
    has_status:
      conclusions: ["success"]
      statuses:
        - "config/requires-stakeholder-one-approval"
  options:
    request_review:
      enabled: true
      mode: teams
  requires:
    count: 1
    teams:
      - "org/stakeholder-one"

Your app posts a successful config/requires-stakeholder-one-approval on the commit if it modifies part of the configuration that requires the stakeholder-one team to approve.

[cbusbey]

Thanks @bluekeyes. The config/requires-stakeholder-one-approval is another workaround, I'll have to experiment to figure out if it is tenable for our use case.

Ideally the system could be configured to automatically request review from those stakeholders and, and display those stakeholders in the UI as it does for required approvals at the file level today. - I don't believe this workaround solves for this. Just spitballing, is it possible to pass additional data from custom status checks (like config/requires-stakeholder-one-approval) to policy-bot?