Skip to content

Commit

Permalink
feat: generate Verification Summary Attestation (#592)
Browse files Browse the repository at this point in the history
This PR adds the Verification Summary Attestation generation feature to Macaron.

During the execution of the `macaron verify-policy` command, Macaron may generate a Verification Summary Attestation (VSA) with the metadata of the verification, if applicable.

Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Co-authored-by: Behnaz Hassanshahi <behnaz.hassanshahi@oracle.com>
  • Loading branch information
nathanwn and behnazh-w authored Jan 11, 2024
1 parent 62a4813 commit 10b168b
Show file tree
Hide file tree
Showing 16 changed files with 1,308 additions and 39 deletions.
1 change: 1 addition & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -112,5 +112,6 @@ intermediate representations as abstractions. Using such abstractions, Macaron i
pages/cli_usage/index
pages/tutorials/index
pages/output_files
pages/vsa
pages/supported_technologies/index
pages/developers_guide/index
1 change: 1 addition & 0 deletions docs/source/pages/developers_guide/apidoc/macaron.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Subpackages
macaron.policy_engine
macaron.repo_finder
macaron.slsa_analyzer
macaron.vsa

Submodules
----------
Expand Down
18 changes: 18 additions & 0 deletions docs/source/pages/developers_guide/apidoc/macaron.vsa.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
macaron.vsa package
===================

.. automodule:: macaron.vsa
:members:
:undoc-members:
:show-inheritance:

Submodules
----------

macaron.vsa.vsa module
----------------------

.. automodule:: macaron.vsa.vsa
:members:
:undoc-members:
:show-inheritance:
65 changes: 49 additions & 16 deletions docs/source/pages/output_files.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,13 @@ Output Files Guide

.. note:: Please see :ref:`pages/cli_usage/index:common options` for the instructions on how to set the output directory of Macaron.

-------------------
--------------------------------
Output files of macaron analyze
--------------------------------

^^^^^^^^^^^^^^^^^^^
Top level structure
-------------------
^^^^^^^^^^^^^^^^^^^

.. code-block::
Expand All @@ -25,9 +29,9 @@ Top level structure
├── macaron.db
└── sbom_debug.json
-------
^^^^^^^
Reports
-------
^^^^^^^

The report files of Macaron (from using the :ref:`analyze command <analyze-command-cli>`) are generated into the ``reports`` directory.

Expand Down Expand Up @@ -102,9 +106,9 @@ For example, for `<https://github.com/micronaut-projects/micronaut-core>`_ the r
├── dependency_2.json
└── ...
-------------------
^^^^^^^^^^^^^^^^^^^
Cloned repositories
-------------------
^^^^^^^^^^^^^^^^^^^

The ``git_repos`` directory is used to clone repositories into during the analysis. Each remote repository is cloned to a unique path
within ``git_repos`` following the same strategy as `Unique result path`_.
Expand All @@ -129,37 +133,66 @@ to the directory:
.. note:: Please see :ref:`pages/using:analyzing a locally cloned repository` to know how to set the directory for analyzing local repositories.

.. _output_files_macaron_verify_policy:

-------------------------------------
Output files of macaron verify-policy
-------------------------------------

As part of the ``macaron verify-policy`` command, Macaron generates a :ref:`Verification Summary Attestation<vsa>` (VSA) with the following strategy:

* If the Datalog policy applies to a unique software component identified by a unique PURL, a VSA is generated based on the latest analysis results for that specific software component in the Macaron database.
* Otherwise, if the Datalog policy applies to multiple software components identified by multiple different PURLs, no VSA will be generated.

The VSA file will be generated into ``output/vsa.intoto.jsonl`` by default.

.. code-block::
output/
└── vsa.intoto.jsonl
Users can manually inspect the payload of the VSA generated by Macaron with the following command:

.. code-block:: bash
cat output/vsa.intoto.jsonl | jq -r '.payload' | base64 -d | jq
For more details about the Macaron-generated VSAs, please refer to the :ref:`Verification Summary Attestation page<vsa>`.


------
Others
------

''''''''''
^^^^^^^^^^
macaron.db
''''''''''
^^^^^^^^^^

The file is the SQLite database used by Macaron for storing analysis results.

'''''''''
^^^^^^^^^
debug.log
'''''''''
^^^^^^^^^

This file stores the log messages from the latest run of Macaron.

'''''''''
^^^^^^^^^
build_log
'''''''''
^^^^^^^^^

This is the directory for storing the log from running external components such as `CycloneDx SBOM Maven plugin <https://github.com/CycloneDX/cyclonedx-maven-plugin>`_, `CycloneDx SBOM Gradle plugin <https://github.com/CycloneDX/cyclonedx-gradle-plugin>`_ or the `slsa-verifier <https://github.com/slsa-framework/slsa-verifier>`_.

'''''''''''''''
^^^^^^^^^^^^^^^
sbom_debug.json
'''''''''''''''
^^^^^^^^^^^^^^^

This file contain the debug information for running the SBOM generator to obtain dependencies of a repository.

'''''''''''''''
^^^^^^^^^^^^^^^
.m2 and .gradle
'''''''''''''''
^^^^^^^^^^^^^^^

These two directories cache the content of ``~/.m2`` and ``~/.gradle`` in the Docker container between different runs (which are
mainly updated by the CycloneDX SBOM plugins).
Expand Down
170 changes: 170 additions & 0 deletions docs/source/pages/vsa.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
=================================
Verification Summary Attestations
=================================

.. _vsa:

Macaron generates Verification Summary Attestations (VSAs) as part of its verification to communicate the fact that "some software component has been verified against a policy".

The concept of VSA in Macaron largely follows the concept of VSA in `SLSA <https://slsa.dev/spec/v1.0/verification_summary>`_ and `in-toto <https://github.com/in-toto/attestation/blob/main/spec/predicates/vsa.md>`_.


---------
Use cases
---------

The use cases of Macaron VSAs includes, but not limited to:

- **Enabling delegated verification**: This allows software consumers to make use of verification results from another party.
- **Caching verification results**: It could be expensive or inconvenient to run a full Macaron verification in certain circumstances. A VSA helps with caching and reusing verification results.


------
Schema
------

.. Type references
.. _PackageURL: https://github.com/package-url/purl-spec
.. _Envelope: https://github.com/in-toto/attestation/blob/main/spec/v1/envelope.md
.. _TypeURI: https://github.com/in-toto/attestation/blob/main/spec/v1/field_types.md#TypeURI
.. _Timestamp: https://github.com/in-toto/attestation/blob/main/spec/v1/field_types.md#timestamp
.. _ResourceURI: https://github.com/in-toto/attestation/blob/main/spec/v1/field_types.md#ResourceURI
.. _ResourceDescriptor: https://github.com/in-toto/attestation/blob/main/spec/v1/resource_descriptor.md
.. _SlsaResult: https://slsa.dev/spec/v1.0/verification_summary#slsaresult

Following in-toto attestation schema, the outermost layer if a Macaron-generated VSA is a `DSSE envelope <https://github.com/in-toto/attestation/blob/main/spec/v1/envelope.md>`_ containing a base64-encoded ``payload`` of type `in-toto Statement <https://github.com/in-toto/attestation/blob/main/spec/v1/statement.md>`_.

The following is the schema of the Statement layer:


.. code-block:: js+jinja

{
"_type": "https://in-toto.io/Statement/v1",
"subject": [
{
"uri": {{ PackageURL of the software component being verified }},
}
],
"predicateType": "https://slsa.dev/verification_summary/v1",
"predicate": {
"verifier": {
"id": "https://github.com/oracle/macaron",
"version": {
"macaron": {{ Macaron version }}
}
},
"timeVerified": {{ The timestamp of when the verification happened }},
"resourceUri": {{ PackageURL of the software component being verified }},
"policy": {
"content": {{ Datalog policy applies to the software component being verified }}
},
"verificationResult": {{ Either "PASSED" or "FAILED" }},
"verifiedLevels": []
}
}



* ``_type``: string (`TypeURI`_)
Identifier for the schema of the Statement layer. This follows `in-toto v1 Statement layer schema <https://github.com/in-toto/attestation/blob/main/spec/v1/statement.md>`_ and is always ``https://in-toto.io/Statement/v1``.

* ``subject``: array of `ResourceDescriptor`_ objects
Subjects of the VSA. Each entry is a software component being verified by Macaron.

*Note: In the current version of Macaron, this field only contains one single software component, identified by a* `PackageURL`_.

* ``predicateType``: string (`TypeURI`_)
Identifier for the type of the Predicate. For Macaron-generated VSAs, this is always ``https://slsa.dev/verification_summary/v1``.

* ``predicate``: object
The Predicate of the attestation, providing information about the verification.

* ``predicate.verifier``: object
Information about the tool running the verification, which is Macaron.

* ``predicate.verifier.id``: string (`TypeURI`_)
The identifier for Macaron.

* ``predicate.timeVerified``: string (`Timestamp`_)
The timestamp of when the verification happened.

* ``predicate.resourceUri``: string (`ResourceURI`_)
URI identifying the resource associated with the software component being verified.

*Note: In the current version of Macaron, the value of this field is similar to the* `PackageURL`_ *identifying the only subject software component of the VSA.*

* ``policy``: object
Details about the policy that the subject software component was verified against.

* ``policy.content``: string
The Souffle datalog policy used for verification, in plain text.

* ``verificationResult``: string, either ``"PASSED"`` or ``"FAILED"``
The verification result. The result of ``"PASSED"`` means the subject software component conforms to the policy.

* ``verificationResult``: array (`SlsaResult`_), required
Indicates the highest level of each SLSA track verified for the software component (and not its dependencies), or ``"FAILED"`` if policy verification failed.

*Note: For the current version of Macaron, this is left empty.*


-------
Example
-------


The following is an example payload (Statement layer) of a Macaron VSA generated from verification on the `slsa-verifier <https://github.com/slsa-framework/slsa-verifier>`_ repository.

.. code-block:: json
{
"_type": "https://in-toto.io/Statement/v1",
"subject": [
{
"uri": "pkg:github.com/slsa-framework/slsa-verifier@7e1e47d7d793930ab0082c15c2b971fdb53a3c95"
}
],
"predicateType": "https://slsa.dev/verification_summary/v1",
"predicate": {
"verifier": {
"id": "https://github.com/oracle/macaron",
"version": {
"macaron": "0.6.0"
}
},
"timeVerified": "2024-01-04T11:13:03.496399Z",
"resourceUri": "pkg:github.com/slsa-framework/slsa-verifier@7e1e47d7d793930ab0082c15c2b971fdb53a3c95",
"policy": {
"content": "#include \"prelude.dl\"\n\nPolicy(\"slsa_verifier_policy\", component_id, \"Policy for SLSA Verifier\") :-\n check_passed(component_id, \"mcn_build_as_code_1\"),\n check_passed(component_id, \"mcn_provenance_level_three_1\"),\n check_passed(component_id, \"mcn_provenance_available_1\").\n\napply_policy_to(\"slsa_verifier_policy\", component_id) :-\n is_repo(\n _, // repo_id\n \"github.com/slsa-framework/slsa-verifier\",\n component_id\n ).\n"
},
"verificationResult": "PASSED",
"verifiedLevels": []
}
}
This VSA communicates that the subject software component ``"pkg:github.com/slsa-framework/slsa-verifier@7e1e47d7d793930ab0082c15c2b971fdb53a3c95"`` passed the following policy in the ``policy.content`` field:

.. code-block:: prolog
#include "prelude.dl"
Policy("slsa_verifier_policy", component_id, "Policy for SLSA Verifier") :-
check_passed(component_id, "mcn_build_as_code_1"),
check_passed(component_id, "mcn_provenance_level_three_1"),
check_passed(component_id, "mcn_provenance_available_1").
apply_policy_to("slsa_verifier_policy", component_id) :-
is_repo(
_, // repo_id
"github.com/slsa-framework/slsa-verifier",
component_id
).
This policy enforces the subject software component to pass 3 Macaron checks:

* ``mcn_build_as_code_1``
* ``mcn_provenance_level_three_1``
* ``mcn_provenance_available_1``

For more details on using the Macaron VSA generation feature and inspecting the resulting VSA, please refer to the :ref:`Output files of macaron verify-policy section <output_files_macaron_verify_policy>`.
31 changes: 24 additions & 7 deletions scripts/dev_scripts/integration_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ HOMEDIR=$2
RESOURCES=$WORKSPACE/src/macaron/resources
COMPARE_DEPS=$WORKSPACE/tests/dependency_analyzer/compare_dependencies.py
COMPARE_JSON_OUT=$WORKSPACE/tests/e2e/compare_e2e_result.py
COMPARE_POLICIES=$WORKSPACE/tests/policy_engine/compare_policy_reports.py
COMPARE_VSA=$WORKSPACE/tests/vsa/compare_vsa.py
TEST_REPO_FINDER=$WORKSPACE/tests/e2e/repo_finder/repo_finder.py
TEST_COMMIT_FINDER=$WORKSPACE/tests/e2e/repo_finder/commit_finder.py
RUN_MACARON="python -m macaron -o $WORKSPACE/output"
Expand All @@ -19,19 +21,32 @@ UPDATE=0
if [ $# -eq 3 ] && [ "$3" == "--update" ] ; then
echo "Updating the expected results to match those currently produced by Macaron."
UPDATE=1
COMPARE_VSA="$COMPARE_VSA --update"
fi

function check_or_update_expected_output() {
if [ $UPDATE -eq 1 ] ; then
# Perform update of expected results by copying over produced output files.
# The copy only takes place if sufficient arguments are present.
# This function assumes arguments #2 and #3 are files: <actual_result>, <expected_result>.
# Perform update of expected results.
# The update only takes place if sufficient arguments are present.
# This function assumes:
# - argument #1 is the path to the compare script.
# - arguments #2 and #3 are files: <actual_result>, <expected_result>.
if [ $# -eq 3 ] ; then
echo "Copying $2 to $3"
cp "$2" "$3"
compare_script_name=$(basename "$1")
case "$compare_script_name" in
# For scripts having an `--update` flag, use it.
compare_vsa.py)
python "$1" --update "$2" "$3"
;;
# For the other scripts, copy over the produced output files.
*)
echo "Copying $2 to $3"
cp "$2" "$3"
;;
esac
else
# Calls with insufficient arguments are ignored to avoid some needless computation during updates.
echo "Ignoring $@"
echo "Ignoring" "$@"
fi
else
# Perform normal operation.
Expand Down Expand Up @@ -627,14 +642,16 @@ echo -e "\n---------------------------------------------------------------------
echo "Run policy CLI with slsa-verifier results."
echo -e "----------------------------------------------------------------------------------\n"
RUN_POLICY="macaron verify-policy"
COMPARE_POLICIES=$WORKSPACE/tests/policy_engine/compare_policy_reports.py
POLICY_FILE=$WORKSPACE/tests/policy_engine/resources/policies/valid/slsa-verifier.dl
POLICY_RESULT=$WORKSPACE/output/policy_report.json
POLICY_EXPECTED=$WORKSPACE/tests/policy_engine/expected_results/policy_report.json
VSA_RESULT=$WORKSPACE/output/vsa.intoto.jsonl
VSA_PAYLOAD_EXPECTED=$WORKSPACE/tests/vsa/integration/github_slsa-framework_slsa-verifier/vsa_payload.json

# Run policy engine on the database and compare results.
$RUN_POLICY -f $POLICY_FILE -d "$WORKSPACE/output/macaron.db" || log_fail
check_or_update_expected_output $COMPARE_POLICIES $POLICY_RESULT $POLICY_EXPECTED || log_fail
check_or_update_expected_output "$COMPARE_VSA" "$VSA_RESULT" "$VSA_PAYLOAD_EXPECTED" || log_fail

# Testing the Repo Finder's remote calls.
# This requires the 'packageurl' Python module
Expand Down
Loading

0 comments on commit 10b168b

Please sign in to comment.