feat: generate Verification Summary Attestation

docs/source/index.rst

@@ -112,5 +112,6 @@ intermediate representations as abstractions. Using such abstractions, Macaron i
    pages/cli_usage/index
    pages/tutorials/index
    pages/output_files
+   pages/vsa
    pages/supported_technologies/index
    pages/developers_guide/index

docs/source/pages/developers_guide/apidoc/macaron.rst

@@ -21,6 +21,7 @@ Subpackages
    macaron.policy_engine
    macaron.repo_finder
    macaron.slsa_analyzer
+   macaron.vsa
 
 Submodules
 ----------

docs/source/pages/developers_guide/apidoc/macaron.vsa.rst

@@ -0,0 +1,18 @@
+macaron.vsa package
+===================
+
+.. automodule:: macaron.vsa
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Submodules
+----------
+
+macaron.vsa.vsa module
+----------------------
+
+.. automodule:: macaron.vsa.vsa
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/pages/output_files.rst

@@ -9,9 +9,13 @@ Output Files Guide
 
 .. note:: Please see :ref:`pages/cli_usage/index:common options` for the instructions on how to set the output directory of Macaron.
 
--------------------
+--------------------------------
+Output files of macaron analyze
+--------------------------------
+
+^^^^^^^^^^^^^^^^^^^
 Top level structure
--------------------
+^^^^^^^^^^^^^^^^^^^
 
 .. code-block::
 
@@ -25,9 +29,9 @@ Top level structure
         ├── macaron.db
         └── sbom_debug.json
 
--------
+^^^^^^^
 Reports
--------
+^^^^^^^
 
 The report files of Macaron (from using the :ref:`analyze command <analyze-command-cli>`) are generated into the ``reports`` directory.
 
@@ -102,9 +106,9 @@ For example, for `<https://github.com/micronaut-projects/micronaut-core>`_ the r
                     ├── dependency_2.json
                     └── ...
 
--------------------
+^^^^^^^^^^^^^^^^^^^
 Cloned repositories
--------------------
+^^^^^^^^^^^^^^^^^^^
 
 The ``git_repos`` directory is used to clone repositories into during the analysis. Each remote repository is cloned to a unique path
 within ``git_repos`` following the same strategy as `Unique result path`_.
@@ -129,37 +133,64 @@ to the directory:
 
 .. note:: Please see :ref:`pages/using:analyzing a locally cloned repository` to know how to set the directory for analyzing local repositories.
 
+-------------------------------------
+Output files of macaron verify-policy
+-------------------------------------
+
+As part of the ``macaron verify-policy`` command, Macaron generates a :ref:`Verification Summary Attestation<vsa>` (VSA) with the following strategy:
+
+* If the Datalog policy applies to a unique software component identified by a unique PURL, a VSA is generated based on the latest analysis results for that specific software component in the Macaron database.
+* Otherwise, if the Datalog policy applies to multiple software components identified by multiple different PURLs, no VSA will be generated.
+
+The VSA file will be generated into ``output/vsa.intoto.jsonl`` by default.
+
+.. code-block::
+
+    output/
+    └── vsa.intoto.jsonl
+
+
+Users can manually inspect the payload of the VSA generated by Macaron with the following command:
+
+.. code-block:: bash
+
+    cat output/vsa.intoto.jsonl | jq -r '.payload' | base64 -d | jq
+
+
+For more details about the Macaron-generated VSAs, please refer to the :ref:`Verification Summary Attestation page<vsa>`.
+
+
 ------
 Others
 ------
 
-''''''''''
+^^^^^^^^^^
 macaron.db
-''''''''''
+^^^^^^^^^^
 
 The file is the SQLite database used by Macaron for storing analysis results.
 
-'''''''''
+^^^^^^^^^
 debug.log
-'''''''''
+^^^^^^^^^
 
 This file stores the log messages from the latest run of Macaron.
 
-'''''''''
+^^^^^^^^^
 build_log
-'''''''''
+^^^^^^^^^
 
 This is the directory for storing the log from running external components such as `CycloneDx SBOM Maven plugin <https://github.com/CycloneDX/cyclonedx-maven-plugin>`_, `CycloneDx SBOM Gradle plugin <https://github.com/CycloneDX/cyclonedx-gradle-plugin>`_ or the `slsa-verifier <https://github.com/slsa-framework/slsa-verifier>`_.
 
-'''''''''''''''
+^^^^^^^^^^^^^^^
 sbom_debug.json
-'''''''''''''''
+^^^^^^^^^^^^^^^
 
 This file contain the debug information for running the SBOM generator to obtain dependencies of a repository.
 
-'''''''''''''''
+^^^^^^^^^^^^^^^
 .m2 and .gradle
-'''''''''''''''
+^^^^^^^^^^^^^^^
 
 These two directories cache the content of ``~/.m2`` and ``~/.gradle`` in the Docker container between different runs (which are
 mainly updated by the CycloneDX SBOM plugins).

docs/source/pages/vsa.rst

@@ -0,0 +1,170 @@
+=================================
+Verification Summary Attestations
+=================================
+
+.. _vsa:
+
+Macaron generates Verification Summary Attestations (VSAs) as part of its verification to communicate the fact that "some software component has been verified against a policy".
+
+The concept of VSA in Macaron largely follows the concept of VSA in `SLSA <https://slsa.dev/spec/v1.0/verification_summary>`_ and `in-toto <https://github.com/in-toto/attestation/blob/main/spec/predicates/vsa.md>`_.
+
+
+---------
+Use cases
+---------
+
+The use cases of Macaron VSAs includes, but not limited to:
+
+- **Caching verification results**: It could be expensive or inconvenient to run a full Macaron verification in certain circumstances. A VSA helps with caching and reusing verification results.
+- **Enabling delegated verification**: This allows software consumers to make use of verification results from another party.
+
+
+------
+Schema
+------
+
+.. Type references
+.. _PackageURL: https://github.com/package-url/purl-spec
+.. _Envelope: https://github.com/in-toto/attestation/blob/main/spec/v1/envelope.md
+.. _TypeURI: https://github.com/in-toto/attestation/blob/main/spec/v1/field_types.md#TypeURI
+.. _Timestamp: https://github.com/in-toto/attestation/blob/main/spec/v1/field_types.md#timestamp
+.. _ResourceURI: https://github.com/in-toto/attestation/blob/main/spec/v1/field_types.md#ResourceURI
+.. _ResourceDescriptor: https://github.com/in-toto/attestation/blob/main/spec/v1/resource_descriptor.md
+.. _SlsaResult: https://slsa.dev/spec/v1.0/verification_summary#slsaresult
+
+Following in-toto attestation schema, the outermost layer if a Macaron-generated VSA is a `DSSE envelope <https://github.com/in-toto/attestation/blob/main/spec/v1/envelope.md>`_ containing a base64-encoded ``payload`` of type `in-toto Statement <https://github.com/in-toto/attestation/blob/main/spec/v1/statement.md>`_.
+
+The following is the schema of the Statement layer:
+
+
+.. code-block:: js+jinja
+
+    {
+        "_type": "https://in-toto.io/Statement/v1",
+        "subject": [
+            {
+                "uri": {{ PackageURL of the software component being verified }},
+            }
+        ],
+        "predicateType": "https://slsa.dev/verification_summary/v1",
+        "predicate": {
+            "verifier": {
+                "id": "https://github.com/oracle/macaron",
+                "version": {
+                    "macaron": {{ Macaron version }}
+                }
+            },
+            "timeVerified": {{ The timestamp of when the verification happened }},
+            "resourceUri": {{ PackageURL of the software component being verified }},
+            "policy": {
+                "content": {{ Datalog policy applies to the software component being verified }}
+            },
+            "verificationResult": {{ Either "PASSED" or "FAILED" }},
+            "verifiedLevels": []
+        }
+    }
+
+
+
+* ``_type``: string (`TypeURI`_)
+    Identifier for the schema of the Statement layer. This follows `in-toto v1 Statement layer schema <https://github.com/in-toto/attestation/blob/main/spec/v1/statement.md>`_ and is always ``https://in-toto.io/Statement/v1``.
+
+* ``subject``: array of `ResourceDescriptor`_ objects
+    Subjects of the VSA. Each entry is a software component being verified by Macaron.
+
+    *Note: In the current version of Macaron, this field only contains one single software component, identified by a `PackageURL`_.*
+
+* ``predicateType``: string (`TypeURI`_)
+    Identifier for the type of the Predicate. For Macaron-generated VSAs, this is always ``https://slsa.dev/verification_summary/v1``.
+
+* ``predicate``: object
+    The Predicate of the attestation, providing information about the verification.
+
+* ``predicate.verifier``: object
+    Information about the tool running the verification, which is Macaron.
+
+* ``predicate.verifier.id``: string (`TypeURI`_)
+    The identifier for Macaron.
+
+* ``predicate.timeVerified``: string (`Timestamp`_)
+    The timestamp of when the verification happened.
+
+* ``predicate.resourceUri``: string (`ResourceURI`_)
+    URI identifying the resource associated with the software component being verified.
+
+    *Note: In the current version of Macaron, the value of this field is similar to the `PackageURL`_ identifying the only subject software component of the VSA.*
+
+* ``policy``: object
+    Details about the policy that the subject software component was verified against.
+
+* ``policy.content``: string
+    The Souffle datalog policy used for verification, in plain text.
+
+* ``verificationResult``: string, either ``"PASSED"`` or ``"FAILED"``
+    The verification result. The result of ``"PASSED"`` means the subject software component conforms to the policy.
+
+* ``verificationResult``: array (`SlsaResult`_), required
+    Indicates the highest level of each SLSA track verified for the software component (and not its dependencies), or “FAILED” if policy verification failed.
+
+    *Note: For the current version of Macaron, this is left empty.*
+
+
+-------
+Example
+-------
+
+
+The following is an example payload (Statement layer) of a Macaron VSA generated from verification on the `slsa-verifier <https://github.com/slsa-framework/slsa-verifier>`_ repository.
+
+.. code-block:: json
+
+    {
+        "_type": "https://in-toto.io/Statement/v1",
+        "subject": [
+            {
+                "uri": "pkg:github.com/slsa-framework/slsa-verifier@7e1e47d7d793930ab0082c15c2b971fdb53a3c95"
+            }
+        ],
+        "predicateType": "https://slsa.dev/verification_summary/v1",
+        "predicate": {
+            "verifier": {
+                "id": "https://github.com/oracle/macaron",
+                "version": {
+                    "macaron": "0.6.0"
+                }
+            },
+            "timeVerified": "2024-01-04T11:13:03.496399Z",
+            "resourceUri": "pkg:github.com/slsa-framework/slsa-verifier@7e1e47d7d793930ab0082c15c2b971fdb53a3c95",
+            "policy": {
+                "content": "#include \"prelude.dl\"\n\nPolicy(\"slsa_verifier_policy\", component_id, \"Policy for SLSA Verifier\") :-\n  check_passed(component_id, \"mcn_build_as_code_1\"),\n  check_passed(component_id, \"mcn_provenance_level_three_1\"),\n  check_passed(component_id, \"mcn_provenance_available_1\").\n\napply_policy_to(\"slsa_verifier_policy\", component_id) :-\n  is_repo(\n    _,  // repo_id\n    \"github.com/slsa-framework/slsa-verifier\",\n    component_id\n  ).\n"
+            },
+            "verificationResult": "PASSED",
+            "verifiedLevels": []
+        }
+    }
+
+This VSA communicates that the subject software component ``"pkg:github.com/slsa-framework/slsa-verifier@7e1e47d7d793930ab0082c15c2b971fdb53a3c95"`` passed the following policy in the ``policy.content`` field:
+
+.. code-block:: prolog
+
+    #include "prelude.dl"
+
+    Policy("slsa_verifier_policy", component_id, "Policy for SLSA Verifier") :-
+      check_passed(component_id, "mcn_build_as_code_1"),
+      check_passed(component_id, "mcn_provenance_level_three_1"),
+      check_passed(component_id, "mcn_provenance_available_1").
+
+    apply_policy_to("slsa_verifier_policy", component_id) :-
+      is_repo(
+        _,  // repo_id
+        "github.com/slsa-framework/slsa-verifier",
+        component_id
+      ).
+
+This policy enforces the subject software component to pass 3 Macaron checks:
+
+* ``mcn_build_as_code_1``
+* ``mcn_provenance_level_three_1``
+* ``mcn_provenance_available_1``
+
+For more details on using the Macaron VSA generation feature and inspecting the resulting VSA, please refer to the :ref:`Output File Guide <output_files_guide>`.

scripts/dev_scripts/integration_tests.sh

@@ -9,6 +9,8 @@ HOMEDIR=$2
 RESOURCES=$WORKSPACE/src/macaron/resources
 COMPARE_DEPS=$WORKSPACE/tests/dependency_analyzer/compare_dependencies.py
 COMPARE_JSON_OUT=$WORKSPACE/tests/e2e/compare_e2e_result.py
+COMPARE_POLICIES=$WORKSPACE/tests/policy_engine/compare_policy_reports.py
+COMPARE_VSA=$WORKSPACE/tests/vsa/compare_vsa.py
 TEST_REPO_FINDER=$WORKSPACE/tests/e2e/repo_finder/repo_finder.py
 TEST_COMMIT_FINDER=$WORKSPACE/tests/e2e/repo_finder/commit_finder.py
 RUN_MACARON="python -m macaron -o $WORKSPACE/output"
@@ -19,6 +21,7 @@ UPDATE=0
 if [ $# -eq 3 ] && [ "$3" == "--update" ] ; then
     echo "Updating the expected results to match those currently produced by Macaron."
     UPDATE=1
+    COMPARE_VSA="$COMPARE_VSA --update"
 fi
 
 function check_or_update_expected_output() {
@@ -627,7 +630,6 @@ echo -e "\n---------------------------------------------------------------------
 echo "Run policy CLI with slsa-verifier results."
 echo -e "----------------------------------------------------------------------------------\n"
 RUN_POLICY="macaron verify-policy"
-COMPARE_POLICIES=$WORKSPACE/tests/policy_engine/compare_policy_reports.py
 POLICY_FILE=$WORKSPACE/tests/policy_engine/resources/policies/valid/slsa-verifier.dl
 POLICY_RESULT=$WORKSPACE/output/policy_report.json
 POLICY_EXPECTED=$WORKSPACE/tests/policy_engine/expected_results/policy_report.json
@@ -636,6 +638,32 @@ POLICY_EXPECTED=$WORKSPACE/tests/policy_engine/expected_results/policy_report.js
 $RUN_POLICY -f $POLICY_FILE -d "$WORKSPACE/output/macaron.db" || log_fail
 check_or_update_expected_output $COMPARE_POLICIES $POLICY_RESULT $POLICY_EXPECTED || log_fail
 
+# Testing the VSA generation feature
+# Running Macaron without config files
+echo -e "\n=================================================================================="
+echo "Run integration tests for VSA generation"
+echo -e "==================================================================================\n"
+TEST_CASE_DIR="$WORKSPACE/tests/vsa/integration/github_slsa-framework_slsa-verifier"
+OUTPUT_DIR="$TEST_CASE_DIR/output"
+
+rm -rf "$OUTPUT_DIR"  # Make sure we regenerate a fresh database every time.
+macaron --output "$OUTPUT_DIR" analyze \
+    --repo-path "https://github.com/slsa-framework/slsa-verifier" \
+    --digest 7e1e47d7d793930ab0082c15c2b971fdb53a3c95 \
+    --skip-deps || log_fail
+check_or_update_expected_output "$COMPARE_JSON_OUT" \
+    "$OUTPUT_DIR/reports/github_com/slsa-framework/slsa-verifier/slsa-verifier.json" \
+    "$TEST_CASE_DIR/slsa-verifier.json" || log_fail
+macaron --output "$OUTPUT_DIR" verify-policy \
+    --database "$OUTPUT_DIR/macaron.db" \
+    --file "$TEST_CASE_DIR/policy.dl" || log_fail
+check_or_update_expected_output "$COMPARE_POLICIES" \
+    "$OUTPUT_DIR/policy_report.json" \
+    "$TEST_CASE_DIR/policy_report.json" || log_fail
+python3 "$COMPARE_VSA" \
+    "$OUTPUT_DIR/vsa.intoto.jsonl" \
+    "$TEST_CASE_DIR/vsa.intoto.jsonl" || log_fail
+
 # Testing the Repo Finder's remote calls.
 # This requires the 'packageurl' Python module
 echo -e "\n----------------------------------------------------------------------------------"