docs: add commit finder tutorial using Arrow library

docs/source/pages/tutorials/index.rst

@@ -276,22 +276,17 @@ You can use this policy in your GitHub Actions to prevent a deployment or fail a
 development. Alternatively, you can treat the result as a warning and manually investigate the
 dependencies to make sure they are secure and can be trusted.
 
------------------------------------------------------------------
-More Accurate Analysis via Mappings of Artifacts to Exact Commits
------------------------------------------------------------------
+---------------------------------------------------------
+Analyzing and comparing different versions of an artifact
+---------------------------------------------------------
 
-This tutorial demonstrates how Macaron can be used to determine the differences between one or more states of the single open source repository that produced one or more related artifacts. In this way, we show how a developer can be potentially mislead by supply chain security information that has been created for the current state of an artifact's source repository, rather than the version of the artifact they are actually using.
+This tutorial demonstrates how Macaron can be used to determine the differences between one or more states of the single open source repository that produced one or more related artifacts. In this way, we show how a developer can be potentially misled by supply chain security information that has been created for the current state of an artifact's source repository, rather than the version of the artifact they are actually using.
 
-The problem of mapping artifacts to the source code that built them is a challenging one, as most artifacts, even open source ones, do not provide a direct URL to their related repository. Services exist to make up for this lack, including Google's `Open Source Insights <https://deps.dev/>`_ tool that is in use by Macaron itself for this exact reason. However, without taking further steps, analysis of these repositories will reflect only the current state at the time of execution. One example of this is OpenSSF Scorecard, an automated tool that performs a number of a software security checks on a given project. These projects are typically provided in the form of a repository's public URL, which will be examined at its current state.
+The problem of mapping artifacts to the source code that built them is a challenging one, as most artifacts, even open source ones, do not provide a direct URL to their related repository. Services exist to make up for this lack, including Google's `Open Source Insights <https://deps.dev/>`_ tool that is in use by Macaron itself for this exact reason. However, without taking further steps, analysis of these repositories will reflect only the current state at the time of execution. One example of this is OpenSSF Scorecard, an automated tool that performs a number of software security checks on a given project. These projects are typically provided in the form of a repository's public URL, which will be examined at its current state.
 
-To facilitate greater accuracy during analysis, Macaron allows a repository to be examined at more than just the current state through use of its Commit Finder feature. This feature performs a best effort attempt to map a given artifact to the exact commit that was used to create it by comparing.repository tags with artifact versions. Therefore, it has a requirement that any repository to be analyzed makes use of tags in a way that closely corresponds to the produced artifact's version numbers.
+To facilitate greater accuracy during analysis, Macaron allows analyzing an artifact and its corresponding repository state by using the Commit Finder feature. This feature performs a best effort attempt to map a given artifact to the exact commit that was used to create it by comparing repository tags with artifact versions. Therefore, it has a requirement that any repository to be analyzed makes use of tags in a way that closely corresponds to the produced artifact's version numbers. Pre-existing techniques that achieve the same as this feature do exist, namely `SLSA`_ or `Witness`_ provenances, which contain the commit that was used to build the artifact they represent. However, currently the adoption rate in the open-source community is quite low, therefore limiting its value for this task.
 
-For this tutorial, we have chosen the Python library, `Arrow <https://github.com/arrow-py/arrow>`_. Arrow is designed to greatly improve the developer experience whenever times or dates are involved.
-
-Arrow has two dependencies as of version 1.3.0:
-
-* ``python-dateutil``
-* ``types-python-dateutil``
+For this tutorial, we have chosen the Python library, `Arrow <https://github.com/arrow-py/arrow>`_. Arrow is a popular library designed to improve the developer experience for manipulating dates and times.
 
 ************
 Installation
@@ -325,7 +320,7 @@ To perform an analysis on Arrow, Macaron can be run with the following command:
 
 .. code-block:: shell
 
-    ./run_macaron.sh analyze -rp https://github.com/arrow-py/arrow
+    ./run_macaron.sh analyze -rp https://github.com/arrow-py/arrow --skip-deps
 
 However, this will return results based only on the current state of the repository, which as described above, is not what we want to achieve in this tutorial. To perform analyses on other repository states, we need to provide Macaron with the target artifact versions in the form of `PURLs <https://github.com/package-url/purl-spec>`_, or Package URLs, which is a convenient way to encodify packages from different ecosystems into the same format.
 
@@ -344,52 +339,95 @@ We will start by running the analysis on the latest version, ``1.3.0``, with the
 
     ./run_macaron.sh analyze -purl pkg:pypi/arrow@1.3.0
 
-The analysis involves Macaron downloading the contents of the target repository to the configured, or default, ``output`` folder. (See :ref:`Output Files Guide <output_files_guide>`). Once the analysis is complete, Macaron will produce a report in the form of a HTML file.
+The analysis involves Macaron downloading the contents of the target repository to the configured, or default, ``output`` folder. Results from the analysis, including checks, are stored in the database found at ``output/macaron.db``  (See :ref:`Output Files Guide <output_files_guide>`). Once the analysis is complete, Macaron will also produce a report in the form of a HTML file.
 
 .. code-block:: shell
 
   open output/reports/pypi/arrow/arrow.html
 
-.. note:: When analyzing multiple versions of the same artifact, keep in mind that Macaron will override the output report in subsequent runs.
+.. note:: When analyzing multiple versions of the same software component, keep in mind that Macaron will override the output HTML and JSON reports in subsequent runs, but still keep results for each version in the database.
+
+.. _fig_arrow_1.3.0_top:
+
+.. figure:: ../../_static/images/tutorial_arrow_1.3.0_report_top.png
+   :alt: HTML report for ``arrow 1.3.0``, summary
+   :align: center
 
 .. _fig_arrow_1.3.0:
 
 .. figure:: ../../_static/images/tutorial_arrow_1.3.0_report.png
-   :alt: HTML report for ``arrow 1.3.0``
+   :alt: HTML report for ``arrow 1.3.0``, checks
    :align: center
 
 The image above shows the results of the checks for the `Arrow <https://github.com/arrow-py/arrow>`_ repository at the commit where version ``1.3.0`` was produced.
 In summary, the repository is:
 
-* a Git repository (``mcn_version_control_system_1``)
-* using a build script (``mcn_build_script_1``)
-* using GitHub Actions to build using ``pip`` (``mcn_build_service_1``)
-
-Before running the analysis again on the other version of Arrow, we should backup the report for future reference.
-
-.. code-block:: shell
-
-    cp output/reports/pypi/arrow/arrow.html output/reports/pypi/arrow/arrow_1.3.0.html
+* A commit at a Git repository that corresponds to the artifact (``mcn_version_control_system_1``)
+* The build tool ``pip`` used in the build scripts (``mcn_build_script_1``)
+* GitHub Actions workflow to build the package (``mcn_build_service_1``)
+* GitHub Actions workflow to deploy and publish the package (``mcn_build_as_code_1``)
 
-Now we are free to run the next analysis, and then open the new report.
+Now we should run the next analysis, and then open the new report.
 
 .. code-block:: shell
 
-    ./run_macaron.sh analyze -purl pkg:pypi/arrow@0.15.0
+    ./run_macaron.sh analyze -purl pkg:pypi/arrow@0.15.0 --skip-deps
     open output/reports/pypi/arrow/arrow.html
 
-.. note:: When analyzing multiple versions of the same artifact, keep in mind that Macaron will override the output report in subsequent runs.
+.. _fig_arrow_0.15.0_top:
+
+.. figure:: ../../_static/images/tutorial_arrow_0.15.0_report_top.png
+   :alt: HTML report for ``arrow 0.15.0``, summary
+   :align: center
 
 .. _fig_arrow_0.15.0:
 
 .. figure:: ../../_static/images/tutorial_arrow_0.15.0_report.png
-   :alt: HTML report for ``arrow 0.15.0``
+   :alt: HTML report for ``arrow 0.15.0``, checks
    :align: center
 
-In the second report for Arrow, we can see that Macaron has returned slightly different results. The previous success of (``mcn_build_script_1``) has now changed to a failure, as the earlier version of the library does not include a GitHub action for building with ``pip``. In this way Macaron has demonstrated the usefulness of being able to analyze a repository at multiple stages, thereby allowing for a more accurate analysis when investigating artifacts that are, or use, outdated libraries.
+In the second report for Arrow, we can see that Macaron has returned slightly different results. Starting with the ``Target Information`` section we can see that the repository for this older artifact is not the same as the current one: ``https://github.com/crsmithdev/arrow`` instead of ``https://github.com/arrow-py/arrow``. In the checks section, we can see that two of the four checks that passed for the previous version, did not pass for this earlier version. Checks ``mcn_build_service_1`` and ``mcn_build_as_code_1`` failed, indicating that the older artifact did not have a GitHub Actions workflow setup to build or publish the package. In this way Macaron has demonstrated the usefulness of being able to analyze a repository at multiple stages, thereby allowing for a more accurate analysis when investigating artifacts that are, or use, outdated libraries.
+
+*****************************
+Run ``verify-policy`` command
+*****************************
+
+Another feature of Macaron is policy verification. This allows Macaron to report on whether a artifact meets the security requirements specified by the user. Policies are written using `Soufflé Datalog <https://souffle-lang.github.io/index.html>`_ , a language similar to SQL. Results collected by the ``analyze`` command can be checked via declarative queries in the created policy, which Macaron can then automatically check.
+
+The security requirement chosen for this tutorial reflects the difference between the two versions in the previous section. That is, we want to ensure that the artifact has a valid build service. If we refer back to :ref:`step <fig_arrow_0.15.0>` and :ref:`step <fig_arrow_1.3.0>`, we can see that the relevant check ID of the difference between the two versions is ``mcn_build_service_1``. To include this in a policy we create the following:
+
+.. code-block:: c++
+
+    #include "prelude.dl"
+
+    Policy("has-build-service", component_id, "Require build with build scripts.") :-
+        check_passed(component_id, "mcn_build_service_1").
+
+    apply_policy_to("has-build-service", component_id) :-
+        is_component(component_id, purl),
+        match("pkg:pypi/arrow.*", purl).
+
+The second part of the above policy, ``apply_policy_to``, applies the policy to components found within Macaron's database based on the conditions within it. In this case, any component whose PURL begins with ``pkg:pypi/arrow``, thanks to the use of regular expression. This will capture both versions of the Arrow library used in the previous section. To use the completed policy, we save it to an easily accessible location, such as the directory Macaron is in, with a name such as ``has-build-service.dl``. With the policy file created and saved, we can have Macaron run is as follows:
+
+.. code-block:: shell
+
+  ./run_macaron.sh verify-policy --database ./output/macaron.db --file ./has-build-service.dl
+
+At the end of the output of this command, Macaron will display the following:
+
+.. code-block:: javascript
+
+    component_satisfies_policy
+        ['176', 'pkg:pypi/arrow@1.3.0', 'has-build-service']
+    component_violates_policy
+        ['177', 'pkg:pypi/arrow@0.15.0', 'has-build-service']
+    failed_policies
+        ['has-build-service']
+
+This confirms the findings of the previous section, showing that the earlier version of the Arrow library does not meet our expectations in that it is lacking a discoverable build service, while the more recent version is just fine and passes.
 
 ***********
 Future Work
 ***********
 
-Mapping artifacts to commits within repositories is a challenging endeavour. Macron's Commit Finder feature relies on repositories having and using version tags in a sensible way (a tag is considered sensible if it closely matches the version it represents). An alternative, or complimentary, approach would be to make use of the information found within provenance files, where information such as the commit hash used to create the artifact can potentially be found. Additionally, it should be noted that the Commit Finder feature was modelled on the intentions of developers (in terms of tag usage) within a large quantity of Java projects. This should translate well to other languages, as tag formatting is generally language agnostic, there may be some improvements to be made by further testing on a large number of non-Java projects.
+Mapping artifact to commits within repositories is a challenging endeavour. Macron's Commit Finder feature relies on repositories having and using version tags in a sensible way (a tag is considered sensible if it closely matches the version it represents). An alternative, or complimentary, approach would be to make use of the information found within provenance files, where information such as the commit hash used to create the artifact can potentially be found. Additionally, it should be noted that the Commit Finder feature was modelled on the intentions of developers (in terms of tag usage) within a large quantity of Java projects. This should translate well to other languages, as tag formatting is generally language agnostic, there may be some improvements to be made by further testing on a large number of non-Java projects.