CURA 10831 #1

jellespijker · 2023-11-11T07:27:18Z

General changes

Developer Conan config

UltiMaker developers working on Cura need to install the conan-config from the dev branch in the https://github.com/Ultimaker/conan-config/tree/dev

conan config install https://github.com/Ultimaker/conan-config.git -a "-b dev"

Pipeline caching over workflows

Ported a lot of workflows to this repo. All of the reusable workflows have pipeline chacing enabled for Conan downloads and Conan data folders. The caching key have a default fallback key. This ensures that the cache is updated with the latest changes on the Artifactory remote server but reuses the previous stored downloads and data, greatly reducing our download bandwidth and flexible costs.

The runner conan-config branches have been updated to ensure that the download cache will be in a known location, see:

Downloading runner pip requirements and helper scripts

Most of the workflows now have a curl or wget command downloading the pip requirements or OS specific helper scripts from a single source of truth. No longer needing to change the requirements.txt of 16 repositories because SIP has a bug.

No longer tagging to create a release package

Creating a Conan package release is now done in this repository. See the workflow conan-package-release.yml You can specify a Repository and specify the git reference: main, 5.7 or a specific git sha. It will check it out and build a conan-package from that commit.

This can be done in as a user action manually (see UI below)

Or in the future automate it, for instance when we create a Release in Draft mode in the Cura repository. We can then change the conandata.yml pinning the versions and dispatching this conan-package-release.yml workflow for the actual repositories. Wait till the workflow are ready, build the Cura release with the newly created conan packages and upload the binaries to the GH draft release.

Conan version and broadcast data

The previous workflow conan-recipe-version.yml was buggy and finickle and determined the version based on the git history. This required su to checkout the whole git repo and have complex traversal logic to get the version or bump the version up. Average running time was ~40s in Cura repo, this was run for each push in the workflows Unit test.

The current solution gets the base version from conandata.yml which should be present in each repo next to conanfile.py A conandata.yml is automatically accessible as a dict self.conan_data in the ConanFile class. Since this is a yaml file it can also be easily set and read in Python.

When getting the broadcast data we now do a spare checkout on just the conandata.yml obtain the version and if it is a beta version is runs on a release branch. If the GitHub inputs flags ity as a release it will be a full blown release tag. No more tag, delete, tag, delete etc.

The version can be manually set by a developer by changing them in the conandata.yml, or automatically when we create a feature freeze workflow. This is yet to be added. This is also the reason why our own Conan requirements have been moved out of the conanfile.py and into the conandata.yml

Benefits:

Quick workflow from ~40s to ~9s
Allow for release automation with workflows

Breaking changes:

Tagging doesn't create a release anymore
conandata.yml needs to be present at each repo with a version
When a PR is made against a release branch it will now use the correct version previously this resulted in minor version bump for the conan-package
No longer a differentiation for PR in the channel, this wasn't used anyhow

Conan export package

Has been moved to this repo, it now uses caches between the pipelines to ensure persistence of Conan download cache and Conan data directory. It should always fall back to the latest created cache. For this to work you would make it run specific and than match on the key before the run uuid.

Conan package create

Same mechanism for pipeline caching as ⬆️ it even reuses the same cache although it is a different workflow

Split up the workflow in to OS specific workflows, such that we have less complexity on the steps

Moved the system specific installation steps to downloaded scripts from this repo, such that every other workflow which might need them will use the same deps. No longer a need to change them throughout different repos.

Unit-tests

Now all use a seperate post workflow to ensure that forks can't execute malicious code.

benchmarks

Check against actors before executing code with write access (no forks or bots)

Related PR's

TODO

The todo's will probably be resolved once the future work is also merged to main

This update includes a new Python script, `get_conan_broadcast_data.py`, to fetch and manage Conan broadcast data. It also includes a new GitHub workflow, `conan-recipe-version.yml`, to trigger this script and capture the output. These additions will enhance automation in the project and streamline the management of Conan recipe versions. Contribute to CURA-10831

Contribute to CURA-10831

This commit introduces a new script, remove_conan_artifacts.py, which is designed to remove Conan binaries based on a given query and targeted remote. The script uses BeautifulSoup to parse HTML tables generated by the Conan search command, identifying packages to remove. It then removes each identified binary package, leaving the recipe intact. Contribute to CURA-10831

Contribute to CURA-10831

Simplified the get_conan_broadcast_data method in runner_scripts to use the information in conandata.yml file. This reduced the complexity of the method significantly and also the number of arguments needed. In addition, it also eliminated the need for gitpython in the requirements file. Contribute to CURA-10831

Contribute to CURA-10831

We will forget to unflag it in the conandata.yml and then every commit afterwards is an actual release Contribute to CURA-10831

Contribute to CURA-10831

introduced a new workflow to export Conan recipes to a server with different configurations including the option to export binaries & extra Conan args. The workflow utilizes several environment settings and handles workflow continuity in case of process termination. Added an Ubuntu setup shell script which configures the system dependencies required for the package. Contribute to CURA-10831

Not really a secret, but this allows us to switch all workflows to a differenr url at once Contribute to CURA-10831

Contribute to CURA-10831

Added two new optional inputs to the Conan recipe export workflow: 'conan_export_binaries' and 'update_system_requirements', both with default values. Adjusted the conditions for system requirements installation and package export based on these new inputs. Contribute to CURA-10831

Contribute to CURA-10831

The commit introduces three separate workflows for creating Conan packages on Linux, MacOS, and Windows. These workflows include the steps to checkout the appropriate branch, install Python requirements, configure Conan profiles, and export and upload the package. Scripts for MacOS system requirements setup have also been added. Lastly, unnecessary flags for exporting binaries in the Conan recipe export workflow have been removed. Contributes to CURA-10831

Contributes to CURA-10831

This is already done when exporting Contributes to CURA-10831

Contributes to CURA-10831

6.8 has a bug Contributes to CURA-10831

This ensures that Cura and Uranium are completly available Contributes to CURA-10831

Contributes to CURA-10831

This reverts commit 408369e.

This reverts commit f471956.

Contributes to CURA-10831

The paths for uploading debug symbols in cura-installer workflows for macOS, Windows, and Linux have been updated. This change includes specifying the subdirectory UltiMaker-Cura within the dist directory, where the executable and library files for the upload command are located. Contributes to CURA-10831

.github/workflows/conan-package-release.yml

.github/workflows/benchmark.yml

.github/workflows/conan-package-create-macos.yml

casperlamboo · 2023-12-07T09:10:00Z

.github/workflows/conan-package-create-macos.yml

+      #  FIXME: point to `main` once merged
+      - name: Install Macos system requirements for building
+        run: |
+          mkdir runner_scripts
+          wget https://raw.githubusercontent.com/Ultimaker/cura-workflows/CURA-10831/runner_scripts/macos_setup.sh -O runner_scripts/macos_setup.sh


Suggested change

# FIXME: point to `main` once merged

- name: Install Macos system requirements for building

run: |

mkdir runner_scripts

wget https://raw.githubusercontent.com/Ultimaker/cura-workflows/CURA-10831/runner_scripts/macos_setup.sh -O runner_scripts/macos_setup.sh

# FIXME: point to `main` once merged

- name: Install Macos system requirements for building

run: |

mkdir runner_scripts

wget https://raw.githubusercontent.com/Ultimaker/cura-workflows/main/runner_scripts/macos_setup.sh -O runner_scripts/macos_setup.sh

casperlamboo · 2023-12-07T13:25:05Z

.github/workflows/conan-package-create-linux.yml

+      #  FIXME: point to `main` once merged
+      - name: Install Linux system requirements for building
+        run: |
+          mkdir runner_scripts
+          wget https://raw.githubusercontent.com/Ultimaker/cura-workflows/CURA-10831/runner_scripts/ubuntu_setup.sh -O runner_scripts/ubuntu_setup.sh


Suggested change

# FIXME: point to `main` once merged

- name: Install Linux system requirements for building

run: |

mkdir runner_scripts

wget https://raw.githubusercontent.com/Ultimaker/cura-workflows/CURA-10831/runner_scripts/ubuntu_setup.sh -O runner_scripts/ubuntu_setup.sh

- name: Install Linux system requirements for building

run: |

mkdir runner_scripts

wget https://raw.githubusercontent.com/Ultimaker/cura-workflows/main/runner_scripts/ubuntu_setup.sh -O runner_scripts/ubuntu_setup.sh

.github/workflows/conan-package-create-linux.yml

.github/workflows/conan-package-create-macos.yml

runner_scripts/cura_installer_filename.py

Co-authored-by: Casper Lamboo <casperlamboo@gmail.com>

runner_scripts/cura_installer_filename.py

Co-authored-by: Casper Lamboo <casperlamboo@gmail.com>

jellespijker added 22 commits November 10, 2023 14:25

use wget to download runner_scripts

b25335f

Contribute to CURA-10831

Ensure every arg has a value

46e6ccb

Contribute to CURA-10831

Ensure that repo's without tags start with version 0.0.1-alpha

b83bc79

Contribute to CURA-10831

Add presentation about the buildsystem

ae7695d

Contribute to CURA-10831

Only checkout conandata.yml

1cf3c9d

Contribute to CURA-10831

No external deps

a77ee56

Contribute to CURA-10831

No pip install needed

53f7c46

Contribute to CURA-10831

add yaml as dependency

5ea29d7

Contribute to CURA-10831

Don't use release key, value from conandata.yml

c5a2643

We will forget to unflag it in the conandata.yml and then every commit afterwards is an actual release Contribute to CURA-10831

Don't require release

1946b9a

Contribute to CURA-10831

Use CONAN_PRIVATE_ARTIFACTORY_URL

1e1ef03

Not really a secret, but this allows us to switch all workflows to a differenr url at once Contribute to CURA-10831

Allow write to the contents

5f3c072

Contribute to CURA-10831

Use the correct pip req file

7e7928f

Contribute to CURA-10831

only try to upload package to cura-private when internal

83d7114

Contribute to CURA-10831

Only create stable packages on release branch

bd455f4

Contribute to CURA-10831

Don't decouple update system and export binaries

a221680

Contribute to CURA-10831

Fix caching of Conan data and downloads

7927882

Contribute to CURA-10831

jellespijker force-pushed the CURA-10831 branch from f26308f to 7927882 Compare November 14, 2023 15:07

jellespijker added 5 commits November 14, 2023 16:35

Use curl on Windows

20089f9

Contributes to CURA-10831

Remove creating the alias (latest)

109ca5c

This is already done when exporting Contributes to CURA-10831

flip slashes

44c1317

Contributes to CURA-10831

fix empty reqs

37d8dee

Contributes to CURA-10831

jellespijker requested review from rajaroraji and casperlamboo November 16, 2023 06:43

jellespijker added 6 commits December 6, 2023 08:55

optional latest alias

e64726f

Contributes to CURA-10831

Remove unused env var CONAN_PRIVATE_ARTIFACTORY_URL

60b2842

Contributes to CURA-10831

Allow setting user/channel

7d9818a

Contributes to CURA-10831

Fix missing arg channel

d02bd79

Contributes to CURA-10831

fix indent

d24e393

Contributes to CURA-10831

Shorten descriptions

6b1c47f

Contributes to CURA-10831

jellespijker added 9 commits December 6, 2023 11:49

Use sip<=6.7.12

a0e7288

6.8 has a bug Contributes to CURA-10831

Also build the binaries after an export

f471956

This ensures that Cura and Uranium are completly available Contributes to CURA-10831

Fix to many inputs

408369e

Contributes to CURA-10831

allow runner_scripts directory to be present

2f02908

Contributes to CURA-10831

Revert "Fix to many inputs"

6c45a7b

This reverts commit 408369e.

Revert "Also build the binaries after an export"

52f8fa4

This reverts commit f471956.

remove private parts and upload later

630d36c

Contributes to CURA-10831

Upload binaries to Sentry

847492e

Contributes to CURA-10831

casperlamboo approved these changes Dec 8, 2023

View reviewed changes

Apply suggestions from code review

45774c3

Co-authored-by: Casper Lamboo <casperlamboo@gmail.com>

jellespijker requested a review from casperlamboo December 8, 2023 13:00

casperlamboo reviewed Dec 8, 2023

View reviewed changes

runner_scripts/cura_installer_filename.py Outdated Show resolved Hide resolved

Update runner_scripts/cura_installer_filename.py

ab43f57

Co-authored-by: Casper Lamboo <casperlamboo@gmail.com>

casperlamboo approved these changes Dec 8, 2023

View reviewed changes

casperlamboo merged commit 7388d59 into main Dec 8, 2023

casperlamboo deleted the CURA-10831 branch December 8, 2023 13:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CURA 10831 #1

CURA 10831 #1

jellespijker commented Nov 11, 2023 •

edited

Loading

casperlamboo Dec 7, 2023

casperlamboo Dec 7, 2023

CURA 10831 #1

CURA 10831 #1

Conversation

jellespijker commented Nov 11, 2023 • edited Loading

General changes

Developer Conan config

Pipeline caching over workflows

Downloading runner pip requirements and helper scripts

No longer tagging to create a release package

Conan version and broadcast data

Benefits:

Breaking changes:

Conan export package

Conan package create

Unit-tests

benchmarks

Related PR's

TODO

casperlamboo Dec 7, 2023

Choose a reason for hiding this comment

casperlamboo Dec 7, 2023

Choose a reason for hiding this comment

jellespijker commented Nov 11, 2023 •

edited

Loading