diff --git a/_episodes/01-introduction.md b/_episodes/01-introduction.md index 4e4041c..75abfca 100644 --- a/_episodes/01-introduction.md +++ b/_episodes/01-introduction.md @@ -12,7 +12,8 @@ keypoints: - CI/CD is crucial for any reproducibility and testing - Take advantage of automation to reduce your workload --- - + + # What is CI/CD? Continuous Integration (CI) is the concept of literal continuous integration of code changes. That is, every time a contributor (student, colleague, random bystander) provides new changes to your codebase, those changes are tested to make sure they don't "break" anything. Continuous Deployment (CD), similarly, is the literal continuous deployment of code changes. That means that, assuming the CI passes, you'd like to automatically deploy those changes. diff --git a/_episodes/02-enter-sandman.md b/_episodes/02-enter-sandman.md index 52839e1..b0568d7 100644 --- a/_episodes/02-enter-sandman.md +++ b/_episodes/02-enter-sandman.md @@ -19,7 +19,8 @@ keypoints: As we enter the first episode of the Continuous Integration / Continuous Deployment (CI/CD) session, we learn how to exit. - + + # Start by Exiting How does a general task know whether or not a script finished correctly or not? You could parse (`grep`) the output: diff --git a/_episodes/03-understanding-yaml.md b/_episodes/03-understanding-yaml.md index 22a15ea..d53315e 100644 --- a/_episodes/03-understanding-yaml.md +++ b/_episodes/03-understanding-yaml.md @@ -12,7 +12,8 @@ keypoints: - YAML is a superset of JSON, so it contains additional features like comments and anchors, while still supporting JSON. --- - + + # YAML YAML (Yet Another Markup Language or sometimes popularly referred to as YAML Ain't Markup Language (a recursive acronym)) is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted. CI systems' modus operandi typically rely on YAML for configuration. We'll cover, briefly, some of the native types involved and what the structure looks like. diff --git a/_episodes/04-understanding-yaml-and-ci.md b/_episodes/04-understanding-yaml-and-ci.md index 9d547fa..bf8dd2c 100644 --- a/_episodes/04-understanding-yaml-and-ci.md +++ b/_episodes/04-understanding-yaml-and-ci.md @@ -13,7 +13,8 @@ keypoints: - A job is defined by a name and a script, at minimum. - Other than job names, reserved keywords are the top-level parameters defined in a YAML file. --- - + + # GitLab CI YAML The GitLab CI configurations are specified using a YAML file called `.gitlab-ci.yml`. Here is an example: @@ -43,18 +44,12 @@ This YAML configuration represents a basic GitLab CI/CD pipeline with one stage > Sometimes, `script` commands will need to be wrapped in single or double quotes. For example, commands that contain a colon (`:`) need to be wrapped in quotes so that the YAML parser knows to interpret the whole thing as a string rather than a “key: value” pair. Be careful when using special characters: `:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `\``. {: .callout} -This is the simplest possible configuration that will work for most code using minimal dependencies with `cmake` and `make`: - -1. Define one job `build_code` (the job names are arbitrary) with different commands to be executed. -2. Before every job, the commands defined by `before_script` are executed. - -The `.gitlab-ci.yml` file defines sets of jobs with constraints of how and when they should be run. The jobs are defined as top-level elements with a name (in our case `build_code`) and always have to contain the `script` keyword. Let's explore this structure in more depth. ## Overall Structure Every single parameter we consider for all configurations are keys under jobs. The YAML is structured using job names. For example, we can define three jobs that run in parallel (more on parallel/serial later) with different sets of parameters. -~~~ +~~~yml job1: param1: null param2: null @@ -68,7 +63,7 @@ job3: param4: null param5: null ~~~ -{: .language-yaml} + > ## Parallel or Serial Execution? > @@ -89,7 +84,7 @@ What can you not use as job names? There are a few reserved keywords (because th Global parameters mean that you can set parameters at the top-level of the YAML file. What does that actually mean? Here's another example: -~~~ +~~~yml stages: [build, test, deploy] : @@ -110,10 +105,6 @@ job_2: - echo "Commands for the second job - Step 2" ~~~ -{: .language-yaml} - -where `stages` is the global parameter being used. We will talk about stages later on. - > ## Stages??? > @@ -126,7 +117,7 @@ where `stages` is the global parameter being used. We will talk about stages lat What are some of the parameters that can be used in a job? Rather than copy/pasting from the reference (linked below in this session), we'll go to the [Configuration parameters](https://docs.gitlab.com/ee/ci/yaml/#configuration-parameters) section in the GitLab docs. The most important parameter, and the only one needed to define a job, is `script` -~~~ +```yml job one: script: make @@ -134,8 +125,7 @@ job two: script: - python test.py - coverage -~~~ -{: .language-yaml} +``` > ## Understanding the Reference > diff --git a/_episodes/06-hello-world-ci.md b/_episodes/06-hello-world-ci.md index 371498d..d7dc848 100644 --- a/_episodes/06-hello-world-ci.md +++ b/_episodes/06-hello-world-ci.md @@ -12,7 +12,8 @@ keypoints: - Pipelines are made of stages, stages are made of jobs. - CI Linters are especially useful to check syntax before pushing changes. --- - + + # Adding CI/CD to a project We've been working on the analysis code which has a lot of work done, but we should be good physicists (and people) by adding tests and CI/CD. The first thing we'll do is create a `.gitlab-ci.yml` file in the project. diff --git a/_episodes/07-adding-ci-to-your-code.md b/_episodes/07-adding-ci-to-your-code.md index 2a91789..8418fc1 100644 --- a/_episodes/07-adding-ci-to-your-code.md +++ b/_episodes/07-adding-ci-to-your-code.md @@ -13,7 +13,8 @@ keypoints: - All defined jobs run in parallel by default - Jobs can be allowed to fail without breaking your CI/CD --- - + + # Time To Skim ## The Naive Attempt diff --git a/_episodes/08-eins-zwei-dry.md b/_episodes/08-eins-zwei-dry.md index 6565803..d5e2d14 100644 --- a/_episodes/08-eins-zwei-dry.md +++ b/_episodes/08-eins-zwei-dry.md @@ -13,7 +13,7 @@ keypoints: - Using job templates allows you to stay DRY! ---
- +
diff --git a/_episodes/08b-matrix.md b/_episodes/08b-matrix.md index 7749107..5ed7746 100644 --- a/_episodes/08b-matrix.md +++ b/_episodes/08b-matrix.md @@ -13,6 +13,7 @@ keypoints: - Use Variables whenever it's convenient --- + # Parallel and Matrix jobs diff --git a/_episodes/08c-images.md b/_episodes/08c-images.md index df5dd18..ecdf6e6 100644 --- a/_episodes/08c-images.md +++ b/_episodes/08c-images.md @@ -11,7 +11,7 @@ hidden: false keypoints: - We can shorten a lot of the setup with Docker images --- - + # Say "Docker" 🐳 diff --git a/_episodes/10-the-worlds-a-stage.md b/_episodes/10-the-worlds-a-stage.md index 3fe0a82..0aedb92 100644 --- a/_episodes/10-the-worlds-a-stage.md +++ b/_episodes/10-the-worlds-a-stage.md @@ -11,7 +11,8 @@ keypoints: - Stages allow for a mix of parallel/serial execution. - Stages help define job dependencies. --- - + + # Defining Stages From the last session, we're starting with diff --git a/_episodes/11-skim-milk.md b/_episodes/11-skim-milk.md index b1f39b2..e51f537 100644 --- a/_episodes/11-skim-milk.md +++ b/_episodes/11-skim-milk.md @@ -12,7 +12,7 @@ keypoints: - Artifacts are pretty neat. - We're too naive. --- - + # The First Naive Attempt diff --git a/_episodes/12-the-spy-game.md b/_episodes/12-the-spy-game.md index 317def4..ce8e340 100644 --- a/_episodes/12-the-spy-game.md +++ b/_episodes/12-the-spy-game.md @@ -12,7 +12,6 @@ keypoints: - Service accounts provide an extra layer of security between the outside world and your account - Environment variables in GitLab CI/CD allow you to hide protected information from others who can see your code --- - diff --git a/_episodes/13-plotting-to-take-over-the-world.md b/_episodes/13-plotting-to-take-over-the-world.md index 157410e..fa4eda1 100644 --- a/_episodes/13-plotting-to-take-over-the-world.md +++ b/_episodes/13-plotting-to-take-over-the-world.md @@ -10,7 +10,8 @@ hidden: false keypoints: - Another script, another job, another stage, another artifact. --- - + + # On Your Own So in order to make plots, we just need to take the skimmed file `skim_ggH.root` and pass it through the `histograms.py` code that exists. This can be run with the following code diff --git a/_episodes/14-a-simple-test.md b/_episodes/14-a-simple-test.md index efacb3c..6a15a64 100644 --- a/_episodes/14-a-simple-test.md +++ b/_episodes/14-a-simple-test.md @@ -12,7 +12,7 @@ keypoints: - This kind of test is a regression test, as we're testing assuming the code up to this point was correct. - This is not a unit test. Unit tests would be testing individual pieces of the `atlas/athena` or `CMSSW` code-base, or specific functionality you wrote into your algorithms. --- - + So at this point, I'm going to be very hands-off, and just explain what you will be doing. Here's where you should be starting from: @@ -73,7 +73,7 @@ plot_ggH: > 1. Add a `test` stage after the `plot` stage. > 2. Add a test job, `test_ggH`, part of the `test` stage, and has the right `dependencies` > - Note: `./skim` needs to be updated to produce a `skim_ggH.log` (hint: `./skim .... > skim_ggH.log`) -> - We also need the hist_ggH.root file produced by the plot job +> - We also need the `hist_ggH.root` file produced by the plot job > 3. Create a directory called `tests/` and make two python files in it named `test_cutflow_ggH.py` and `test_plot_ggH.py` that uses `PyROOT` and `python3` > - you might find the following lines (below) helpful to set up the tests > 4. Write a few different tests of your choosing that tests (and asserts) something about `hist_ggH.root`. Some ideas are: diff --git a/setup.md b/setup.md index 5b0ee93..7756114 100644 --- a/setup.md +++ b/setup.md @@ -22,7 +22,7 @@ If you're having issues, **please let us know immediately** since you won't be able to follow this lesson without access to a GitLab instance. - + ## Creating a New Project