From 288a03dcf97e8983c7d069da21eb28e0e2325577 Mon Sep 17 00:00:00 2001 From: Obi Griffith Date: Wed, 11 Dec 2024 22:47:37 -0600 Subject: [PATCH 1/5] additional details and clarifications, mostly around practicals for module 3 --- 03-using-containers.qmd | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/03-using-containers.qmd b/03-using-containers.qmd index c943cb2..665a650 100644 --- a/03-using-containers.qmd +++ b/03-using-containers.qmd @@ -11,7 +11,7 @@ Let's actually dive into using containers. You can follow along with this activi When learning another software skill like containers, it sometimes means you'll have to keep track of more than one window. We call this "Window Juggling". It's an overlooked skill but something you'll become more comfortable with. -**IMPORTANT**: Use the checkboxes to follow along with each step in the activities! This is to help you keep track because steps have to be done in order! +**IMPORTANT**: Use the checkboxes to follow along with each step in the activities! This is to help you keep track because the steps have to be done in order! ```{r panel-setup, include = FALSE} install.packages("xaringanExtra", repos = 'https://cloud.r-project.org') @@ -23,9 +23,9 @@ xaringanExtra::style_panelset_tabs(font_family = "inherit") ### Docker #### Set up - Go here to [install Docker](https://www.docker.com/get-started), following the instructions for your particular operating system.
+ If not already installed, go here to [install Docker](https://www.docker.com/get-started), following the instructions for your particular operating system.
If you don't have a Docker account create an account when prompted, or [go here](https://hub.docker.com/).
- After you install Docker, start up Docker desktop by double clicking on the app. It may take some time to start up.
+ Start up Docker desktop by double clicking on the app. It may take some time to start up.
```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_686") @@ -46,7 +46,7 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qa #### Activity 1: Pull and run a Docker image **This activity is built so that you will encounter errors that this guide will work you through!** -Most common container stumbling block is understanding the idea that containers are isolated from your computer +One of the most common container stumbling blocks is understanding the idea that containers are isolated from your computer. ```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_700") @@ -58,12 +58,12 @@ Which means: If your computer has files, software, or anything else. Your contai ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_713") ``` -We will encounter both an error of when we don't have a file we need AND an error of when we don't have a software package we need. We will guide you through what to do when these expected errors occur. +We will encounter both an error when we don't have a file we need AND an error when we don't have a software package we need. We will guide you through what to do when these expected errors occur. #### Step 1: Docker pull We need to get the Docker **image** we want to use. We do this by "pulling" it. -Pulling is a term used in GitHub terminology too. It just means we're taking something from the internet and getting a copy locally (on our computer). +Pulling is a term used in GitHub terminology too. It just means we're taking something from the internet and getting a copy locally (on our computer). Pulling is basically like downloading. ```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_732") @@ -84,15 +84,16 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qa ##### Step 2: Check what images you have To see what images we have we can run `docker image ls`.
+ Alternatively, you can visit the Images tab of the docker desktop app.
We should see `cansav09/practice-image:1` show up in the output.
```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_752") ``` -##### Step 3: Run the docker image to start a container! +##### Step 3: Use the Run command to start a container from the image! -Remember images can be shared easily but to actually use the stuff we need to turn our image into a container! To do that we will do `docker run` +Remember images can be shared easily but to actually use the stuff we need to turn our image into a container! To do that we will use the `docker run` command ```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_760") @@ -103,12 +104,18 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qa docker run cansav09/practice-image:1 ``` - To see what containers we have running we have we can run: + Alternatively, we can press the play button for the image in images tab of the desktop app. + + + To see what containers we have running we have we can run (in a new terminal session): ``` docker ps ``` + Alternatively, you can visit the Containers tab of the docker desktop app. + + ```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_775") ``` @@ -137,8 +144,9 @@ docker exec -it bash ```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_803") ``` +You can find the `CONTAINER ID` in the containers tab of the desktop app or by using the `docker container ls` command in the terminal. Note that whereas the `docker run` command creates a new container from a docker image, the `docker exec` command executes a command inside an already running container. The `-it` option tells docker to execute the command interactively. In this case the actual command being specified is `bash`. Altogether the result is an interactive Bash shell inside the already running container. -If you run this you should see your terminal/command prompt now have the initial part change to something like: +If you run the above command you should see your terminal/command prompt now have the initial part change to something like: ``` root@e7b19cfb4461:/ @@ -165,9 +173,11 @@ root@e7b19cfb4461:/# ##### Step 5: Try calling a script - To run a script using the docker container we could just add reference to a script at the end: + Make sure you are in the unzipped workshop directory. + To run a script using the docker container we could just add reference to a script at the end. ``` +cd containers-for-scientists-sandbox-main docker exec -it bash run_analysis.sh ``` From ff45146bc1aef285b7e72a38b173e9006c49db38 Mon Sep 17 00:00:00 2001 From: Obi Griffith Date: Thu, 12 Dec 2024 22:48:30 -0600 Subject: [PATCH 2/5] edits and clarifications in module 4 --- 04-using-volumes.qmd | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/04-using-volumes.qmd b/04-using-volumes.qmd index a73f9ac..6d47df0 100644 --- a/04-using-volumes.qmd +++ b/04-using-volumes.qmd @@ -11,12 +11,11 @@ So how do we get files we need onto a container we are using? There are a few options: -- We could run our container and within that we could download the files we need from online! For example we could `git clone` files from a repository or otherwise `wget` or `curl` files that are stored. -- We could `COPY` a file on to the docker image when we are building it (more on this next chapter). But this strategy we want to be careful with for two reasons: - - We don't want our image to be too big. so this strategy can be fine if the file is small and something we know will always be needed for any use of the container. +- We could run our container and within that we could download the files we need from online! For example we could `git clone` files from a repository or otherwise `wget` or `curl` files that are stored online somewhere. +- We could `COPY` a file on to the docker image when we are building it (more on this next chapter). But we want to be careful with this strategy for two reasons: + - We don't want our image to be too big. So, this strategy can be fine if the file is small and something we know will always be needed for any use of the container. - We DEFINITELY need to avoid ever having protected data on images that are shared. NEVER put protected data on images that we are sharing -- more on this later too. -- Lastly we can add a `volume` of files that are local to our computer. We'll dive into this strategy in this chapter. This will probably be a common way you'll use to get access to files on your container. - +- Lastly we can add a `volume` of files that are local to our computer. We'll dive into this strategy in this chapter. This will probably be the most common way you'll use to get access to files on your container. What is a volume? A volume is a folder, likely from your computer, that can be accessed by your container. @@ -40,13 +39,13 @@ xaringanExtra::style_panelset_tabs(font_family = "inherit") ### Docker -Our container is separate from our computer so if we want to use a file we have to attach it using a "volume". +Our container is separate from our computer so if we want to use a file from our computer we have to attach it using a "volume". ##### Step 1: Let’s add our containers-for-scientists-sandbox files Let's point a volume to our workshop files so we have them on our container. -We can specify a particular file path on our computer or give it `$PWD` Then optionally we can give a `:` and a file path we'd like this to be stored on on the container. Otherwise it will be stored at the absolute top of the container. +We can specify a particular file path on our computer or give it `$PWD`. Then optionally we can give a `:` and a file path where we'd like it to be stored on the container. Otherwise it will be stored at the absolute top of the container. Note that `$PWD` is a special environment variable that stores the absolute path of the current working directory. You will need to be in the `containers-for-scientists-sandbox-main` for this to work. ```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_1128") @@ -54,7 +53,7 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qa Now we can run: ``` -docker run -v $pwd:/home cansav09/practice-image:1 +docker run -v $PWD:/home cansav09/practice-image:1 ``` If you have a windows machine you may have to run this variant instead. This version has a different `${}` around the `pwd` part. ``` @@ -69,7 +68,8 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qa #### Step 2: Retry calling the script - Now we can run the following command but we will have to run `docker ps` and get the container ID we need to put here. + Now we can run the following command but we will have to run `docker ps` or `docker container ls` and get the container ID we need to put here. + ``` docker exec -it bash /home/run_analysis.sh ``` From 2708b537f36c373dcd1f315708b25cc1092a1da9 Mon Sep 17 00:00:00 2001 From: Obi Griffith Date: Fri, 13 Dec 2024 22:35:26 -0600 Subject: [PATCH 3/5] more edits/clarifications this time for module 5 --- 05-modifying-containers.qmd | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/05-modifying-containers.qmd b/05-modifying-containers.qmd index 35caf70..1d3ad2e 100644 --- a/05-modifying-containers.qmd +++ b/05-modifying-containers.qmd @@ -18,8 +18,6 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qa ## Activity Instructions -A Dockerfile is a recipe for how to build a docker image. The best way to learn to write Dockerfiles is to start off with one that is already written and modify it for your needs. - Open up the Dockerfile in the `activity-files` folder. ```{r, out.width = "100%", echo = FALSE} @@ -32,13 +30,13 @@ FROM cansav09/practice-image:1 ``` This means we're going to take the existing image called, `cansav09/practice-image:1` and build on to it. -This image will be our base. There are so many Docker images out there, that it might be that someone has already created a docker image with most of the functionality you need for your project. +This image will be our base. There are so many Docker images out there that it might be that someone has already created a docker image with most of the functionality you need for your project. ```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a4ed49e59_0_1200") ``` -The trick is to find a base image that has most of the software things you need, but not extra stuff you don't need. You want to make your image have all the stuff it needs but also be as small as possible. IMages that are overly big are harder to deal with and to download. +The trick is to find a base image that has most of the software things you need, but not extra stuff you don't need. You want to make your image have all the stuff it needs but also be as small as possible. Images that are overly big are harder to deal with and to download. _There are (at least) two strategies you can take_: @@ -49,7 +47,7 @@ _There are (at least) two strategies you can take_: Let's take a look at a Dockerfile. #### Step 1: Use any file editor to open up the Dockerfile - Double click it and open the file to take a look at it with any default text editor. + Open the file to take a look at it with any text editor. #### Step 2: Change your working directory to activity-files @@ -94,7 +92,6 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qa #### Step 5: Run the new image - Navigate back to your Docker desktop and the `images` window or run `docker ps` or `podman ps`. If your image built successfully, you should see a new image in your list! Let's try running that image. ``` @@ -116,16 +113,18 @@ ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qa #### Step 7: Edit the Dockerfile so it has the installation step for rmarkdown package and remove the CMD step -For anything we need ran in this image we are building we need to use the `RUN` command followed by the installation steps we'd need. +For anything we need to run in the image we are building we need to use the `RUN` command followed by the installation steps we'd need. Open up the file called `Dockerfile` in `activity-files`. + **Copy and paste** this into your Dockerfile below where it says `# Add a new package here ` so we can add the `rmarkdown` package. ``` RUN Rscript -e "options(warn = 2);install.packages('rmarkdown', \ repos = 'https://cloud.r-project.org/')" ``` - **AND remove the `CMD` line**. + **remove the `CMD` line**. + Save your edited Dockerfile. @@ -151,7 +150,7 @@ If all built successfully, you should see a message like: #### Step 9: Run container from cool-new-image:2 -Now let's retry running the script from here but we will need to specify the volume again! +Now let's retry running the script from here but we will need to specify the volume again! Make sure that you are in the top level `containers-for-scientists-sandbox-main` directory. First run the container using the `2` image: ``` @@ -161,6 +160,7 @@ docker run -v $PWD:/home cool-new-image:2 #### Step 10: Re-Retry calling the script Run `docker ps` or `podman ps` can get you the container ID. Or look on your Docker Desktop. + Try running the script using the following command: ``` docker exec -it bash /home/run_analysis.sh @@ -302,8 +302,8 @@ RUN pip3 install \ "somepackage==0.1.0" ``` -There are so many things you can add to your Docker image. (Picture whatever software and packages you are using on your computer). We can only get you started for the feel of how to build a Dockerfile, and what you put on your Docker image will be up to you. +There are so many things you can add to your Docker image. (Picture whatever software and packages you are using on your computer). We have gotten you started with a simple example of how to write a Dockerfil and build a docker image from a base image plus some additional packages. But, what you put on your Docker image will be up to you. -To figure out how to add something, a good strategy is to look for other Dockerfiles that might have the package you want installed and borrow their `RUN` command. Then try to re-build your Docker image with that added `RUN` command and see if it builds successfully. +To figure out how to add something, a good strategy is to look for other Dockerfiles that might have the package you want installed and borrow their `RUN` command. Then try to re-build your Docker image with that added `RUN` command and see if it builds successfully. Another strategy is to enter an interactive terminal session on your base image, work out the required commands for installing the missing tool/package, then add those `RUN` commands to your Dockerfile. And lastly, make sure that whatever changes you make to your Dockerfile, that you add it to your GitHub repository! From a585e20bc13342c450b2c0ee7e5602de54ae4015 Mon Sep 17 00:00:00 2001 From: Obi Griffith Date: Tue, 17 Dec 2024 16:45:06 -0600 Subject: [PATCH 4/5] minor edits --- 06-writing-dockerfiles.qmd | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/06-writing-dockerfiles.qmd b/06-writing-dockerfiles.qmd index 41a748f..034c973 100644 --- a/06-writing-dockerfiles.qmd +++ b/06-writing-dockerfiles.qmd @@ -1,21 +1,17 @@ # Writing Dockerfiles - - ```{r, out.width = "100%", echo = FALSE} ottrpal::include_slide("https://docs.google.com/presentation/d/1T5Lfei2UVou9b0qaUCrWXmkcIwAao-UcN4pHMPEE4CY/edit#slide=id.g30a80783034_1_34") ``` - - Now that you're familiar with the basics of Dockerfiles and how to use them to build images, let's dive into some more of the things you can do with them. -`FROM` is one of the [main commands that a Dockerfile can take as described by their documentation](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/). +`FROM` is one of the [main commands that a Dockerfile can take, as described by their documentation](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/). -Now you are also familiar with `CMD` which runs something when the container is built +Now you are also familiar with `CMD` which runs something when the container is built. -> **FROM** creates a layer from the another Docker image. +> **FROM** creates a layer from another Docker image. > **CMD** specifies what command to run within the container. > **RUN** builds your application with make. > **COPY** adds files from your Docker client’s current directory. From e38547412f04eefc5433d723cbbad30befd562d5 Mon Sep 17 00:00:00 2001 From: Candace Savonen Date: Thu, 19 Dec 2024 12:20:13 -0500 Subject: [PATCH 5/5] Update 03-using-containers.qmd --- 03-using-containers.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/03-using-containers.qmd b/03-using-containers.qmd index 665a650..5f0608a 100644 --- a/03-using-containers.qmd +++ b/03-using-containers.qmd @@ -113,7 +113,7 @@ docker run cansav09/practice-image:1 docker ps ``` - Alternatively, you can visit the Containers tab of the docker desktop app. + Alternatively, you can visit the Containers tab of the Docker Desktop app. ```{r, out.width = "100%", echo = FALSE}