Add container information

_toc.yml

@@ -14,4 +14,8 @@ parts:
   - caption: Preprocessing Notebooks for NCAR RDA
     chapters:
       - file: notebooks/05_data_preprocessing
-      - file: notebooks/06_era5_anomaly
+      - file: notebooks/06_era5_anomaly
+
+  - caption: Creating the Dashboard in a Container 
+    chapters:
+      - file: notebooks/07_container_create

notebooks/07_container_create.ipynb

@@ -0,0 +1,343 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Create a container image of the dashboard"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Containers package software and dependencies into a single image that can be run on any platform with a container engine installed. The image is portable and immutable, the software inside won't change. It will reliably run in the same state years from now making it powerful for easy reproducibility."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The most popular container engine is [Docker](https://www.docker.com/) and it will be used for this example. The principles learned can be applied to other container engines. Before proceeding make sure Docker is installed on the machine you are using. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A container engine is the core software that manages containers. It provides a user interface that can be used to create, run, and interact with containers while isolating the resources required "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Container image file"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A container image is a lightweight, standalone, and executable software package that includes everything needed to run a piece of software, such as the code, runtime, libraries, environment variables, and configuration files. A container image file, popularly referred to as a Dockerfile, is a set of instructions on how to build and run a container image. The filename can be anything and a flag can be specified at build time with the filename that should be used to create an image. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Container images are built in layers. Each layer corresponds to instructions provided in the container image file. They are built in sequence before being stacked together for the final image. Once a layer is created it cannot be changed. If a layer in the container image file is changed, on build, any layers that come before it are reused while new layers are built for the changed, and subsequent, layer. This is used to speed up builds and save storage space by sharing common layers between images. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Example Containerfile"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "There is a file in the notebooks directory named Containerfile that contains instructions to create a container image that hosts the interactive notebook as a web server. That file can be used directly to build a container image without having to change anything. The contents of that file can be viewed below with comments describing each command found inline.   "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```docker\n",
+    "# Use the latest official micromamba image as the base image\n",
+    "FROM docker.io/mambaorg/micromamba:latest\n",
+    "\n",
+    "# Change to the root user in order to install OS packages\n",
+    "USER root\n",
+    "\n",
+    "# Update apt and install git\n",
+    "RUN apt-get update && apt-get install -y git-all\n",
+    "\n",
+    "# Set the working directory in the container to /home/mambauser/app\n",
+    "WORKDIR /home/mambauser/app\n",
+    "\n",
+    "# Run git clone to get the repo contents in to /home/mambauser/app\n",
+    "RUN git clone https://github.com/ProjectPythia/ERA5_interactive-cookbook.git\n",
+    "\n",
+    "# Create a conda environment from the environment.yml file included in the repo\n",
+    "RUN micromamba env create -f ERA5_interactive-cookbook/environment.yml\n",
+    "# Activate the environment by providing ENV_NAME as an environment variable at runtime \n",
+    "\n",
+    "# Copy the notebook with the panel application to the working directory\n",
+    "RUN mv ERA5_interactive-cookbook/notebooks/04_dashboard.ipynb .\n",
+    "\n",
+    "# Remove the rest of the repository as no other files are required \n",
+    "RUN rm -r ERA5_interactive-cookbook/\n",
+    "\n",
+    "# Make panel application port to the world outside this container\n",
+    "EXPOSE 5006\n",
+    "\n",
+    "# Set the user to be mambauser instead of root when the container runs\n",
+    "USER mambauser\n",
+    "\n",
+    "# Specify the command to run that starts the application\n",
+    "CMD [\"panel\", \"serve\", \"04_dashboard.ipynb\", \"--allow-websocket-origin=*\", \"--autoreload\"]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Each line in the example starts with a command that creates an individual layer in the image. The layers can contain multiple lines and are typically linked with `&& \\` before starting the next line. Each layer is cached to increase subsequent builds. For instance if we wanted to change the last line to have a specific websocket origin instead of the wildcard the next build would only update that layer since it changed and is last. If we decided to add a package to the environment.yml file it would start building layers at the COPY command, and continue to rebuild all subsequent layers since they may be dependant on the change that was made to the middle layer. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Build the container image"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "With the container image build instructions in place a command can now be run to build it. The command below is an example of building an image in the ncote namespace (my Docker Hub namespace), with the name pythia-era5-viz. It is tagged with the default of `:latest` which does not provide great version control and will be updated later. When running this command the image tag, `-t`, should be updated to something that makes sense for you."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`docker buildx build -f Containerfile -t ncote/pythia-era5-viz .`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Docker only recognizes container image files named `Dockerfile` by default so it's required to pass the container image file name `Containerfile` with the `-f` flag. It will take a few minutes for the image to build. When it is complete the output will look something like the following:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```shell\n",
+    " => exporting to image\n",
+    " => => exporting layers\n",
+    " => => writing image sha256:76b3466afbc89f6f4043d9135d2e1c3a2ea6da0d79329b2b80635e7966ba9025\n",
+    " => => naming to docker.io/ncote/pythia-era5-viz  \n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Run the container image"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that the image has been built it can also be ran locally to test functionality. Because the environment name is not set inside the container it's required to pass the name of the conda environment to activate by using the following flag `-e ENV_NAME=ERA5_interactive`. The command below will run the container and make it accessible via a web browser:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`docker run -e ENV_NAME=ERA5_interactive -p 5006:5006 ncote/pythia-era5-viz`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The site is now accessible at [http://localhost:5006/04_dashboard](http://localhost:5006/04_dashboard). "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Container registries"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A container registry is a centralized place where container images can be stored and distributed to others. There are a lot of different options when it comes to container registries.  Docker Hub will be used in the following example but the concepts would apply directly to other container registries. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Docker Hub"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "[Docker Hub](https://hub.docker.com/) is a container registry provided by Docker that is the default registry for most container engines, including Podman. The link at the beginning of this section will navigate to the Docker Hub page where a new account can be created by using the Sign up button in the top right, or the Sign in button can be used if you already have an account."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Login"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Once the Docker Hub account is setup it can be used to login from the command line"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```bash\n",
+    "docker login\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It will prompt for a username and password. Instead of using an actual password an Access Token can be created and supplied. At this [link to Docker Hub account settings](https://hub.docker.com/settings/security) a new access token can be created with only Read & Write access by selecting the New Access Token button. It will only be displayed one time and it should be stored somewhere secure."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Tag images"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Additional tags can be added to an image with the command `docker tag <original_tag> <new_tag>`. Since the image built before was tagged with latest it needs to be changed to something unique and more descriptive for version control. This can be done with a command like:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```bash\n",
+    "docker tag docker.io/ncote/pythia-era5-viz:latest docker.io/ncote/pythia-era5-viz:2024-06-20\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "There is now an additional tag for the image called `docker.io/ncote/pythia-era5-viz:2024-06-20`. No new space was used to add this tag. It still references the same image ID from the original build. An image can be tagged multiple times with different names without using up more space."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Push images"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Once logged in to Docker Hub the ability to push images directly to the namespace specified is opened up. This is accomplished for the new image tag by running the following:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```bash\n",
+    "docker push docker.io/ncote/pythia-era5-viz:2024-06-20\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It will take a bit to upload all the layers to the registry, but once complete the image is available for others to use by specifying the full image tag. In the example provided the full image tag is `docker.io/ncote/pythia-era5-viz:2024-06-20`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Sharing the image"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that the image has been pushed to a public container registry it's available for anyone to use. They can either run it directly with a command similar to what was run earlier:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`docker run -e ENV_NAME=ERA5_interactive -p 5006:5006 docker.io/ncote/pythia-era5-viz:2024-06-20`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If the image does not exist on the local machine the run command will pull it directly. If it's preferred to pull the image down prior to running it that can also be accomplished by running:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`docker pull docker.io/ncote/pythia-era5-viz:2024-06-20`"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}