Merge pull request #34 from timclegg/feat/no-custom-image

feat: removing custom image usage

beginners/README.md

@@ -2,7 +2,7 @@
 
 ## Prerequisites
 
-You'll need an OCI free trial account: <a href="https://signup.cloud.oracle.com/?sourceType=_ref_coc-asset-opcSignIn&language=en_US" target="_blank" title="Sign up for free trial">click here to sign up</a> (right click and open in a new tab so you can keep these instructions open). We're going to use a ready-to-go image to install the required resources, so all you need to start is a free account.
+You'll need an OCI free trial account: <a href="https://signup.cloud.oracle.com/?sourceType=_ref_coc-asset-opcSignIn&language=en_US" target="_blank" title="Sign up for free trial">click here to sign up</a> (right click and open in a new tab so you can keep these instructions open).
 
 Registered lab participants should have received $500 in credits to use for Data Science operations.
 
@@ -29,36 +29,43 @@ For Windows, and step-by-step instructions for Mac/Linux, please see the [Oracle
     <a href="https://cloud.oracle.com/resourcemanager/stacks/create?region=home&zipUrl=https://github.com/oracle-devrel/redbull-analytics-hol/releases/latest/download/redbull-analytics-hol-latest.zip" target="_blank"><img src="https://oci-resourcemanager-plugin.plugins.oci.oraclecloud.com/latest/deploy-to-oracle-cloud.svg" alt="Deploy to Oracle Cloud"/></a>
 2. If needed, log into your account. You should then be presented with the **Create Stack** page. 
 
-    These next few steps will deploy a stack to your OCI tenancy. This will include a Compute instance and the necessary tools to deploy and run JupyterLab from within your OCI account.
+    These next few steps will deploy a stack to your OCI tenancy. This will include the necessary tools to deploy and run JupyterLab from within your OCI account.
 
     Under *Stack Information* (the first screen), check the box *I have reviewed and accept the Oracle Terms of Use*. Once that box is checked, the information for the stack will be populated automatically.
 
     ![Create Stack](./docs/red-bull-hol-1a-create-stack-information.jpg)
-3. Click **Next** at the bottom of the screen. This will take you to the **Configure Variables** page. On this page you'll need to provide the SSH key we created in the prerequisites if you want SSH access to your Compute instance.
+3. Click **Next** at the bottom of the screen. This will take you to the **Configure Variables** page. On this page you'll need to provide the SSH key we created in the prerequisites. You can copy and paste the contents of the public key (`.pub`), or use the file selector to upload the `.pub` file directly.
 
     ![Configure Variables](./docs/red-bull-hol-configure-variables.jpg)
 4. On the **Review** page, be sure *Run Apply* is checked, and click **Create**.
 
     ![Review and Create](./docs/red-bull-hol-1c-create-stack-review.jpg)
-5. This will take you to the **Job Details** page, and OCI will begin creating the stack and deploying the custom image for the lab. This will take about 11 minutes. When it completes (assuming everything went smoothly), the **Job Details** will show a bright green square with "Succeeded" below it.
+5. This will take you to the **Job Details** page, and OCI will begin creating the stack and deploying the scripts for the lab. This will take 8-10 minutes. When it completes (assuming everything went smoothly), the **Job Details** will show a bright green square with "Succeeded" below it.
 
     ![Create Stack Succeeded](./docs/red-bull-hol-1d-create-stack-succeeded.jpg)
-6. Once the Create Stack job has succeeded, click the hamburger menu in the upper left, select **Compute** in the sidebar, and click **Instances** in the menu.
+6. Once the Create Stack job has succeeded, click "Outputs."
 
-    ![Instances in the Menu](./docs/red-bull-hol-2a-menu-instances.jpg)
-7. On the **Instances** screen, make sure "redbullhol" is selected under *Compartment*. If "redbullhol" isn't in the dropdown menu, you may need to refresh the page for the new compartment to show up.
+    ![Show Outputs](./docs/red-bull-hol-show-outputs.jpg)
 
-    ![Instances Compartment](./docs/red-bull-hol-2c-instances-compartment.jpg)
-8. Once the "redbullhol" compartment is selected, you should see a running Instance in the list. The address you'll need to access it is in the *Public IP* column. Copy the IP address shown.
+    The outputs will include a terminal command you'll need to run. Replace the private key reference with the path to the private key that matches the public key you uploaded. The final command should look similar to:
 
-    ![Public IP](./docs/red-bull-hol-2d-instances-public-ip.jpg)
-9. Next, open a new tab in your browser to load up the web UI for JupyterLab. Paste the IP address you just copied with `:8001` added to the end. The URL should look like `http://xxx.xxx.xxx.xxx:8001` (substituting the public IP we copied in the previous step). JupyterLab is running on port 8001, so when you navigate to this URL you should see the Juypter login.
+    ```console
+    $ ssh -i ~/.ssh/id_rsa2 opc@123.456.789.100 'source redbullenv/bin/activate; jupyter server list'
+    ```
 
-    _**Note:** You should not be on VPN when opening JupyterLab._
+7. Open a terminal and run the above command. The output of the command will include a url containing a token. You'll need the token portion of the url (the part after `?token=`) for the next step.
+
+    ![SSH Output](./docs/red-bull-hol-terminal-ssh-output.jpg)
+8. Back in the **Outputs** display in your browser, copy the IP address for your Jupyter Lab (ending with `:8888`). Paste that into a new browser window/tab and append `?token=YOUR_TOKEN`, replacing `YOUR_TOKEN` with the token from step 7. Load the resulting URL.
+
+    ![Jupyter Lab URL](./docs/red-bull-hol-jupyter-url.jpg)
+9. Your new deployment uses a self-signed certificate, so you'll get a warning from your browser that the connection is insecure. You'll need to approve the loading of the page. In most browsers you'll see an "Advanced" or similar button, and clicking it will offer you the option to ignore security concerns and load the page anyway.
 
-    ![Jupyter Login](./docs/red-bull-hol-3b-jupyter-login.jpg)
-10. Log in with the password `Redbull1`.
-11. You should now see the JupyterLab. Navigate in the sidebar to `/redbull-analytics-hol/beginners/` to see the Jupyter notebooks for this lab.
+    You'll need to repeat the security approval procedure when loading the Flask app created by the lab.
+
+    _**Note:** You should not be on VPN when opening JupyterLab._
+    
+10. You should now see the JupyterLab. Navigate in the sidebar to `/redbull-analytics-hol/beginners/` to see the Jupyter notebooks for this lab.
 
 ## Up and Running
 

beginners/web/app.py

@@ -67,6 +67,6 @@ def predict_position():
     return render_template('index.html',tables=[df1.to_html(classes='driver')])
 
 if __name__ == '__main__':
-    app.run(host='0.0.0.0', port=8080, debug=True)
+    app.run(host='0.0.0.0', port=8443, debug=True, ssl_context='adhoc')
 
 

terraform/README.md

@@ -196,3 +196,73 @@ When you no longer need the deployment, you can run this command to destroy the
 ```
 terraform destroy
 ```
+
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.5 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_oci"></a> [oci](#provider\_oci) | 4.31.0 |
+| <a name="provider_oci.home"></a> [oci.home](#provider\_oci.home) | 4.31.0 |
+| <a name="provider_random"></a> [random](#provider\_random) | 3.1.0 |
+| <a name="provider_time"></a> [time](#provider\_time) | 0.7.2 |
+| <a name="provider_tls"></a> [tls](#provider\_tls) | 3.1.0 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [oci_core_default_dhcp_options.Default-DHCP-Options-for-redbullvcn](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_default_dhcp_options) | resource |
+| [oci_core_default_route_table.Default-Route-Table-for-redbullvcn](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_default_route_table) | resource |
+| [oci_core_default_security_list.Default-Security-List-for-redbullvcn](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_default_security_list) | resource |
+| [oci_core_instance.redbull_lab1](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_instance) | resource |
+| [oci_core_internet_gateway.redbullig](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_internet_gateway) | resource |
+| [oci_core_subnet.redbullsubnet](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_subnet) | resource |
+| [oci_core_vcn.redbullvcn](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_vcn) | resource |
+| [oci_identity_compartment.redbullhol](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/identity_compartment) | resource |
+| [oci_identity_tag.release](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/identity_tag) | resource |
+| [oci_identity_tag_namespace.devrel](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/identity_tag_namespace) | resource |
+| [random_id.tag](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |
+| [time_sleep.wait_60_seconds](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |
+| [tls_private_key.this](https://registry.terraform.io/providers/hashicorp/tls/latest/docs/resources/private_key) | resource |
+| [oci_core_images.this](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/data-sources/core_images) | data source |
+| [oci_identity_availability_domain.AD1](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/data-sources/identity_availability_domain) | data source |
+| [oci_identity_compartment.current_compartment](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/data-sources/identity_compartment) | data source |
+| [oci_identity_region_subscriptions.home_region_subscriptions](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/data-sources/identity_region_subscriptions) | data source |
+| [oci_identity_regions.current_region](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/data-sources/identity_regions) | data source |
+| [oci_identity_regions.home-region](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/data-sources/identity_regions) | data source |
+| [oci_identity_tenancy.tenant_details](https://registry.terraform.io/providers/hashicorp/oci/latest/docs/data-sources/identity_tenancy) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_compartment_ocid"></a> [compartment\_ocid](#input\_compartment\_ocid) | The compartment OCID to deploy resources to | `string` | `""` | no |
+| <a name="input_compute_image_name"></a> [compute\_image\_name](#input\_compute\_image\_name) | The name of the compute image to use for the compute instances. | `string` | `"Oracle-Linux-7.9-2021.08.27-0"` | no |
+| <a name="input_fingerprint"></a> [fingerprint](#input\_fingerprint) | 'API Key' fingerprint, more details can be found at https://docs.cloud.oracle.com/en-us/iaas/Content/General/Concepts/credentials.htm#two | `string` | `""` | no |
+| <a name="input_private_key"></a> [private\_key](#input\_private\_key) | The private key (provided as a string value) | `string` | `""` | no |
+| <a name="input_private_key_password"></a> [private\_key\_password](#input\_private\_key\_password) | The password to use for the private key | `string` | `""` | no |
+| <a name="input_private_key_path"></a> [private\_key\_path](#input\_private\_key\_path) | Path to private key used to create OCI 'API Key', more details can be found at https://docs.cloud.oracle.com/en-us/iaas/Content/General/Concepts/credentials.htm#two | `string` | `""` | no |
+| <a name="input_redbull_compartment"></a> [redbull\_compartment](#input\_redbull\_compartment) | The name of the compartment created to hold all of the resources | `string` | `"redbullhol"` | no |
+| <a name="input_region"></a> [region](#input\_region) | OCI Region as documented at https://docs.cloud.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm | `string` | n/a | yes |
+| <a name="input_ssh_public_key"></a> [ssh\_public\_key](#input\_ssh\_public\_key) | The public SSH key to use for the compute instance | `string` | `""` | no |
+| <a name="input_ssh_public_key_path"></a> [ssh\_public\_key\_path](#input\_ssh\_public\_key\_path) | The path to the public SSH key to use for the compute instance | `string` | `""` | no |
+| <a name="input_tenancy_ocid"></a> [tenancy\_ocid](#input\_tenancy\_ocid) | OCI tenant OCID, more details can be found at https://docs.cloud.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#five | `string` | n/a | yes |
+| <a name="input_user_ocid"></a> [user\_ocid](#input\_user\_ocid) | OCI user OCID, more details can be found at https://docs.cloud.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#five | `string` | `""` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_get_jupyter_token"></a> [get\_jupyter\_token](#output\_get\_jupyter\_token) | n/a |
+| <a name="output_instance_pub_ip"></a> [instance\_pub\_ip](#output\_instance\_pub\_ip) | n/a |
+| <a name="output_jupyter_url"></a> [jupyter\_url](#output\_jupyter\_url) | n/a |

terraform/computes.tf

@@ -1,24 +1,10 @@
 # Copyright (c) 2021 Oracle and/or its affiliates.
 
-### custom image
-
-resource "oci_core_image" "redbullhol_image" {
-  compartment_id = oci_identity_compartment.redbullhol.id
-  depends_on = [time_sleep.wait_60_seconds]
-
-  display_name = "redbullhol_image"
-
-  image_source_details {
-    source_type = "objectStorageUri"
-    source_uri = "https://objectstorage.eu-frankfurt-1.oraclecloud.com/p/i0H4G1oKfKV7nOqjyeJ_SsXKhIZFgfmXFcFVEcr_9XEAVpkTOHc2MWtoz2WNmMXr/n/emeasespainsandbox/b/publichol/o/redbullhol-20210809-1523"
-  }
-}
-
 #### redbull_lab1
 resource oci_core_instance redbull_lab1 {
   availability_domain = data.oci_identity_availability_domain.AD1.name
   compartment_id      = oci_identity_compartment.redbullhol.id
-  shape               = "VM.Standard2.2"
+  shape = "VM.Standard2.2"
 
   agent_config {
     is_management_disabled = "false"
@@ -37,11 +23,11 @@ resource oci_core_instance redbull_lab1 {
   display_name      = "redbulllab1"
 
   metadata = {
-    "ssh_authorized_keys" = var.ssh_public_key
+    ssh_authorized_keys = local.ssh_public_keys
   }
-
+  
   source_details {
-    source_id = oci_core_image.redbullhol_image.id    
+    source_id = local.list_images[var.compute_image_name].id
     source_type = "image"
   }
 
@@ -51,4 +37,95 @@ resource oci_core_instance redbull_lab1 {
   defined_tags = {
     "${oci_identity_tag_namespace.devrel.name}.${oci_identity_tag.release.name}" = local.release
   }
+
+    connection {
+    agent       = false
+    timeout     = "60m"
+    host        = oci_core_instance.redbull_lab1.public_ip
+    user        = "opc"
+    private_key = tls_private_key.public_private_key_pair.private_key_pem
+  }
+
+  provisioner "file" {
+    source      = "scripts/jupyterlab.service"
+    destination = "/home/opc/jupyterlab.service"
+
+    connection {
+      private_key = tls_private_key.this.private_key_pem
+      user = "opc"
+      host = self.public_ip
+    }
+  }
+
+  provisioner "file" {
+    source      = "scripts/launchapp.sh"
+    destination = "/home/opc/launchapp.sh"
+
+    connection {
+      private_key = tls_private_key.this.private_key_pem
+      user = "opc"
+      host = self.public_ip
+    }
+  }
+
+  provisioner "file" {
+    source      = "scripts/launchjupyterlab.sh"
+    destination = "/home/opc/launchjupyterlab.sh"
+
+    connection {
+      private_key = tls_private_key.this.private_key_pem
+      user = "opc"
+      host = self.public_ip
+    }
+  }
+
+  provisioner "file" {
+    source      = "scripts/script_install_pyenv.sh"
+    destination = "/home/opc/script_install_pyenv.sh"
+
+    connection {
+      private_key = tls_private_key.this.private_key_pem
+      user = "opc"
+      host = self.public_ip
+    }
+  }
+
+  provisioner "file" {
+    source      = "scripts/script_install_root.sh"
+    destination = "/home/opc/script_install_root.sh"
+
+    connection {
+      private_key = tls_private_key.this.private_key_pem
+      user = "opc"
+      host = self.public_ip
+    }
+  }
+
+  provisioner "file" {
+    source      = "scripts/checkpyenv.py"
+    destination = "/home/opc/checkpyenv.py"
+
+    connection {
+      private_key = tls_private_key.this.private_key_pem
+      user = "opc"
+      host = self.public_ip
+    }
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "chmod +x /home/opc/script_install_pyenv.sh",
+      "chmod +x /home/opc/script_install_root.sh",
+      "chmod +x /home/opc/launchjupyterlab.sh",
+      "chmod +x /home/opc/launchapp.sh",
+      "/home/opc/script_install_pyenv.sh",
+      "sudo -i /home/opc/script_install_root.sh"
+    ]
+
+    connection {
+      private_key = tls_private_key.this.private_key_pem
+      user = "opc"
+      host = self.public_ip
+    }
+  }
 }

terraform/data_sources.tf

@@ -34,4 +34,12 @@ data "oci_identity_region_subscriptions" "home_region_subscriptions" {
 data oci_identity_availability_domain AD1 {
   compartment_id = var.compartment_ocid
   ad_number      = "1"
-}
+}
+
+data "oci_core_images" "this" {
+  compartment_id = var.compartment_ocid
+  filter {
+    name = "state"
+    values = ["AVAILABLE"]
+  }
+}

terraform/locals.tf

@@ -0,0 +1,22 @@
+# Copyright (c) 2021 Oracle and/or its affiliates.
+
+locals {
+  private_key = try(file(var.private_key_path), var.private_key)
+  ssh_public_key = try(file(var.ssh_public_key_path), var.ssh_public_key)
+
+  ssh_public_keys = join("\n", [
+    trimspace(local.ssh_public_key),
+    trimspace(tls_private_key.this.public_key_openssh)
+  ])
+
+  release = "1.0"
+
+  #Transform the list of images in a tuple
+  list_images = { for s in data.oci_core_images.this.images :
+    s.display_name =>
+    { id = s.id,
+      operating_system = s.operating_system
+    }
+  }
+
+}

terraform/orm/provider.tf

@@ -1,4 +1,4 @@
-# Copyright (c) 2020 Oracle and/or its affiliates.
+# Copyright (c) 2020-2021 Oracle and/or its affiliates.
 # Licensed under the Universal Permissive License v 1.0 as shown at http:#oss.oracle.com/licenses/upl.
 # 
 

terraform/outputs.tf

@@ -0,0 +1,13 @@
+# Copyright (c) 2021 Oracle and/or its affiliates.
+
+output "instance_pub_ip" {
+  value = oci_core_instance.redbull_lab1.public_ip
+}
+
+output "jupyter_url" {
+  value = "https://${oci_core_instance.redbull_lab1.public_ip}:8888"
+}
+
+output "get_jupyter_token" {
+  value = "ssh -i PATH_TO_YOUR_SSH_PRIV_KEY_HERE opc@${oci_core_instance.redbull_lab1.public_ip} 'source redbullenv/bin/activate; jupyter server list'"
+}

terraform/provider.tf

@@ -1,4 +1,4 @@
-# Copyright (c) 2020 Oracle and/or its affiliates.
+# Copyright (c) 2020-2021 Oracle and/or its affiliates.
 
 terraform {
   required_version = ">= 0.13.5"