Instructions-and-workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c9fb5d5b-0442-473e-ae15-ebc4e987e4ec",
   "metadata": {},
   "source": [
    "# Fireveg DB analysis -- Instructions and workflow for this repository\n",
    "\n",
    "Author: [José R. Ferrer-Paris](https://github.com/jrfep)\n",
    "\n",
    "Date: 24 August 2024\n",
    "\n",
    "This repository includes **[R](https://www.r-project.org/) code to analyse and visualise data** from the the Fireveg database. This Jupyter Notebook contains instructions on how we set up our instance of Jupyter Lab and on how to navigate this repository.\n",
    "\n",
    " <div class=\"alert alert-info\">\n",
    "<img src='img/open-code-1.png' width=35 alt=\"open data icon\"/> Code and version history is available from the following GitHub repository:\n",
    "\n",
    "> https://github.com/ces-unsw-edu-au/fireveg-analysis\n",
    "     \n",
    "</div>\n",
    " <div class=\"alert alert-info\">\n",
    "<img src='img/open-data-2.png' width=35 alt=\"open data icon\"/> Data used in this repository is available from the following OSF repository:\n",
    "\n",
    "> Ferrer-Paris, J. R., Keith, D., & Sánchez-Mercado, A. (2024, August 15). Export data records from Fire Ecology Traits for Plants database. Retrieved from [osf.io/h96q2](https://osf.io/h96q2/)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b2be093-5ffb-46e5-82bb-0648f8426efb",
   "metadata": {},
   "source": [
    "## Repository structure\n",
    "\n",
    "All the script are saved as Jupyter Notebooks and include documentation and comments on each step. Items with a green checkmark (✅) have been updated for version 1.1 (August 2024). Some items in this list are still work in progress (⌛), and others are just old or outdated, and will be deleted soon (🦕). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d683177-c64e-4320-82a5-53758270679b",
   "metadata": {},
   "source": [
    "### Images/icons"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f40aa55f-b174-4a7a-8da8-16b0ae27b218",
   "metadata": {},
   "source": [
    "Shared images and icons in `img` folder ✅\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c322160-bf35-45b1-98b6-240ad17da35c",
   "metadata": {},
   "source": [
    "### Credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b10e3bc2-9e27-4220-8d04-95a8dd805410",
   "metadata": {},
   "source": [
    "🤫 We use a folder named `secrets` to keep the credentials for connection to different services (database credentials, API keys, etc). We checked this folder in our `.gitignore` so that its content are not tracked by git and not exposed. Future users need to copy the contents of this folder manually.\n",
    "\n",
    "For R I use a `Renviron.local` file with a <key>=<value> format like this:\n",
    "\n",
    "```sh\n",
    "STADIA_API_KEY=\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aec7801f-6720-4647-8477-c5c2e8ef0330",
   "metadata": {},
   "source": [
    "### Data folder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f001e045-161a-4402-a543-71ee3651e0ae",
   "metadata": {},
   "source": [
    "🐘 The data folder is checked in our `.gitignore` so that its contents are not tracked by git. We use this folder locally, and then download the files from a cloud storage. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a87daf73-0340-4964-8f73-85515c1bb19d",
   "metadata": {},
   "source": [
    "### Notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8a80bfe-05b1-43ca-b1e3-9ebc98db87dc",
   "metadata": {},
   "source": [
    "Folder `Notebooks` contains all the documented code for data analysis and visualisation:\n",
    "\n",
    "- [Graph of existing sources for fire ecology traits](Notebooks/Graph-existing-sources-traits.ipynb) ✅ \n",
    "- [Overview of taxonomic coverage](RDS-output/Overview-taxonomic-coverage.ipynb) ✅ \n",
    "- [Overview of spatial coverage](RDS-output/Overview-spatial-coverage.ipynb) ⌛\n",
    "- "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2599c551-4175-4ac4-ad8d-0d48b582cccf",
   "metadata": {},
   "source": [
    "### Miscellaneous"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fb64728-7a35-4823-99fa-b03bec07ee0e",
   "metadata": {},
   "source": [
    "Older/outdated or work in progress ⌛\n",
    "This folder is a mess!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9eda153-15a0-47a4-ac0a-58e17d2d4053",
   "metadata": {},
   "source": [
    "## How to use Jupyter Lab\n",
    "\n",
    "I use Jupyter Lab to organise and document all the code in notebooks.  \n",
    "\n",
    "These are the steps I followed to configure a Jupyter Lab environment. This is a simple collection of note, not an exhaustive how to.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a687a655-5ff2-401a-9f93-2df399080992",
   "metadata": {},
   "source": [
    "### Create and activate python environment\n",
    "\n",
    "#### with venv\n",
    "\n",
    "This is my preferred method after some frustrations with conda.\n"
   ]
  },
  {
   "cell_type": "raw",
   "id": "f073f09c-cb6c-42a8-8f2c-5beca521f955",
   "metadata": {},
   "source": [
    "# Run this in a shell or terminal (bash or zsh):\n",
    "conda deactivate\n",
    "mkdir -p $HOME/venv\n",
    "/usr/local/bin/python3 -m venv $HOME/venv/jptr\n",
    "source $HOME/venv/jptr/bin/activate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1586f0d-4eea-4eb2-ba03-65f328cc2de3",
   "metadata": {},
   "source": [
    "Check python version with `python --version`\n",
    "\n",
    "Update and install modules with:"
   ]
  },
  {
   "cell_type": "raw",
   "id": "70261ca9-88fa-4a07-9ae3-0c495110a6e9",
   "metadata": {},
   "source": [
    "pip install --upgrade pip\n",
    "pip3 install jupyterlab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87a7971d-f7f9-4c06-8765-837922113877",
   "metadata": {},
   "source": [
    "And if we need to upgrade to a newer version"
   ]
  },
  {
   "cell_type": "raw",
   "id": "dfa23a5f-4000-43c5-b72a-5a8633f0c91b",
   "metadata": {},
   "source": [
    "pip install --upgrade jupyterlab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93a997b5-30fa-4e61-8d92-b02e63fc4ef3",
   "metadata": {},
   "source": [
    "#### Alternative with Conda\n",
    "\n",
    "Of course, you can still use Conda **instead** of venv.\n",
    "\n",
    "I followed these steps to a) create a new environment with conda, and b) install the appropriate Python modules and R packages."
   ]
  },
  {
   "cell_type": "raw",
   "id": "a9b3483d-342d-4929-80de-a7991be9d4ee",
   "metadata": {},
   "source": [
    "# Run this in a shell or terminal (bash or zsh):\n",
    "conda create --name jptr\n",
    "conda activate jptr\n",
    "conda install -c conda-forge r-rpostgresql r-readxl devtools\n",
    "conda install -c conda-forge jupyterlab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b90c3e2-acfd-4666-8d97-ac69ac967aac",
   "metadata": {},
   "source": [
    "### Installing python libraries\n",
    "Using the `venv` environment, it is easier to install packages with `pip`:"
   ]
  },
  {
   "cell_type": "raw",
   "id": "b1cac398-616c-45dd-9c10-9c64ac419912",
   "metadata": {},
   "source": [
    "pip install openpyxl pandas\n",
    "pip install pyprojroot\n",
    "# connection to aws s3\n",
    "pip install boto3\n",
    "# for iNat observations\n",
    "pip install pyinaturalist\n",
    "pip install pillow\n",
    "pip install ipyplot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f0036d1-4001-4989-af78-e5ed4f073780",
   "metadata": {},
   "source": [
    "### Adding the R kernel\n",
    "Activate the right R kernel for Jupyter lab with:"
   ]
  },
  {
   "cell_type": "raw",
   "id": "e0f6f0aa-22c3-4583-9b19-29ec216406a4",
   "metadata": {},
   "source": [
    "# Run this in an R session\n",
    "install.packages('IRkernel')\n",
    "IRkernel::installspec()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db2c83ce-7c16-46b2-8352-11047d63b3be",
   "metadata": {},
   "source": [
    "### Install R packages\n",
    "\n",
    "We need to install some packages in our R session for example `ozmaps`:\n",
    "```{r}\n",
    "install.packages(\"ozmaps\")\n",
    "```\n",
    "\n",
    "And `galah` from the Atlas of Living Australia\n",
    "\n",
    "```{r}\n",
    "#install.packages(\"remotes\")\n",
    "remotes::install_github(\"AtlasOfLivingAustralia/galah\")\n",
    "```\n",
    "\n",
    "Package `V.PhyloMaker` is available from this repo: <https://github.com/jinyizju/V.PhyloMaker>\n",
    "\n",
    "```{r}\n",
    "install.packages(\"devtools\")\n",
    "devtools::install_github(\"jinyizju/V.PhyloMaker\")\n",
    "install.packages(\"BiocManager\")\n",
    "BiocManager::install(\"ggtree\")\n",
    "install.packages(\"ggnewscale\")\n",
    "```\n",
    "\n",
    "For `treemapify`:\n",
    "\n",
    "```{r}\n",
    "install.packages(\"treemapify\")\n",
    "## devtools::install_github(\"wilkox/treemapify\")\n",
    "```\n",
    "\n",
    "The `rWCVP`is a companion R package for the World Checklist of Vascular Plants\n",
    "Citation: <https://doi.org/10.1111/nph.18919>\n",
    "\n",
    "```{r}\n",
    "install.packages(\"rWCVP\")\n",
    "remotes::install_github('matildabrown/rWCVPdata')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e59630d4-58e7-472f-9078-d56e86446533",
   "metadata": {},
   "source": [
    "### Version control with Jupyter\n",
    "\n",
    "There are some problems associated with version control of jupyter notebooks when copies of the notebook are edited in different sessions concurrently.\n",
    "\n",
    "Look at some recommendations here:\n",
    "<https://nextjournal.com/schmudde/how-to-version-control-jupyter>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "948336b7-9c9c-4c0a-8119-f8510ad9ec0d",
   "metadata": {},
   "source": [
    "## This is the end...\n",
    "\n",
    "... Of this short document.\n",
    "\n",
    "You are welcome to continue exploring the [links above](#Repository-structure), or:\n",
    "- continue navigating the repo on [GitHub](https://github.com/ces-unsw-edu-au/fireveg-analysis)\n",
    "- continue exploring the repo on [OSF](https://osf.io/h96q2/).\n",
    "- visit the database at <http://fireecologyplants.net>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e434b77-d2c7-4539-a2f0-cb2a16c2c4be",
   "metadata": {},
   "source": [
    "### Credits \n",
    "- <a href=\"https://www.flaticon.com/free-icons/open-data\" title=\"open data icons\">Open data icons created by Iconjam - Flaticon</a>\n",
    "- <a href=\"https://www.flaticon.com/free-icons/open-source\" title=\"open source icons\">Open source icons created by Smashicons - Flaticon</a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85c538ca-d4d6-4054-b1d9-cb99fbc3dec9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}