Add standardized/normalized data to catalog

README.md

@@ -14,7 +14,8 @@ Configuration files
   - rdavm : at CISL/RDA
   - remotetest : where? opendap/netcdf C test server
   - startup: this should be the minimal configuration for running a thredds server
-  - thredds : motherlode (/opt/tds-live) (idd + casestudies)
+  - thredds : atm-nwsc (/opt/tds) (idd + casestudies)
+  - threddsTest : atm-nwsc (/opt/tds-test) (idd + casestudies + preprocessed)
   - threddsDev : lead (/opt/tds-dev)
   - awsL2 : jetstream machine serving AWS Level2 radar archive data
 

threddsTest/build.info

@@ -0,0 +1,2 @@
+../wmsConfig.xml
+../idd

threddsTest/catalog.xml

@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
+         xmlns:xlink="http://www.w3.org/1999/xlink"
+         name="Unidata THREDDS Data Server"
+         version="1.0.6">
+
+  <dataset name="Realtime data from IDD">
+    <catalogRef xlink:href="idd/forecastModels.xml" xlink:title="Forecast Model Data" name=""/>
+    <catalogRef xlink:href="idd/forecastProdsAndAna.xml" xlink:title="Forecast Products and Analyses" name="" />
+    <catalogRef xlink:href="idd/radars.xml" xlink:title="Radar Data" name=""/>
+    <catalogRef xlink:href="idd/satellite.xml" xlink:title="Satellite Data" name=""/>
+    <catalogRef xlink:href="idd/textProds.xml" xlink:title="Text Products" name=""/>
+  </dataset>
+
+  <dataset name="Other Unidata Data">
+    <catalogRef xlink:href="casestudies/catalog.xml" xlink:title="Unidata case studies" name=""/>
+    <catalogRef xlink:href="preprocessed/catalog.xml" xlink:title="Preprocessed data" name=""/>
+  </dataset>
+</catalog>

threddsTest/notebooks/dataset_to_xarray_notebook.ipynb

@@ -0,0 +1,184 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<img src=\"https://unidata.ucar.edu/images/logos/badges/badge_unidata_100.jpg\" alt=\"Unidata Logo\" style=\"float: right; height: 98px;\">\n",
+    "\n",
+    "# Siphon THREDDS Jupyter Notebook - loading data into xarray\n",
+    "\n",
+    "## Dataset: {{datasetName}}\n",
+    "___"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Dependencies:\n",
+    "* *Siphon*: `pip install siphon`\n",
+    "* *xarray*: `pip install xarray` or 'conda install -c conda-forge xarray dask netCDF4 bottleneck'\n",
+    "\n",
+    "* *ipywidgets*:`pip install ipywidgets` or `conda install -c conda-forge ipywidgets`  \n",
+    "* enable *ipywidgets*:\n",
+    "    * using Juputer Notebooks: `jupyter nbextension enable --py widgetsnbextension`\n",
+    "    * using JupyterLab:\n",
+    "        * nodejs: `conda install nodejs`\n",
+    "        * `jupyter labextension install @jupyter-widgets/jupyterlab-manage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from siphon.catalog import TDSCatalog\n",
+    "import xarray as xr\n",
+    "\n",
+    "import ipywidgets as widgets"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "catUrl = \"{{catUrl}}\";\n",
+    "datasetName = \"{{datasetName}}\";"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Access a dataset\n",
+    "With the TDS catalog url, we can use Siphon to get the dataset named `datasetName`.\n",
+    "\n",
+    "The docs for Siphon are [here](https://unidata.github.io/siphon/latest/). "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "catalog = TDSCatalog(catUrl)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds = catalog.datasets[datasetName].remote_access(use_xarray=True)\n",
+    "print('ds is a:', type(ds))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## We can make a small widget to select the variable of interest"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "var_name = widgets.RadioButtons(\n",
+    "    options=list(ds.variables),\n",
+    "    description='Variable:')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "display(var_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### We now can make a new xarray data array with the variable of interest:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "new_da = ds[var_name.value]\n",
+    "new_da"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Further xarray turtorial can be found [here](https://tutorial.xarray.dev/overview/xarray-in-45-min.html) and documentation is [here](https://docs.xarray.dev/en/stable/). It is a very powerful package for multi-index (x,y,z,time, etc.) datasets!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# your code here!"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.9.13 ('ml_scratch')",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.13"
+  },
+ "viewer_info": {
+   "accepts": {
+    "accept_catalogs": ["Unidata THREDDS Data Server - NCEP models"]
+   },
+   "description": "TDS datasaets to xarray"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "3538842ee661b57f24cfeaeb491ecfd345bd2dd43fab6b8744c56a63a0fbbd62"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}