Useful-Notes

Command Line Promps

Update all Python packages with pip

• pip list --outdated --format=freeze | grep -v '^-e' | cut -d = -f 1 | xargs -n1 pip install -U
  or
• pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U

Install package without installing the dependencies

• pip install -U --no-deps mypackage

ssh USER@hadoop.rcc.uchicago.edu login to RCC, then enter password
ls list all directories
mkdir newdir makes new directory named newdir
rm -rf newdir deletes directory named newdir
clear clears screen
cd newdir goes into newdir
cd goes to home
pwd to see which file you're in
rm file to remove 'file'
ssh user@host connect to host as user
ctrl+c halt current command
crtl+z stop current command
ctrl+d log out of current session
ctrl+w erases one word in current line
ctrl+u erases whole line
!! repeat last command
exit log out of current session
wget <url> download file directly
mv orignalFile name newFileName to change file name
tar zxvf instacart.tar.gz unzip file
hadoop fs -put /home/$USER/data/instacart /user/$USER/instacart to put it to hadoop

Classes

• Harvard Introduction to Computer Science
• Notes that accompany the Stanford CS class: Convolutional Neural Networks for Visual Recognition

Tips, Tutorials, How-tos, Visualizations

• Plotting and Graphing
  • Top 5 tricks to make plots look better
  • Plotting with categorical data with Seaborn
  • Sankey Diagram in Python
• Best Practices
  • The Best of the Best Practices (BOBP) Guide for Python
  • Best practices for file naming
• Principal Component Analysis
  • Eigenvectors and Eigenvalues explained
  • Principal Component Analysis explained
• Clustering
  • In Depth: k-Means Clustering
  • How to Use t-SNE
• Validation
  • Simple guide to confusion matrix terminology
  • Sensitivity and specificity
  • Understanding ROC curves
  • Understanding the Bias-Variance Tradeoff
• Ensemble Methods
  • Bagging and Boosting
  • Implementation of a majority voting EnsembleVoteClassifier for classification
• Gradient Descent Optimisation
  • 10 Gradient Descent Optimisation Algorithms
  • Momentum gradient descent
  • An overview of gradient descent optimization algorithms
• Neural Network/Deep Learning Overview and Training
  • How to build a simple artificial neural network
  • Visualizing how a Neural Network works
  • Setting hyperparameters for Neural Networks
  • Training Deep Learning models entirely in your browser
  • An intuitive approach to Backpropagation
  • Priming neural networks with an appropriate TensorFlow initializer
• Convolutional Neural Networks and Images
  • Good explainers here and here
  • Feature Visualization how neural networks build up their understanding of images
  • Combine the content of one image with the style of another image and another Photo Style Transfer and [another example](https://github.com/jcjohnson/neural-style/
  • CNN example with MNIST dataset
  • Convolution animations
  • Convolutional Neural Networks for Visual Recognition
  • ResNet, AlexNet, VGGNet, Inception: Understanding various architectures of Convolutional Networks
  • An Overview of ResNet and its Variants
  • What is wrong with Convolutional neural networks?
  • Convolutional Neural Networks
  • Understanding layers in NN from google AI blog
  • ImageNet Classification with Deep Convolutional Neural Networks
  • Pretrained Image Classifier here and here
  • DeepFace: Closing the Gap to Human-Level Performance in Face Verification
  • OpenFace: A general-purpose face recognition library
  • Convolutions and Backpropagations
• Transfer Learning
  • How transferable are features in deep neural networks?
  • Transfer Learning stanford notes
  • A Gentle Introduction to Transfer Learning for Deep Learning
• Natural Language Processing
  • Word Embeddings Projected
  • Word Embeddings
  • GloVe: Global Vectors for Word Representation
  • Word2Vec For Phrases — Learning Embeddings For More Than One Word
  • Learning Word Embeddings
  • Word2Vec For Phrases
  • Word embeddings in tensorflow
• Recommender Systems
  • 5 Types of Recommender Systems
  • How Shopify Uses Recommender Systems to Empower Entrepreneurs
  • Various Implementations of Collaborative Filtering
  • Collaborative filtering for movie recommendations
  • Deep learning for recommender systems
  • Association Rules and the Apriori Algorithm
• Recurrent Neural Networks and Long Short-Term Memory (LSTM)
  • Auto-Generating Clickbait With Recurrent Neural Networks
  • Recurrent Neural Networks
  • LSTM by Example using Tensorflow
  • Understanding, Deriving and Extending the LSTM
• Blockchain
  • A Practical Introduction to Blockchain with Python
  • Brief overview of Blockchain
  • Blockchain summary (devopedia)
• Big Data
  • Differences between Hive Internal and External Tables
  • Predicting Breast Cancer Using Apache Spark Machine Learning Logistic Regression
  • MapReduce processes explained but also explained here
  • Apache YARN (Yet Another Resource Negotiator): Hadoop’s cluster resource management system
  • Wide VS Narrow Dependecies: Representation/DAG of what Spark analyzes to do optimizations
• Time Series
  • ARIMA vs. LSTM slides
• Miscellaneous
  • Anscombe's quartet
  • Kernel Density Estimation
  • Glossary of ML Terms
  • Image Caption Generation with CNN + RNN
  • Human pose estimation
  • Chris Olah Blog on ML/DL topics
  • How to apply standardization/normalization to train and test

Topics Explained or Visualized

• 

Textbooks

• R for Data Science textbook
• Intro to Stats
• MIT Deep Learning
• O'Reilly book Hands-on Machine Learning with Scikit-Learn and TensorFlow

Datasets

• Pre-trained models: Model Zoo for transfer learning
• Chicago Data
• India Data
• Airline Dataset
• List of Public Data Sources Fit for Machine Learning
• Million Song Dataset
• Classification datasets
• 2013-2016 Cleaned & Parsed 10-K Filings with the SEC
• Stanford Large Network Dataset Collection
• Google Dataset Search
• Fashion MNIST with 60k train and 10k test images
• notMNIST dataset Similar to MNIST but used different fonts
• The CIFAR-10 dataset 10 classes with 6k images per class
• Large Scale Visual Recognition Challenge 2012 and 2014

Spatial Datasets/Sources

Plotting in R

• Useful resources for tmap
• Code for tmap: Thematic Maps in R
• Making maps with R

Learning

• GeoDa Data Portal mostly smaller sample datasets for use in learning spatial analysis
• spData R package accompanies Geocomputation with R
• geodaData R package currently in development, but includes all the packages in the GeoDa tutorials
• Tidycensus R package useful for getting US boundaries, variables

Social Science/Planning

• OpenStreetMap can use API or osmdata R package to get data CMAP Data Hub: Chicago Metropolitan Agency for Planning (36 datasets, including community areas)
• NYU Spatial Data Repository lots of data and links to other institutions
• ICPSR data portal data archive for the social sciences

Land Use/Ecological

• USGS Data Products
• NEON Ecological data

Health/Demographics

• CDC GIS Data also has ArcGIS training
• Demographic and Health Survey / USAID Spatial Data Repository

Historical

• Library of Congress
• UChicago Maps and librarians can help make into shapefiles

Others

• GeoDa Datasets on Github. In R:

  install.packages('remotes')    
  remotes::install_github("spatialanalysis/geodaData")    
  library(geodaData)    
  data("chicago_comm")   
  

• Search on StackExchange here and here