30_dgi_base_data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6b94b07d-c145-44ab-8566-1d5a793bf077",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Modify the file 00_setup.py to define input/output file paths on your system\n",
    "# The information in 00_setup.py will be used across notebooks\n",
    "from importlib.machinery import SourceFileLoader\n",
    "setup = SourceFileLoader(\"setup\", \"./00_setup.py\").load_module()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a58b5dd-6863-43f9-b28c-da9378afb1d7",
   "metadata": {},
   "source": [
    "# GNN - Data for Unsupervised Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20b91454-640d-4055-aac4-067fe1fe58b8",
   "metadata": {},
   "source": [
    "Create Stellargraph objects for unsupervised GNN runs.  The goal is to get embeddings for use in e.g. XGBoost models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c447a772-dc1a-4157-8d34-08e3f99ebfe6",
   "metadata": {},
   "source": [
    "Here I use raw training features, with no label feature information"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc0f8895-68a0-435b-b427-04dd1fca5466",
   "metadata": {},
   "source": [
    "The unsupervised model has both business and NAICS nodes, and uses NAICS nodes as the head type."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66e51bbc-1184-4389-a31a-c130109b3b16",
   "metadata": {},
   "source": [
    "This script is nearly identical to # 20, but I reproduce code here as I may change things in one type of model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce6fed7e-3548-4bee-8d57-9244be1222b4",
   "metadata": {},
   "source": [
    "*This script takes about 15 minutes on my MacBook Air*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "52c010c9-235e-4845-bc23-dd9b82acc741",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dd932005-3f42-434a-a7e3-34d6fdab71b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import importlib\n",
    "import pickle, re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3c0a0d65-40df-4f60-a5ba-5a19d817765c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sba_gnn.sba_gnn import sg_gnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "59ba58e0-e2a1-4b3d-b125-d77a463fab1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-16 13:51:07.553185: I metal_plugin/src/device/metal_device.cc:1154] Metal device set to: Apple M1\n",
      "2024-03-16 13:51:07.553210: I metal_plugin/src/device/metal_device.cc:296] systemMemory: 16.00 GB\n",
      "2024-03-16 13:51:07.553218: I metal_plugin/src/device/metal_device.cc:313] maxCacheSize: 5.33 GB\n",
      "2024-03-16 13:51:07.553782: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:303] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.\n",
      "2024-03-16 13:51:07.554078: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:269] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)\n"
     ]
    }
   ],
   "source": [
    "import stellargraph as sg\n",
    "from stellargraph import StellarGraph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da4b71d-5108-49e6-80e1-521c23def729",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Import Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8b7ce228-82c7-4943-930d-72a171b48195",
   "metadata": {},
   "outputs": [],
   "source": [
    "## NN scaled input data\n",
    "comb_df = pd.read_parquet(Path(setup.temp_path).joinpath('20_DATA_combined_scaled_all.parquet'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "92b9e3ec-b772-4c70-9cf8-c88186ac110f",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(Path(setup.temp_path).joinpath('20_DATA_features.pkl'), 'rb') as fin:\n",
    "    imputed_features = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3ca13685-f529-45e7-8d3c-c4f7f23b972d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['NoEmp', 'CreateJob', 'LowDoc', 'DisbursementGross', 'new_business', 'urban_flag', 'franchise_flag', 'missingindicator_LowDoc', 'missingindicator_new_business', 'missingindicator_urban_flag']\n"
     ]
    }
   ],
   "source": [
    "num_feat =  [c for c in imputed_features if 'NAICS' not in c]\n",
    "print(num_feat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba09160d-4623-4829-8d9e-00f551305372",
   "metadata": {},
   "source": [
    "## Create Edge Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbc1aef4-827e-4ca4-989b-f11156a087a6",
   "metadata": {},
   "source": [
    "I will have edges between a business and its NAICS, as well as edges to same-sector NAICS nodes.  This section creates general edge data, which may be filtered to create graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10e564f2-1f31-4a8b-8583-f2e7adb81cd5",
   "metadata": {},
   "source": [
    "##### Get edges from businesses to their exact NAICS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "93a337e4-5f29-4221-b8e4-cb9fb3da457a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edges from businesses to their NAICS code\n",
    "naics_info_df = comb_df[['LoanNr_ChkDgt', 'NAICS_orig', 'dset']].copy() \\\n",
    "    .rename(columns={'LoanNr_ChkDgt':'source'}) \n",
    "naics_info_df['target'] = sg_gnn.get_naics_index(naics_info_df['NAICS_orig'])\n",
    "\n",
    "edges_all = naics_info_df[['source', 'target', 'dset']].copy()\n",
    "edges_all['type'] = 'loan_naics'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dcbba7bf-efc6-4be6-9d90-653cd3392e5e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "type\n",
       "loan_naics    688081\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "edges_all['type'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "909273ae-0265-4adb-99cf-15f88198585d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(688081, 2)\n",
      "(688081, 4)\n"
     ]
    }
   ],
   "source": [
    "print(edges_all[['source', 'target']].drop_duplicates().shape)\n",
    "print(edges_all.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8eac7471-cda3-40fd-a543-21e2fd3fcfda",
   "metadata": {},
   "outputs": [],
   "source": [
    "edges_all.to_parquet(Path(setup.temp_path).joinpath('30_DATA_graph_edges.parquet'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eb23221-aaec-4aa5-b19c-86441375b4fc",
   "metadata": {},
   "source": [
    "## Get Node Features\n",
    "Use the scaled NN data.  For NAICS, just set an indicator for now.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4d999dba-b0b6-4969-9ef9-a5dd2615d7d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "features_business = comb_df[['LoanNr_ChkDgt', 'dset', 'NAICS_orig'] + num_feat] \\\n",
    "    .set_index('LoanNr_ChkDgt')\n",
    "features_business.to_parquet(Path(setup.temp_path).joinpath('30_DATA_graph_node_features_loans.parquet'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8c380d15-a731-42ff-a7f9-c1e4b535ce59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1311, 2)\n"
     ]
    }
   ],
   "source": [
    "# NAICS features - Just use indicator.  Include original code for mapping\n",
    "features_naics = comb_df[['NAICS_orig']].drop_duplicates()\n",
    "features_naics['feat'] = 1\n",
    "features_naics['source'] = sg_gnn.get_naics_index(features_naics['NAICS_orig'])\n",
    "features_naics.set_index('source', inplace=True) \n",
    "print(features_naics.shape)\n",
    "features_naics.to_parquet(Path(setup.temp_path).joinpath('30_DATA_graph_node_features_naics.parquet'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95cf8304-a674-49d4-b873-b56e9bf53f42",
   "metadata": {},
   "source": [
    "## Stellargraph - train and validation\n",
    "Graph without test nodes (make sure set-aside NAICS not included)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1dc0aa32-b79a-46f9-881b-7499fa3d1977",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get only relevant edges and node data\n",
    "edges_train_val, features_business_train_val, features_naics_train_val = \\\n",
    "    sg_gnn.limit_data(edges_all, features_business, features_naics.drop(columns='NAICS_orig'),\n",
    "                     ['train', 'val'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f509751c-45d8-4f2e-ae64-47985af49eb4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train+val graph data\n",
      "business features, start rows 688081, end 529574\n",
      "naics features, start rows 1311, end 1173\n",
      "edges, start rows 688081, end 529574\n"
     ]
    }
   ],
   "source": [
    "print(f'Train+val graph data')\n",
    "print(f'business features, start rows {features_business.shape[0]}, end {features_business_train_val.shape[0]}')\n",
    "print(f'naics features, start rows {features_naics.shape[0]}, end {features_naics_train_val.shape[0]}')\n",
    "print(f'edges, start rows {edges_all.shape[0]}, end {edges_train_val.shape[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8dbf7ca3-42f9-4a4b-bfdf-74d37aad9a16",
   "metadata": {},
   "outputs": [],
   "source": [
    "sba_graph_train_val = StellarGraph({'LoanNr_ChkDgt':features_business_train_val.drop(columns='NAICS_orig'),\n",
    "                                    'NAICS': features_naics_train_val},\n",
    "                                   edges_train_val, \n",
    "                                   source_column=\"source\", target_column=\"target\",\n",
    "                                   edge_type_column=\"type\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "410b7928-83b4-428a-af5b-05ecb9c7a748",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 530747, Edges: 529574\n",
      "\n",
      " Node types:\n",
      "  LoanNr_ChkDgt: [529574]\n",
      "    Features: float32 vector, length 10\n",
      "    Edge types: LoanNr_ChkDgt-loan_naics->NAICS\n",
      "  NAICS: [1173]\n",
      "    Features: float32 vector, length 1\n",
      "    Edge types: NAICS-loan_naics->LoanNr_ChkDgt\n",
      "\n",
      " Edge types:\n",
      "    LoanNr_ChkDgt-loan_naics->NAICS: [529574]\n",
      "        Weights: all 1 (default)\n",
      "        Features: none\n"
     ]
    }
   ],
   "source": [
    "print(sba_graph_train_val.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "99b1b627-0cc0-492d-91d3-f9d92e2136e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save stellargraph object\n",
    "with open(Path(setup.temp_path).joinpath('30_DATA_stellargraph_train_val.pkl'), 'wb') as fout:\n",
    "      pickle.dump(sba_graph_train_val, fout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d31508f8-f487-4e56-b2a4-7bd5da425e2e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['NAICS_orig', 'feat'], dtype='object')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_naics.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e82b9311-7f4e-4cab-92a7-633670a0398f",
   "metadata": {},
   "source": [
    "## Stellargraph - all nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2fcf66ec-e079-4c91-a0cb-70439013577d",
   "metadata": {},
   "outputs": [],
   "source": [
    "sba_graph_all = StellarGraph({'LoanNr_ChkDgt':features_business.drop(columns=['dset', 'NAICS_orig']) ,\n",
    "                              'NAICS': features_naics.drop(columns='NAICS_orig')},\n",
    "                             edges_all.drop(columns=['dset']).drop_duplicates(),\n",
    "                             source_column=\"source\", target_column=\"target\",\n",
    "                             edge_type_column=\"type\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "63089961-2b1f-4312-aafe-e2e7de4feede",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 689392, Edges: 688081\n",
      "\n",
      " Node types:\n",
      "  LoanNr_ChkDgt: [688081]\n",
      "    Features: float32 vector, length 10\n",
      "    Edge types: LoanNr_ChkDgt-loan_naics->NAICS\n",
      "  NAICS: [1311]\n",
      "    Features: float32 vector, length 1\n",
      "    Edge types: NAICS-loan_naics->LoanNr_ChkDgt\n",
      "\n",
      " Edge types:\n",
      "    LoanNr_ChkDgt-loan_naics->NAICS: [688081]\n",
      "        Weights: all 1 (default)\n",
      "        Features: none\n"
     ]
    }
   ],
   "source": [
    "print(sba_graph_all.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "18015753-3c65-4ffd-af45-00fa3e4a2427",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save stellargraph object\n",
    "with open(Path(setup.temp_path).joinpath('30_DATA_stellargraph_all.pkl'), 'wb') as fout:\n",
    "      pickle.dump(sba_graph_all, fout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e5ca7fb-3a51-4e31-9c6f-9b7beb016f7d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf_p39",
   "language": "python",
   "name": "tf_p39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}