4_DC_Keras_PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0d232e43",
   "metadata": {},
   "source": [
    "# DeepChem\n",
    "\n",
    "- [Installation](https://github.com/deepchem/deepchem#installation)\n",
    "- [Tutorial](https://deepchem.readthedocs.io/en/latest/get_started/tutorials.html)\n",
    "- [Sample Notebooks](https://github.com/deepchem/deepchem/tree/master/examples/tutorials) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c74d796",
   "metadata": {},
   "source": [
    "DeepChem has implementation of several different graph models. \n",
    "\n",
    "Some of the models are implemented by Keras and others are implemented by PyTorch. \n",
    "\n",
    "We compare several Keras & PyTorch models. \n",
    "\n",
    "\n",
    "**Additional Dependencies:**\n",
    "\n",
    "Weave Model requires \"TensorFlow Probability\" library:\n",
    "- https://www.tensorflow.org/probability\n",
    " \n",
    " \n",
    "PyTorch models require \"DGL\" and \"DGL LifeSci\" to be installed:\n",
    "- https://www.dgl.ai/\n",
    "- https://github.com/awslabs/dgl-lifesci\n",
    "\n",
    "\n",
    "**Documentation:**\n",
    "\n",
    "https://deepchem.readthedocs.io/en/latest/api_reference/models.html#keras-models\n",
    "\n",
    "https://deepchem.readthedocs.io/en/latest/api_reference/models.html#pytorch-models\n",
    "\n",
    "\n",
    "**Note:**\n",
    "\n",
    "https://deepchem.readthedocs.io/en/latest/api_reference/featurizers.html#graph-convolution-featurizers\n",
    "\n",
    "Keras & PyTorch models use different featurizers. Also, some models have their own specific featurizer.\n",
    "In summary:\n",
    "\n",
    "- ConvMolFeaturizer and WeaveFeaturizer are used with graph convolution models which inherited Keras Model. \n",
    "\n",
    "- ConvMolFeaturizer is used with graph convolution models except WeaveModel. WeaveFeaturizer are only used with WeaveModel. \n",
    "\n",
    "- MolGraphConvFeaturizer is used with graph convolution models which inherited TorchModel. \n",
    "\n",
    "\n",
    "**Important:**\n",
    "\n",
    "Before model fitting, test all graph featurizers on all data to be sure that featurizers work on all daat.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3a39c9c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import deepchem as dc\n",
    "\n",
    "from sklearn.metrics import r2_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8480526",
   "metadata": {},
   "source": [
    "# Parameters , Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "39e79c12",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_filepath = './data/ESOL.csv'   \n",
    "\n",
    "model_dir = './result'   # folder to save fitted model\n",
    "\n",
    "n_tasks = 1   # No. of tasks (No. of dependent variables)\n",
    "\n",
    "nb_epoch = 100   # No. of epochs\n",
    "\n",
    "# Initialze the metrics\n",
    "# https://deepchem.readthedocs.io/en/latest/api_reference/metrics.html\n",
    "metric_r2 = dc.metrics.Metric(dc.metrics.r2_score)\n",
    "metric_mse = dc.metrics.Metric(dc.metrics.mean_squared_error)\n",
    "metrics = [metric_r2, metric_mse]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06ad72b6",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fc5ac075",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(data_filepath)   \n",
    "smiles = data['smiles']   # should be be 1D\n",
    "y = data['measured log solubility in mols per litre']   # can be 1D or 2D"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88092412",
   "metadata": {},
   "source": [
    "# Keras Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec39c60a",
   "metadata": {},
   "source": [
    "# GraphConvModel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38917a45",
   "metadata": {},
   "source": [
    "### Make Dataset by ConvMolFeaturizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a44c0116",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "deepchem.feat.mol_graphs.ConvMol"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "''' ConvMolFeaturizer '''\n",
    "# Duvenaud graph convolutions ; Can be used with Keras models\n",
    "# https://deepchem.readthedocs.io/en/latest/api_reference/featurizers.html#convmolfeaturizer\n",
    "\n",
    "featurizer = dc.feat.ConvMolFeaturizer(use_chirality=False, per_atom_fragmentation=False)\n",
    "features = featurizer.featurize(smiles)   # numpy array, it returns a graph object for every molecule\n",
    "type(features[0])   # deepchem.feat.mol_graphs.ConvMol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b13ab1c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<NumpyDataset X.shape: (1128,), y.shape: (1128,), w.shape: (1128,), task_names: [0]>\n"
     ]
    }
   ],
   "source": [
    "dataset = dc.data.NumpyDataset(X=features, y=np.array(y))\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "54355195",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<NumpyDataset X.shape: (226,), y.shape: (226,), w.shape: (226,), ids: [148 1014 1021 ... 835 559 684], task_names: [0]>\n"
     ]
    }
   ],
   "source": [
    "# Split Dataset\n",
    "splitter = dc.splits.RandomSplitter()\n",
    "train_dataset, test_dataset = splitter.train_test_split(dataset=dataset, frac_train=0.8, seed=0)\n",
    "print(test_dataset)\n",
    "\n",
    "y_test = test_dataset.y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "662c95c1",
   "metadata": {},
   "source": [
    "### Model , Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9a520eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://deepchem.readthedocs.io/en/latest/api_reference/models.html#graphconvmodel\n",
    "# This Class uses Keras models\n",
    "model = dc.models.GraphConvModel(n_tasks=n_tasks,              # No. of tasks \n",
    "                                 mode='regression',            # Either “classification” or “regression”\n",
    "                                 batch_size=100,               # Batch size for training and evaluating\n",
    "                                 learning_rate=0.001,\n",
    "                                 dropout=0.2,                  # Dropout probablity to use for each layer. The length of this list should equal len(graph_conv_layers)+1 (one value for each convolution layer, and one for the dense layer). Alternatively this may be a single value instead of a list, in which case the same value is used for every layer.\n",
    "                                 graph_conv_layers=[64, 64],   # Width of channels for the Graph Convolution Layers\n",
    "                                 dense_layer_size=128          # Width of channels for Atom Level Dense Layer before GraphPool\n",
    "                                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d4b36937",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_14:0\", shape=(353,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_13:0\", shape=(353, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_17:0\", shape=(1314,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_16:0\", shape=(1314, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_20:0\", shape=(1149,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_19:0\", shape=(1149, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_23:0\", shape=(132,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_22:0\", shape=(132, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_11:0\", shape=(353,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_10:0\", shape=(353, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_13:0\", shape=(1314,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_12:0\", shape=(1314, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_1:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_15:0\", shape=(1149,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_14:0\", shape=(1149, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_2:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_17:0\", shape=(132,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_16:0\", shape=(132, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_3:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_19:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_18:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_21:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_20:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_23:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_22:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_25:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_24:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_27:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_26:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_8:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_29:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_28:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_9:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_14:0\", shape=(353,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_13:0\", shape=(353, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_17:0\", shape=(1314,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_16:0\", shape=(1314, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_20:0\", shape=(1149,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_19:0\", shape=(1149, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_23:0\", shape=(132,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_22:0\", shape=(132, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_14:0\", shape=(349,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_13:0\", shape=(349, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_17:0\", shape=(1264,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_16:0\", shape=(1264, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_20:0\", shape=(1131,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_19:0\", shape=(1131, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_23:0\", shape=(140,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_22:0\", shape=(140, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_11:0\", shape=(349,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_10:0\", shape=(349, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_13:0\", shape=(1264,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_12:0\", shape=(1264, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_1:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_15:0\", shape=(1131,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_14:0\", shape=(1131, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_2:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_17:0\", shape=(140,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_16:0\", shape=(140, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_3:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_14:0\", shape=(349,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_13:0\", shape=(349, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_17:0\", shape=(1264,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_16:0\", shape=(1264, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_20:0\", shape=(1131,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_19:0\", shape=(1131, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_23:0\", shape=(140,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_22:0\", shape=(140, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_14:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_13:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_17:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_16:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_20:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_19:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_23:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Reshape_22:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_3/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_11:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_10:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_13:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_12:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_1:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_15:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_14:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_2:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_17:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Reshape_16:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_conv_3/Cast_3:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_14:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_13:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_17:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_16:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_20:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_19:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/tensorflow/python/framework/indexed_slices.py:449: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_23:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Reshape_22:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model_1/graph_pool_2/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    }
   ],
   "source": [
    "loss_avg = model.fit(train_dataset, nb_epoch=nb_epoch)\n",
    "loss_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0e6ff79c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: {'r2_score': 0.7699905728761282, 'mean_squared_error': 1.0149233396525408}\n",
      "Test set score: {'r2_score': 0.7233671241375563, 'mean_squared_error': 1.1911568851754661}\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set score:\", model.evaluate(train_dataset, metrics))\n",
    "print(\"Test set score:\", model.evaluate(test_dataset, metrics))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dc76aafb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7233671241375563"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(test_dataset)   # numpy array [n, 1]\n",
    "r2_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "845c31be",
   "metadata": {},
   "source": [
    "# WeaveModel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1a62bee",
   "metadata": {},
   "source": [
    "### Make Dataset by WeaveFeaturizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "288dc65e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "deepchem.feat.mol_graphs.WeaveMol"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "''' WeaveFeaturizer '''\n",
    "# Weave convolutions ; Can be used with WeaveModel (WeaveModel has its own featurizer)\n",
    "# Compared to \"ConvMolFeaturizer\", it has extra descriptors and may provide for additional descriptive power but at the cost of a larger featurized dataset.\n",
    "# https://deepchem.readthedocs.io/en/latest/api_reference/featurizers.html#weavefeaturizer\n",
    "\n",
    "# \"graph_distance\" : If True, use graph distance for distance features. Otherwise, use Euclidean distance. Note that this means that molecules that this featurizer is invoked on must have valid conformer information if this option is set.\n",
    "featurizer = dc.feat.WeaveFeaturizer(graph_distance=True)\n",
    "features = featurizer.featurize(smiles)\n",
    "type(features[0])   # deepchem.feat.mol_graphs.WeaveMol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "493ebc87",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<NumpyDataset X.shape: (1128,), y.shape: (1128,), w.shape: (1128,), task_names: [0]>\n"
     ]
    }
   ],
   "source": [
    "dataset = dc.data.NumpyDataset(X=features, y=np.array(y))\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "90466f81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<NumpyDataset X.shape: (226,), y.shape: (226,), w.shape: (226,), ids: [148 1014 1021 ... 835 559 684], task_names: [0]>\n"
     ]
    }
   ],
   "source": [
    "# Split Dataset\n",
    "splitter = dc.splits.RandomSplitter()\n",
    "train_dataset, test_dataset = splitter.train_test_split(dataset=dataset, frac_train=0.8, seed=0)\n",
    "print(test_dataset)\n",
    "\n",
    "y_test = test_dataset.y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a97b662c",
   "metadata": {},
   "source": [
    "### Model , Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d513b248",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://deepchem.readthedocs.io/en/latest/api_reference/models.html#weavemodel\n",
    "# This model implements the Weave style graph convolution. This Class uses Keras models.\n",
    "# Weave model has different architectures. The default settings in this class correspond to the W2N2 variant which is the most commonly used variant.\n",
    "# This model cannot compute uncertainties\n",
    "\n",
    "del model\n",
    "\n",
    "model = dc.models.WeaveModel(n_tasks=n_tasks,              # No. of tasks \n",
    "                             mode='regression',            # Either “classification” or “regression”\n",
    "                             batch_size=100,               # Batch size for training and evaluating\n",
    "                             learning_rate=0.001,\n",
    "                             dropout=0.25,                 # Dropout probablity to use for each fully connected layer. Default value is 0.25. Name of parameter is 'dropouts'. For other models is called 'dropout'. If you use 'dropout' here, still works.       \n",
    "                             n_weave=2,                    # No. of weave layers\n",
    "                             fully_connected_layer_sizes=[2000, 100],   # Size of each dense layer in the network. The length of this list determines the number of layers.                                 \n",
    "                             batch_normalize=False         # Use of batch normalization can cause issues with NaNs. If you’re having trouble with NaNs while using this model, consider setting batch_normalize=False.\n",
    "                             )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a011fcaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0'] when minimizing the loss.\n",
      "WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0'] when minimizing the loss.\n",
      "WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0'] when minimizing the loss.\n",
      "WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0'] when minimizing the loss.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7614833831787109"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slower than GraphConvModel\n",
    "model.fit(train_dataset, nb_epoch=nb_epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "30e6c070",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: {'r2_score': 0.9269669655017868, 'mean_squared_error': 0.322260405604883}\n",
      "Test set score: {'r2_score': 0.9001402970449006, 'mean_squared_error': 0.4299871168808256}\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set score:\", model.evaluate(train_dataset, metrics))\n",
    "print(\"Test set score:\", model.evaluate(test_dataset, metrics))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6783973b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9001402970449006"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(test_dataset)   # numpy array [n, 1]\n",
    "r2_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd2e1503",
   "metadata": {},
   "source": [
    "# PyTorch Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5728c81",
   "metadata": {},
   "source": [
    "# GAT Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd06877d",
   "metadata": {},
   "source": [
    "### Make Dataset by MolGraphConvFeaturizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "01f918bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to featurize datapoint 934, C. Appending empty array\n",
      "Exception message: zero-size array to reduction operation maximum which has no identity\n",
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/numpy/core/_asarray.py:102: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
      "  return array(a, dtype, copy=False, order=order)\n"
     ]
    }
   ],
   "source": [
    "''' MolGraphConvFeaturizer '''\n",
    "# General graph convolution networks for molecules , Can be used with PyTorch models\n",
    "# https://deepchem.readthedocs.io/en/latest/api_reference/featurizers.html#molgraphconvfeaturizer\n",
    "# Note: If SMILES is only one atom and you set \"use_edges=True\", it gives error \"Failed to featurize datapoint\".\n",
    "# Some PyTorch models require \"use_edges=True\". Therefore, we set it as True. It means that we use both node features and edge features. \n",
    "featurizer = dc.feat.MolGraphConvFeaturizer(use_edges=True)\n",
    "features = featurizer.featurize(smiles)   # numpy array, it returns a graph object for every molecule"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cb558ea",
   "metadata": {},
   "source": [
    "There is a warning about datapoint #934. We remove this observation from dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ec714717",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([], dtype=float64)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features[934]   # empty array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "45f4201f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "deepchem.feat.graph_data.GraphData"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(features[0])   # deepchem.feat.graph_data.GraphData"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6e64315e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1127, 1)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Warning (Failed to featurize datapoint 934, C. Appending empty array)\n",
    "# Remove this feature\n",
    "features = np.delete(features, 934)\n",
    "\n",
    "# Also we need to remove from y. We reshape y into 2D to avoid another warning when fitting model (Using a target size (torch.Size([100])) that is different to the input size (torch.Size([100, 1])). This will likely lead to incorrect results due to broadcasting.)\n",
    "# Some PyTorch models require y as a 2D array\n",
    "y = np.array(y)\n",
    "y = np.delete(y, 934).reshape(-1,1)\n",
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "051bb254",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<NumpyDataset X.shape: (1127,), y.shape: (1127, 1), w.shape: (1127, 1), task_names: [0]>\n"
     ]
    }
   ],
   "source": [
    "dataset = dc.data.NumpyDataset(X=features, y=y)\n",
    "y_test = test_dataset.y\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7ec14d78",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split Dataset (splitter object is already initialized)\n",
    "train_dataset, test_dataset = splitter.train_test_split(dataset=dataset, frac_train=0.8, seed=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06db1454",
   "metadata": {},
   "source": [
    "### Model , Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "83ce88e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using backend: pytorch\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GATModel(activation=None, alpha=None, dropout=None, mode=None,\n",
      "         n_attention_heads=None, n_classes=None, n_tasks=None,\n",
      "         number_atom_features=None, predictor_dropout=None,\n",
      "         predictor_hidden_feats=None, residual=None, self_loop=None)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/GNN/lib/python3.7/site-packages/sklearn/base.py:213: FutureWarning: From version 0.24, get_params will raise an AttributeError if a parameter cannot be retrieved as an instance attribute. Previously it would return None.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# https://deepchem.readthedocs.io/en/latest/api_reference/models.html#gatmodel\n",
    "# Model for Graph Property Prediction Based on Graph Attention Networks (GAT)\n",
    "# It works with both \"use_edges=True or False\" (Parameter of MolGraphConvFeaturizer)\n",
    "# This model cannot compute uncertainties\n",
    "\n",
    "del model\n",
    "\n",
    "model = dc.models.GATModel(n_tasks=n_tasks,              # No. of tasks \n",
    "                           mode='regression',            # Either “classification” or “regression”\n",
    "                           batch_size=100,               # Batch size for training and evaluating\n",
    "                           learning_rate=0.001,\n",
    "                           dropout=0                     # Dropout probability within each GAT layer\n",
    "                           )\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "04b51818",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://deepchem.readthedocs.io/en/latest/api_reference/models.html#pytorch-models\n",
    "loss_avg = model.fit(train_dataset, nb_epoch=nb_epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "01fb580c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: {'r2_score': 0.9435374576239887, 'mean_squared_error': 0.25392900477060387}\n",
      "Test set score: {'r2_score': 0.8751975961613548, 'mean_squared_error': 0.49507168892227976}\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set score:\", model.evaluate(train_dataset, metrics))\n",
    "print(\"Test set score:\", model.evaluate(test_dataset, metrics))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "4649dbff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8751975961613548"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(test_dataset)   # numpy array [n, 1]\n",
    "r2_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edcc0bb5",
   "metadata": {},
   "source": [
    "# GCN Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea629082",
   "metadata": {},
   "source": [
    "### Make Dataset by MolGraphConvFeaturizer\n",
    "\n",
    "We use dataset prepared for GAT model because featurizer is the same."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081b0771",
   "metadata": {},
   "source": [
    "### Model , Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "3f5de294",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://deepchem.readthedocs.io/en/latest/api_reference/models.html#gcnmodel\n",
    "# Model for Graph Property Prediction Based on Graph Convolution Networks (GCN)\n",
    "# This model is different from deepchem.models.GraphConvModel\n",
    "# It works with both \"use_edges=True or False\" (Parameter of MolGraphConvFeaturizer)\n",
    "# This model cannot compute uncertainties\n",
    "\n",
    "del model\n",
    "\n",
    "model = dc.models.GCNModel(n_tasks=n_tasks,              # No. of tasks \n",
    "                           mode='regression',            # Either “classification” or “regression”\n",
    "                           batch_size=100,               # Batch size for training and evaluating\n",
    "                           learning_rate=0.001,\n",
    "                           graph_conv_layers=[64, 64],   # Width of channels for GCN layers\n",
    "                           dropout=0.1,                  # Dropout probability for the output of each GCN layer\n",
    "                           predictor_dropout=0.1,        # Dropout probability in the output MLP predictor\n",
    "                           batchnorm=False               # Whether to apply batch normalization to the output of each GCN layer\n",
    "                           )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "c12f4eb4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.38655326843261717"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_avg = model.fit(train_dataset, nb_epoch=nb_epoch)\n",
    "loss_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "d4de2807",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: {'r2_score': 0.8913996043320103, 'mean_squared_error': 0.48840858433223366}\n",
      "Test set score: {'r2_score': 0.8709806313139318, 'mean_squared_error': 0.511799731371196}\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set score:\", model.evaluate(train_dataset, metrics))\n",
    "print(\"Test set score:\", model.evaluate(test_dataset, metrics))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "e4b7d773",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8709806313139318"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(test_dataset)   # numpy array [n, 1]\n",
    "r2_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0519d0b",
   "metadata": {},
   "source": [
    "# AttentiveFPModel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b113faa",
   "metadata": {},
   "source": [
    "### Make Dataset by MolGraphConvFeaturizer\n",
    "\n",
    "We use dataset prepared for GAT model because featurizer is the same."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7b045ed",
   "metadata": {},
   "source": [
    "### Model , Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "3cd19d3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://deepchem.readthedocs.io/en/latest/api_reference/models.html#attentivefpmodel\n",
    "# Model for Graph Property Prediction. This model combines node features and edge features for initializing node representations.\n",
    "# For each graph, compute its representation by combining the representations of all nodes in it, which involves a gated recurrent unit (GRU).\n",
    "# It requires \"use_edges=True\" (Parameter of MolGraphConvFeaturizer)\n",
    "# This model cannot compute uncertainties\n",
    "\n",
    "del model\n",
    "\n",
    "model = dc.models.AttentiveFPModel(n_tasks=n_tasks,              # No. of tasks \n",
    "                                   mode='regression',            # Either “classification” or “regression”\n",
    "                                   batch_size=100,               # Batch size for training and evaluating\n",
    "                                   learning_rate=0.001,\n",
    "                                   num_layers=2,                 # No. of graph neural network layers\n",
    "                                   dropout=0.1                   # Dropout probability \n",
    "                                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "5750ade1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13810452461242675"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_avg = model.fit(train_dataset, nb_epoch=nb_epoch)\n",
    "loss_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "bffca007",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: {'r2_score': 0.9744812788976148, 'mean_squared_error': 0.11476535026344031}\n",
      "Test set score: {'r2_score': 0.917496779318387, 'mean_squared_error': 0.32727742053095}\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set score:\", model.evaluate(train_dataset, metrics))\n",
    "print(\"Test set score:\", model.evaluate(test_dataset, metrics))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "96ea855a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.917496779318387"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(test_dataset)   # numpy array [n, 1]\n",
    "r2_score(y_test, y_pred)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}