Demo II Sec 4.1 Gohr Ensemble Distinguisher - Speck32 Round 7.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "31aeb3c9-f6f2-4ac5-a9b0-57e4531de772",
   "metadata": {},
   "source": [
    "# Purpose of this notebook \n",
    "\n",
    "This notebook provides a demonstration of how NNBits is used to construct the ensemble distinguisher presented in our manuscript, section 4.1 for SPECK 32/64 round 7."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f136d969-65a6-425c-8889-26172fc34f7d",
   "metadata": {},
   "source": [
    "\n",
    "<div class=\"alert-info\">\n",
    "Please note that this is only a demo and you will need to adjust the paths at the positions indicated by `# <========= adjust this path`.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "314a628a-8da0-485a-a90d-3634b0d5ca13",
   "metadata": {},
   "source": [
    "# Create the dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1714115-d52a-462a-8fe8-42aac76396f0",
   "metadata": {},
   "source": [
    "To create new data for round 7 please run:\n",
    "\n",
    "```python \n",
    "nr = 7 \n",
    "n  = 10**7\n",
    "\n",
    "from gohr.gohr import *\n",
    "\n",
    "X, Y = make_train_data(n, nr);\n",
    "X_val, Y_val = make_train_data(n//10, nr);\n",
    "\n",
    "np.save('../D7/Round7/data_train_full.npy', X); # <========= adjust this path\n",
    "np.save('../D7/Round7/data_train_labels.npy', Y); # <========= adjust this path\n",
    "np.save('../D7/Round7/data_test_full.npy', X_val); # <========= adjust this path\n",
    "np.save('../D7/Round7/data_test_labels.npy', Y_val); # <========= adjust this path\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa76d3b0-7b5b-4577-8604-4ba96d1d9fc4",
   "metadata": {},
   "source": [
    "## Load round 7 data \n",
    "\n",
    "We assume the data has already been created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1cd2f171-3c36-4214-8093-b7d377221305",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T06:30:33.133150Z",
     "iopub.status.busy": "2023-02-07T06:30:33.132969Z",
     "iopub.status.idle": "2023-02-07T06:30:33.186181Z",
     "shell.execute_reply": "2023-02-07T06:30:33.185762Z",
     "shell.execute_reply.started": "2023-02-07T06:30:33.133130Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "X = np.load('../D7/Round7/data_train_full.npy') # <========= adjust this path\n",
    "Y = np.load('../D7/Round7/data_train_labels.npy') # <========= adjust this path\n",
    "X_val = np.load('../D7/Round7/data_test_full.npy') # <========= adjust this path\n",
    "Y_val = np.load('../D7/Round7/data_test_labels.npy') # <========= adjust this path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "335c7619-d0bf-419d-9f1b-22d920c29784",
   "metadata": {},
   "source": [
    "## Save two NNBits compatible versions of the dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4139a497-cde0-4839-b803-c9766306394a",
   "metadata": {},
   "source": [
    "### Version 1: Training and validation data for labels Y=[0,1] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0aef2ba-c122-4b0a-9414-b080c70e4ced",
   "metadata": {},
   "outputs": [],
   "source": [
    "#combine the data: concatenate rows\n",
    "final   = np.r_[X, X_val]\n",
    "#save final\n",
    "np.save('../D7/Round7/combined.npy', final) # <========= adjust this path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc106f4e-985d-470c-a55c-e09fa1631e02",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-10T06:28:00.627474Z",
     "iopub.status.busy": "2023-02-10T06:28:00.627273Z",
     "iopub.status.idle": "2023-02-10T06:28:00.630575Z",
     "shell.execute_reply": "2023-02-10T06:28:00.630045Z",
     "shell.execute_reply.started": "2023-02-10T06:28:00.627422Z"
    }
   },
   "source": [
    "### Version 2: Training and validation data for labels Y=[1] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "84a61363-0a3d-418e-bef0-7277363b6031",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T06:30:38.334220Z",
     "iopub.status.busy": "2023-02-07T06:30:38.333759Z",
     "iopub.status.idle": "2023-02-07T06:30:49.213317Z",
     "shell.execute_reply": "2023-02-07T06:30:49.212796Z",
     "shell.execute_reply.started": "2023-02-07T06:30:38.334196Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#--------------------------------------------------#\n",
    "# select only not-random samples to train the ensemble\n",
    "X = X[Y==1] \n",
    "X_val = X_val[Y_val==1]\n",
    "\n",
    "#combine the data: concatenate rows\n",
    "final   = np.r_[X, X_val]\n",
    "\n",
    "#save final\n",
    "np.save('../D7/Round7/combined_Y1.npy', final) # <========= adjust this path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbe901fb-b4b8-483e-bbe2-243870d0eaed",
   "metadata": {},
   "source": [
    "# Train ensemble (Y=1 data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6b9cabe4-55f8-4672-ad33-756a4a650926",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-13T05:53:44.725540Z",
     "start_time": "2022-05-13T05:53:44.723757Z"
    },
    "execution": {
     "iopub.execute_input": "2023-02-07T06:30:49.214455Z",
     "iopub.status.busy": "2023-02-07T06:30:49.214332Z",
     "iopub.status.idle": "2023-02-07T06:30:49.217043Z",
     "shell.execute_reply": "2023-02-07T06:30:49.216656Z",
     "shell.execute_reply.started": "2023-02-07T06:30:49.214440Z"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "savepath = f'gohr_ensemble_longR7run1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7fe8f030-8919-4348-85fb-34d680a8670c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-05-13T05:53:45.318242Z",
     "start_time": "2022-05-13T05:53:45.132498Z"
    },
    "execution": {
     "iopub.execute_input": "2023-02-07T06:30:49.217838Z",
     "iopub.status.busy": "2023-02-07T06:30:49.217604Z",
     "iopub.status.idle": "2023-02-07T06:30:49.523559Z",
     "shell.execute_reply": "2023-02-07T06:30:49.523091Z",
     "shell.execute_reply.started": "2023-02-07T06:30:49.217822Z"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import toml\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from nnbits.filemanager import FileManager\n",
    "from nnbits.bitanalysis import get_X"
   ]
  },
  {
   "cell_type": "raw",
   "id": "9d777e69-2f8a-4aab-bcf8-647b6ca39cb1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T04:39:46.336603Z",
     "iopub.status.busy": "2023-02-07T04:39:46.336471Z",
     "iopub.status.idle": "2023-02-07T04:39:46.943230Z",
     "shell.execute_reply": "2023-02-07T04:39:46.942603Z",
     "shell.execute_reply.started": "2023-02-07T04:39:46.336586Z"
    },
    "tags": []
   },
   "source": [
    "!rm -r '{savepath}'\n",
    "!mkdir '{savepath}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ee885ee0-ddcd-4ce4-be05-91d14bb29b6d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T06:30:54.219111Z",
     "iopub.status.busy": "2023-02-07T06:30:54.218880Z",
     "iopub.status.idle": "2023-02-07T06:30:54.230769Z",
     "shell.execute_reply": "2023-02-07T06:30:54.230355Z",
     "shell.execute_reply.started": "2023-02-07T06:30:54.219091Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========================================================================\n",
      "/opt/cryptanalysis_servers_shared_folder/NNBits/D7/Round7/combined_Y1.npy\n"
     ]
    }
   ],
   "source": [
    "F = FileManager(savepath)\n",
    "\n",
    "datapath = '/opt/cryptanalysis_servers_shared_folder/NNBits/D7/Round7/combined_Y1.npy' # <========= adjust this path\n",
    "cfgdict = {'DATAPATH': datapath, \n",
    "            # ensemble settings\n",
    "            'NEURAL_NETWORK_MODEL': 'gohr', \n",
    "            'NEURAL_NETWORKS': 64,  \n",
    "            'PREDICT_LABEL': False, \n",
    "            'SELECT_BITS_STRATEGY': 'target',\n",
    "            'TARGET_BITS': [0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,\n",
    "            17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,\n",
    "            34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,\n",
    "            51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], \n",
    "            'INPUT_DATA_OP': 'zero',\n",
    "            # hardware settings\n",
    "            'N_GPUS': 4,\n",
    "            'N_ACTORS_PER_GPU': 4,\n",
    "            'GPU_PER_ACTOR': 0.25,\n",
    "            'CPU_PER_ACTOR': 5, \n",
    "            # training settings\n",
    "            'N_EPOCHS': 200,\n",
    "            'N_TRAIN': 4997120,\n",
    "            'N_VAL': 499712,\n",
    "            'BATCHSIZE': 4096,\n",
    "            'EARLY_STOPPING': False, \n",
    "            'SAVE_BEST_WEIGHTS': True}\n",
    "\n",
    "with open(F.filename_config(), 'w') as configfile:\n",
    "    toml.dump(cfgdict, configfile)\n",
    "    \n",
    "print(\"=\"*len(datapath))\n",
    "print(datapath)\n",
    "    \n",
    "#!python -m nnbits.run --savepath '{savepath}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "976a6508-9c8e-4b6b-8b0b-5988af83a4fd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T06:31:05.634558Z",
     "iopub.status.busy": "2023-02-07T06:31:05.633903Z",
     "iopub.status.idle": "2023-02-07T11:14:53.150295Z",
     "shell.execute_reply": "2023-02-07T11:14:53.149679Z",
     "shell.execute_reply.started": "2023-02-07T06:31:05.634502Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-02-07_06h31m08s \t started to load data from harddisk...\n",
      "2023-02-07_06h31m11s \t finished.\n",
      "======================================================================\n",
      "|                             | value                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "|:----------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
      "| DATAPATH                    | /opt/cryptanalysis_servers_shared_folder/NNBits/D7/Round7/combined_Y1.npy                                                                                                                                                                                                                                                                                                              |\n",
      "| NEURAL_NETWORK_MODEL        | gohr                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| NEURAL_NETWORKS             | 64                                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "| PREDICT_LABEL               | False                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| SELECT_BITS_STRATEGY        | target                                                                                                                                                                                                                                                                                                                                                                                 |\n",
      "| TARGET_BITS                 | [[0], [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63]] |\n",
      "| INPUT_DATA_OP               | zero                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| N_GPUS                      | 4                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| N_ACTORS_PER_GPU            | 4                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| GPU_PER_ACTOR               | 0.25                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| CPU_PER_ACTOR               | 5                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| N_EPOCHS                    | 200                                                                                                                                                                                                                                                                                                                                                                                    |\n",
      "| N_TRAIN                     | 4997120                                                                                                                                                                                                                                                                                                                                                                                |\n",
      "| N_VAL                       | 499712                                                                                                                                                                                                                                                                                                                                                                                 |\n",
      "| BATCHSIZE                   | 4096                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| EARLY_STOPPING              | False                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| SAVE_BEST_WEIGHTS           | True                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| N_RANDOM_BITS               | 0                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| EARLY_STOPPING_P_VALUE      | 1e-10                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| CREATE_NEW_SELECTIONS       | True                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| MODEL_STRENGTH              | 1                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| VALIDATION_BATCH_SIZE       |                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "| SAVE_WEIGHTS                | False                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| N_TEST                      | 0                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| TESTING_BATCH_SIZE          | 0                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| TEST_ONLY                   | False                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| RESULTING BITS IN DATA-ROW  | 64                                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "| RESULTING N SELECTED BITS   | 1                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| RESULTING N TOTAL BITS      | 64                                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "| RESULTING NN INPUT NEURONS  | 64                                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "| RESULTING NN OUTPUT NEURONS | 1                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "\n",
      "\n",
      "2023-02-07 06:31:12,801\tINFO services.py:1245 -- View the Ray dashboard at \u001B[1m\u001B[32mhttp://127.0.0.1:8265\u001B[39m\u001B[22m\n",
      "||           time            |   NN finished   | pred. bits ||  best bit  |  acc (%)   |   n pred   |  p value   ||\n",
      "===================================================================================================================\n",
      "||   2023-02-07_06h31m15s    |      0/64       |    0/64    ||    nan     |    nan     |    nan     |    nan     ||\n",
      "||   2023-02-07_07h39m21s    |      1/64       |    1/64    ||     11     |   50.014   |     1      |    0.84    ||\n",
      "||   2023-02-07_07h39m36s    |      2/64       |    2/64    ||     11     |   50.014   |     1      |    0.84    ||\n",
      "||   2023-02-07_07h39m41s    |      3/64       |    3/64    ||     7      |   50.058   |     1      |    0.41    ||\n",
      "||   2023-02-07_07h40m13s    |      4/64       |    4/64    ||     15     |   50.087   |     1      |    0.22    ||\n",
      "||   2023-02-07_07h40m13s    |      5/64       |    5/64    ||     15     |   50.087   |     1      |    0.22    ||\n",
      "||   2023-02-07_07h40m18s    |      6/64       |    6/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h40m24s    |      7/64       |    7/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h40m28s    |      8/64       |    8/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h41m14s    |      9/64       |    9/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h41m39s    |      10/64      |   10/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h42m15s    |      11/64      |   11/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h42m27s    |      12/64      |   12/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h42m28s    |      13/64      |   13/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h42m47s    |      14/64      |   14/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h42m51s    |      15/64      |   15/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_07h42m52s    |      16/64      |   16/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h50m45s    |      17/64      |   17/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h51m24s    |      18/64      |   18/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h51m31s    |      19/64      |   19/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h51m42s    |      20/64      |   20/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h51m47s    |      21/64      |   21/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h52m40s    |      22/64      |   22/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h52m45s    |      23/64      |   23/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h52m46s    |      24/64      |   24/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h52m46s    |      25/64      |   25/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h52m47s    |      26/64      |   26/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h52m48s    |      27/64      |   27/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h52m48s    |      28/64      |   28/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h52m51s    |      29/64      |   29/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h53m19s    |      30/64      |   30/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h54m29s    |      31/64      |   31/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_08h54m47s    |      32/64      |   32/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_09h59m54s    |      33/64      |   33/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_10h01m04s    |      34/64      |   34/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_10h01m19s    |      35/64      |   35/64    ||     3      |   66.771   |     1      |     0      ||\n",
      "||   2023-02-07_10h01m37s    |      36/64      |   36/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h02m00s    |      37/64      |   37/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h02m07s    |      38/64      |   38/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h02m22s    |      39/64      |   39/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h02m26s    |      40/64      |   40/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h02m58s    |      41/64      |   41/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h03m18s    |      42/64      |   42/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h03m36s    |      43/64      |   43/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h03m45s    |      44/64      |   44/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h04m47s    |      45/64      |   45/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h04m48s    |      46/64      |   46/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h04m59s    |      47/64      |   47/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_10h05m41s    |      48/64      |   48/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h10m18s    |      49/64      |   49/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h11m12s    |      50/64      |   50/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h12m01s    |      51/64      |   51/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h12m13s    |      52/64      |   52/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h12m22s    |      53/64      |   53/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h13m00s    |      54/64      |   54/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h13m23s    |      55/64      |   55/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h13m25s    |      57/64      |   57/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h13m26s    |      58/64      |   58/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h13m31s    |      59/64      |   59/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h13m34s    |      60/64      |   60/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h13m37s    |      61/64      |   61/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h14m05s    |      62/64      |   62/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h14m45s    |      63/64      |   63/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "||   2023-02-07_11h14m50s    |      64/64      |   64/64    ||     35     |   66.841   |     1      |     0      ||\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "!python -m nnbits.run --savepath 'gohr_ensemble_longR7run1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d45dd59f-15d6-460d-b164-7fc0d12a93c0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T04:41:22.713082Z",
     "iopub.status.busy": "2023-02-07T04:41:22.712783Z",
     "iopub.status.idle": "2023-02-07T04:41:22.715755Z",
     "shell.execute_reply": "2023-02-07T04:41:22.715322Z",
     "shell.execute_reply.started": "2023-02-07T04:41:22.713062Z"
    }
   },
   "source": [
    "# Test ensemble (Y=0 and Y=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "88ecd9bf-c2f4-4df1-91bd-e15f73dfa703",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T11:14:53.151590Z",
     "iopub.status.busy": "2023-02-07T11:14:53.151456Z",
     "iopub.status.idle": "2023-02-07T11:14:53.161384Z",
     "shell.execute_reply": "2023-02-07T11:14:53.161082Z",
     "shell.execute_reply.started": "2023-02-07T11:14:53.151571Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================\n",
      "/opt/cryptanalysis_servers_shared_folder/NNBits/D7/Round7/combined.npy\n"
     ]
    }
   ],
   "source": [
    "F = FileManager(savepath)\n",
    "\n",
    "datapath = '/opt/cryptanalysis_servers_shared_folder/NNBits/D7/Round7/combined.npy' # <========= adjust this path\n",
    "cfgdict = {'DATAPATH': datapath, \n",
    "            # ensemble settings\n",
    "            'NEURAL_NETWORK_MODEL': 'gohr', \n",
    "            'NEURAL_NETWORKS': 64,  \n",
    "            'PREDICT_LABEL': False, \n",
    "            'SELECT_BITS_STRATEGY': 'target',\n",
    "            'TARGET_BITS': [0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,\n",
    "            17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,\n",
    "            34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,\n",
    "            51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], \n",
    "            'INPUT_DATA_OP': 'zero',\n",
    "            # hardware settings\n",
    "            'N_GPUS': 4,\n",
    "            'N_ACTORS_PER_GPU': 4,\n",
    "            'GPU_PER_ACTOR': 0.25,\n",
    "            'CPU_PER_ACTOR': 5, \n",
    "            # testing settings \n",
    "            'N_TEST': 11_000_000, \n",
    "            'TEST_ONLY': True, \n",
    "            # training settings\n",
    "            'N_EPOCHS': 2,\n",
    "            'N_TRAIN': 0,\n",
    "            'N_VAL': 0,\n",
    "            'BATCHSIZE': 4096,\n",
    "            'EARLY_STOPPING': False, \n",
    "            'SAVE_BEST_WEIGHTS': True\n",
    "             }\n",
    "\n",
    "with open(F.filename_config(), 'w') as configfile:\n",
    "    toml.dump(cfgdict, configfile)\n",
    "    \n",
    "print(\"=\"*len(datapath))\n",
    "print(datapath) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "37bdc11a-a3b5-4b50-bbe6-6013363eb900",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T11:14:53.161993Z",
     "iopub.status.busy": "2023-02-07T11:14:53.161882Z",
     "iopub.status.idle": "2023-02-07T12:14:01.561064Z",
     "shell.execute_reply": "2023-02-07T12:14:01.560447Z",
     "shell.execute_reply.started": "2023-02-07T11:14:53.161979Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-02-07_11h14m56s \t started to load data from harddisk...\n",
      "2023-02-07_11h15m03s \t finished.\n",
      "======================================================================\n",
      "|                             | value                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "|:----------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
      "| DATAPATH                    | /opt/cryptanalysis_servers_shared_folder/NNBits/D7/Round7/combined.npy                                                                                                                                                                                                                                                                                                                 |\n",
      "| NEURAL_NETWORK_MODEL        | gohr                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| NEURAL_NETWORKS             | 64                                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "| PREDICT_LABEL               | False                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| SELECT_BITS_STRATEGY        | target                                                                                                                                                                                                                                                                                                                                                                                 |\n",
      "| TARGET_BITS                 | [[0], [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63]] |\n",
      "| INPUT_DATA_OP               | zero                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| N_GPUS                      | 4                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| N_ACTORS_PER_GPU            | 4                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| GPU_PER_ACTOR               | 0.25                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| CPU_PER_ACTOR               | 5                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| N_TEST                      | 11000000                                                                                                                                                                                                                                                                                                                                                                               |\n",
      "| TEST_ONLY                   | True                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| N_EPOCHS                    | 2                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| N_TRAIN                     | 0                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| N_VAL                       | 0                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| BATCHSIZE                   | 4096                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| EARLY_STOPPING              | False                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| SAVE_BEST_WEIGHTS           | True                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| N_RANDOM_BITS               | 0                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| EARLY_STOPPING_P_VALUE      | 1e-10                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| CREATE_NEW_SELECTIONS       | True                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "| MODEL_STRENGTH              | 1                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| VALIDATION_BATCH_SIZE       |                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "| SAVE_WEIGHTS                | False                                                                                                                                                                                                                                                                                                                                                                                  |\n",
      "| TESTING_BATCH_SIZE          | 11000000                                                                                                                                                                                                                                                                                                                                                                               |\n",
      "| RESULTING BITS IN DATA-ROW  | 64                                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "| RESULTING N SELECTED BITS   | 1                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "| RESULTING N TOTAL BITS      | 64                                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "| RESULTING NN INPUT NEURONS  | 64                                                                                                                                                                                                                                                                                                                                                                                     |\n",
      "| RESULTING NN OUTPUT NEURONS | 1                                                                                                                                                                                                                                                                                                                                                                                      |\n",
      "\n",
      "\n",
      "2023-02-07 11:15:05,611\tINFO services.py:1245 -- View the Ray dashboard at \u001B[1m\u001B[32mhttp://127.0.0.1:8265\u001B[39m\u001B[22m\n",
      "||           time            |   NN finished   | pred. bits ||  best bit  |  acc (%)   |   n pred   |  p value   ||\n",
      "===================================================================================================================\n",
      "||   2023-02-07_11h15m08s    |      64/64      |   64/64    ||    nan     |    nan     |    nan     |    nan     ||\n",
      "\n",
      "TEST RESULTS\n",
      "============\n",
      "All neural networks have been tested.\n",
      "You can load the correctness of each single bit prediction via\n",
      "          >>> y_pred = np.load(F.filename_bitbybit_test_accs(nn))\n",
      "          >>> accuracy = y_pred.sum()/len(y_pred)*100\n",
      "\n",
      "The test accuracies calculated as above are as follows:\n",
      "|   network ID |   test accuracy (%) |\n",
      "|-------------:|--------------------:|\n",
      "|            0 |             50.0192 |\n",
      "|            1 |             50.0073 |\n",
      "|            2 |             50.0388 |\n",
      "|            3 |             58.6493 |\n",
      "|            4 |             50.0243 |\n",
      "|            5 |             56.0747 |\n",
      "|            6 |             50.0227 |\n",
      "|            7 |             50.008  |\n",
      "|            8 |             50.0233 |\n",
      "|            9 |             49.9963 |\n",
      "|           10 |             49.993  |\n",
      "|           11 |             49.9975 |\n",
      "|           12 |             55.0417 |\n",
      "|           13 |             50.4671 |\n",
      "|           14 |             51.3467 |\n",
      "|           15 |             50.1659 |\n",
      "|           16 |             50.0191 |\n",
      "|           17 |             50.0168 |\n",
      "|           18 |             50.0272 |\n",
      "|           19 |             57.4483 |\n",
      "|           20 |             50.0426 |\n",
      "|           21 |             49.9975 |\n",
      "|           22 |             50.005  |\n",
      "|           23 |             50.0685 |\n",
      "|           24 |             50.021  |\n",
      "|           25 |             49.9563 |\n",
      "|           26 |             50.0115 |\n",
      "|           27 |             50.012  |\n",
      "|           28 |             53.2519 |\n",
      "|           29 |             51.6939 |\n",
      "|           30 |             49.9938 |\n",
      "|           31 |             49.9969 |\n",
      "|           32 |             50.0111 |\n",
      "|           33 |             50.0038 |\n",
      "|           34 |             50.049  |\n",
      "|           35 |             58.7001 |\n",
      "|           36 |             50.0242 |\n",
      "|           37 |             56.0305 |\n",
      "|           38 |             50.0141 |\n",
      "|           39 |             50.0049 |\n",
      "|           40 |             50.0205 |\n",
      "|           41 |             49.9956 |\n",
      "|           42 |             50.0015 |\n",
      "|           43 |             50.0455 |\n",
      "|           44 |             54.7912 |\n",
      "|           45 |             53.6552 |\n",
      "|           46 |             51.3442 |\n",
      "|           47 |             50.172  |\n",
      "|           48 |             50.003  |\n",
      "|           49 |             49.994  |\n",
      "|           50 |             49.9923 |\n",
      "|           51 |             57.4002 |\n",
      "|           52 |             50.0067 |\n",
      "|           53 |             49.9936 |\n",
      "|           54 |             49.9887 |\n",
      "|           55 |             49.9898 |\n",
      "|           56 |             50.0691 |\n",
      "|           57 |             50.0161 |\n",
      "|           58 |             50.0138 |\n",
      "|           59 |             50.0186 |\n",
      "|           60 |             53.3147 |\n",
      "|           61 |             51.7534 |\n",
      "|           62 |             50.0416 |\n",
      "|           63 |             50.0111 |\n",
      "\u001B[0m"
     ]
    }
   ],
   "source": [
    "!python -m nnbits.run --savepath 'gohr_ensemble_longR7run1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a91be20-7f8d-4e9d-afcc-48b45769a8c4",
   "metadata": {},
   "source": [
    "# Train and evaluate MLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5fa4627e-2fde-4c79-a5f5-b3010473057f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T12:14:01.562547Z",
     "iopub.status.busy": "2023-02-07T12:14:01.562412Z",
     "iopub.status.idle": "2023-02-07T12:30:03.799339Z",
     "shell.execute_reply": "2023-02-07T12:30:03.798799Z",
     "shell.execute_reply.started": "2023-02-07T12:14:01.562529Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/200\n",
      "2000/2000 [==============================] - 8s 3ms/step - loss: 0.2370 - acc: 0.6026 - val_loss: 0.2369 - val_acc: 0.6006 - lr: 0.0020\n",
      "Epoch 2/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2356 - acc: 0.6043 - val_loss: 0.2366 - val_acc: 0.6006 - lr: 0.0018\n",
      "Epoch 3/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2355 - acc: 0.6045 - val_loss: 0.2367 - val_acc: 0.6006 - lr: 0.0016\n",
      "Epoch 4/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2354 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0014\n",
      "Epoch 5/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0012\n",
      "Epoch 6/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2353 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 9.4444e-04\n",
      "Epoch 7/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 8/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6049 - val_loss: 0.2364 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 9/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 10/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6051 - val_loss: 0.2363 - val_acc: 0.6011 - lr: 1.0000e-04\n",
      "Epoch 11/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0020\n",
      "Epoch 12/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6047 - val_loss: 0.2365 - val_acc: 0.6007 - lr: 0.0018\n",
      "Epoch 13/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2353 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6009 - lr: 0.0016\n",
      "Epoch 14/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6046 - val_loss: 0.2367 - val_acc: 0.6006 - lr: 0.0014\n",
      "Epoch 15/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0012\n",
      "Epoch 16/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 9.4444e-04\n",
      "Epoch 17/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 18/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 19/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 3.1111e-04\n",
      "Epoch 20/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6051 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 21/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2353 - acc: 0.6046 - val_loss: 0.2368 - val_acc: 0.6005 - lr: 0.0020\n",
      "Epoch 22/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6009 - lr: 0.0018\n",
      "Epoch 23/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6048 - val_loss: 0.2366 - val_acc: 0.6006 - lr: 0.0016\n",
      "Epoch 24/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0014\n",
      "Epoch 25/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2364 - val_acc: 0.6010 - lr: 0.0012\n",
      "Epoch 26/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 9.4444e-04\n",
      "Epoch 27/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 28/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6011 - lr: 5.2222e-04\n",
      "Epoch 29/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2363 - val_acc: 0.6012 - lr: 3.1111e-04\n",
      "Epoch 30/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6051 - val_loss: 0.2362 - val_acc: 0.6011 - lr: 1.0000e-04\n",
      "Epoch 31/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0020\n",
      "Epoch 32/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6046 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0018\n",
      "Epoch 33/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0016\n",
      "Epoch 34/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2367 - val_acc: 0.6007 - lr: 0.0014\n",
      "Epoch 35/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0012\n",
      "Epoch 36/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 9.4444e-04\n",
      "Epoch 37/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 38/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 39/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 40/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 41/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0020\n",
      "Epoch 42/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2353 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0018\n",
      "Epoch 43/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0016\n",
      "Epoch 44/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0014\n",
      "Epoch 45/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0012\n",
      "Epoch 46/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 9.4444e-04\n",
      "Epoch 47/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 7.3333e-04\n",
      "Epoch 48/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 5.2222e-04\n",
      "Epoch 49/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 50/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 51/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 0.0020\n",
      "Epoch 52/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6010 - lr: 0.0018\n",
      "Epoch 53/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 0.0016\n",
      "Epoch 54/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0014\n",
      "Epoch 55/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 0.0012\n",
      "Epoch 56/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6007 - lr: 9.4444e-04\n",
      "Epoch 57/200\n",
      "2000/2000 [==============================] - 4s 2ms/step - loss: 0.2352 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 7.3333e-04\n",
      "Epoch 58/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2351 - acc: 0.6048 - val_loss: 0.2362 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 59/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 60/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 61/200\n",
      "2000/2000 [==============================] - 4s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0020\n",
      "Epoch 62/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0018\n",
      "Epoch 63/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0016\n",
      "Epoch 64/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0014\n",
      "Epoch 65/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0012\n",
      "Epoch 66/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 9.4444e-04\n",
      "Epoch 67/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 68/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 5.2222e-04\n",
      "Epoch 69/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6009 - lr: 3.1111e-04\n",
      "Epoch 70/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6011 - lr: 1.0000e-04\n",
      "Epoch 71/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6004 - lr: 0.0020\n",
      "Epoch 72/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6003 - lr: 0.0018\n",
      "Epoch 73/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0016\n",
      "Epoch 74/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0014\n",
      "Epoch 75/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6006 - lr: 0.0012\n",
      "Epoch 76/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 9.4444e-04\n",
      "Epoch 77/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6011 - lr: 7.3333e-04\n",
      "Epoch 78/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6048 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 5.2222e-04\n",
      "Epoch 79/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6009 - lr: 3.1111e-04\n",
      "Epoch 80/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6011 - lr: 1.0000e-04\n",
      "Epoch 81/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0020\n",
      "Epoch 82/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6009 - lr: 0.0018\n",
      "Epoch 83/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6010 - lr: 0.0016\n",
      "Epoch 84/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0014\n",
      "Epoch 85/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 0.0012\n",
      "Epoch 86/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 9.4444e-04\n",
      "Epoch 87/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 7.3333e-04\n",
      "Epoch 88/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6048 - val_loss: 0.2362 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 89/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 90/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6011 - lr: 1.0000e-04\n",
      "Epoch 91/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0020\n",
      "Epoch 92/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6005 - lr: 0.0018\n",
      "Epoch 93/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0016\n",
      "Epoch 94/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0014\n",
      "Epoch 95/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0012\n",
      "Epoch 96/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 9.4444e-04\n",
      "Epoch 97/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 98/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6008 - lr: 5.2222e-04\n",
      "Epoch 99/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 100/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 101/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0020\n",
      "Epoch 102/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0018\n",
      "Epoch 103/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0016\n",
      "Epoch 104/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6010 - lr: 0.0014\n",
      "Epoch 105/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6011 - lr: 0.0012\n",
      "Epoch 106/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6006 - lr: 9.4444e-04\n",
      "Epoch 107/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 7.3333e-04\n",
      "Epoch 108/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 5.2222e-04\n",
      "Epoch 109/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 110/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 111/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6006 - lr: 0.0020\n",
      "Epoch 112/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0018\n",
      "Epoch 113/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6009 - lr: 0.0016\n",
      "Epoch 114/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6009 - lr: 0.0014\n",
      "Epoch 115/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 0.0012\n",
      "Epoch 116/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 9.4444e-04\n",
      "Epoch 117/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6007 - lr: 7.3333e-04\n",
      "Epoch 118/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 119/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 3.1111e-04\n",
      "Epoch 120/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 121/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0020\n",
      "Epoch 122/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2365 - val_acc: 0.6001 - lr: 0.0018\n",
      "Epoch 123/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6005 - lr: 0.0016\n",
      "Epoch 124/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0014\n",
      "Epoch 125/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0012\n",
      "Epoch 126/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 9.4444e-04\n",
      "Epoch 127/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 7.3333e-04\n",
      "Epoch 128/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 5.2222e-04\n",
      "Epoch 129/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6008 - lr: 3.1111e-04\n",
      "Epoch 130/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 131/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6005 - lr: 0.0020\n",
      "Epoch 132/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0018\n",
      "Epoch 133/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0016\n",
      "Epoch 134/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0014\n",
      "Epoch 135/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6006 - lr: 0.0012\n",
      "Epoch 136/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 9.4444e-04\n",
      "Epoch 137/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 7.3333e-04\n",
      "Epoch 138/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 5.2222e-04\n",
      "Epoch 139/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6007 - lr: 3.1111e-04\n",
      "Epoch 140/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 141/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0020\n",
      "Epoch 142/200\n",
      "2000/2000 [==============================] - 4s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0018\n",
      "Epoch 143/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6009 - lr: 0.0016\n",
      "Epoch 144/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6007 - lr: 0.0014\n",
      "Epoch 145/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0012\n",
      "Epoch 146/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6011 - lr: 9.4444e-04\n",
      "Epoch 147/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 7.3333e-04\n",
      "Epoch 148/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 149/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 150/200\n",
      "2000/2000 [==============================] - 4s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6009 - lr: 1.0000e-04\n",
      "Epoch 151/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6005 - lr: 0.0020\n",
      "Epoch 152/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6007 - lr: 0.0018\n",
      "Epoch 153/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0016\n",
      "Epoch 154/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0014\n",
      "Epoch 155/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0012\n",
      "Epoch 156/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 9.4444e-04\n",
      "Epoch 157/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 7.3333e-04\n",
      "Epoch 158/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 159/200\n",
      "2000/2000 [==============================] - 4s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 160/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 161/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2365 - val_acc: 0.6006 - lr: 0.0020\n",
      "Epoch 162/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6007 - lr: 0.0018\n",
      "Epoch 163/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0016\n",
      "Epoch 164/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0014\n",
      "Epoch 165/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6007 - lr: 0.0012\n",
      "Epoch 166/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 9.4444e-04\n",
      "Epoch 167/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 7.3333e-04\n",
      "Epoch 168/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 169/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 170/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6009 - lr: 1.0000e-04\n",
      "Epoch 171/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0020\n",
      "Epoch 172/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2363 - val_acc: 0.6006 - lr: 0.0018\n",
      "Epoch 173/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6007 - lr: 0.0016\n",
      "Epoch 174/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6011 - lr: 0.0014\n",
      "Epoch 175/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6007 - lr: 0.0012\n",
      "Epoch 176/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 9.4444e-04\n",
      "Epoch 177/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 178/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 179/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 3.1111e-04\n",
      "Epoch 180/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 181/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0020\n",
      "Epoch 182/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2365 - val_acc: 0.6007 - lr: 0.0018\n",
      "Epoch 183/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6008 - lr: 0.0016\n",
      "Epoch 184/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0014\n",
      "Epoch 185/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2364 - val_acc: 0.6005 - lr: 0.0012\n",
      "Epoch 186/200\n",
      "2000/2000 [==============================] - 4s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2362 - val_acc: 0.6008 - lr: 9.4444e-04\n",
      "Epoch 187/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 188/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6010 - lr: 5.2222e-04\n",
      "Epoch 189/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6009 - lr: 3.1111e-04\n",
      "Epoch 190/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n",
      "Epoch 191/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6005 - lr: 0.0020\n",
      "Epoch 192/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2364 - val_acc: 0.6006 - lr: 0.0018\n",
      "Epoch 193/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2352 - acc: 0.6046 - val_loss: 0.2364 - val_acc: 0.6008 - lr: 0.0016\n",
      "Epoch 194/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6006 - lr: 0.0014\n",
      "Epoch 195/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6047 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 0.0012\n",
      "Epoch 196/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6011 - lr: 9.4444e-04\n",
      "Epoch 197/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2352 - acc: 0.6048 - val_loss: 0.2363 - val_acc: 0.6009 - lr: 7.3333e-04\n",
      "Epoch 198/200\n",
      "2000/2000 [==============================] - 5s 3ms/step - loss: 0.2351 - acc: 0.6048 - val_loss: 0.2362 - val_acc: 0.6009 - lr: 5.2222e-04\n",
      "Epoch 199/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6049 - val_loss: 0.2362 - val_acc: 0.6011 - lr: 3.1111e-04\n",
      "Epoch 200/200\n",
      "2000/2000 [==============================] - 5s 2ms/step - loss: 0.2351 - acc: 0.6050 - val_loss: 0.2362 - val_acc: 0.6010 - lr: 1.0000e-04\n"
     ]
    }
   ],
   "source": [
    "#---------- Create MLP Model -----------\n",
    "from gohr.gohr import bs, LearningRateScheduler, cyclic_lr\n",
    "\n",
    "def create_model(d1=64, d2=64, reg_param=1e-5):\n",
    "        # --- prepare GPU\n",
    "    import tensorflow as tf\n",
    "    gpus = tf.config.experimental.list_physical_devices(\"GPU\")\n",
    "    tf.config.experimental.set_visible_devices(gpus[0], 'GPU')\n",
    "    from keras.models import Model\n",
    "    from keras.layers import Dense, Conv1D, Conv2D, Input, Reshape, Permute, Add, Flatten, BatchNormalization, \\\n",
    "        Activation, Lambda\n",
    "    from keras.layers import Concatenate, MaxPooling2D\n",
    "    from keras.regularizers import l2\n",
    "    from keras import backend as K\n",
    "    import numpy as np\n",
    "    \n",
    "    inp = Input(shape=(64,))\n",
    "    dense1 = Dense(d1,kernel_regularizer=l2(reg_param))(inp);\n",
    "    dense1 = BatchNormalization()(dense1);\n",
    "    dense1 = Activation('relu')(dense1);\n",
    "    dense2 = Dense(d2, kernel_regularizer=l2(reg_param))(dense1);\n",
    "    dense2 = BatchNormalization()(dense2);\n",
    "    dense2 = Activation('relu')(dense2);\n",
    "    out = Dense(1, activation='sigmoid', kernel_regularizer=l2(reg_param))(dense2);\n",
    "    model = Model(inputs=inp, outputs=out);\n",
    "    return(model);\n",
    "\n",
    "N_TRAIN = 10_000_000\n",
    "N_EVAL = 1_000_000\n",
    "\n",
    "#--------- Training Data ---------\n",
    "Y = np.load('../D7/Round7/data_train_labels.npy')[:N_TRAIN] # <========= adjust this path\n",
    "\n",
    "# load the correctness of predictions from single neural networks: \n",
    "X = np.zeros((N_TRAIN, 64))\n",
    "\n",
    "for network_id in np.arange(64):\n",
    "    #dist_id = best_dist_ids[network_id]\n",
    "    #savepath = savepaths[dist_id]\n",
    "    filename = f'{savepath}/test_accuracies_bit_by_bit/{network_id}_testing.npy'\n",
    "    x = np.load(filename)\n",
    "    X[:,network_id] = x[:N_TRAIN]\n",
    "\n",
    "#--------- Test Data ---------\n",
    "Y_eval = np.load('../D7/Round7/data_test_labels.npy')[:N_EVAL] # <========= adjust this path\n",
    "\n",
    "#Y_eval = Y_eval[1:] # remove the first entry \n",
    "# predictions from single neural networks: \n",
    "X_eval = np.zeros((N_EVAL, 64))\n",
    "\n",
    "for network_id in np.arange(64):\n",
    "    #dist_id = best_dist_ids[network_id]\n",
    "    #savepath = savepaths[dist_id]\n",
    "    filename = f'{savepath}/test_accuracies_bit_by_bit/{network_id}_testing.npy'\n",
    "    x = np.load(filename)\n",
    "    X_eval[:,network_id] = x[N_TRAIN:]\n",
    "\n",
    "num_epochs = 200\n",
    "\n",
    "model = create_model(reg_param=10**-5);\n",
    "model.compile(optimizer='adam',loss='mse',metrics=['acc']);\n",
    "\n",
    "#create learnrate schedule\n",
    "lr = LearningRateScheduler(cyclic_lr(10,0.002, 0.0001));\n",
    "\n",
    "#train and evaluate\n",
    "h = model.fit(X, Y, epochs=num_epochs, batch_size=bs, callbacks=[lr], validation_data=(X_eval, Y_eval))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ddae3a5c-3f97-4615-9d6f-b28fad685045",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-07T12:30:03.800544Z",
     "iopub.status.busy": "2023-02-07T12:30:03.800417Z",
     "iopub.status.idle": "2023-02-07T12:30:03.803716Z",
     "shell.execute_reply": "2023-02-07T12:30:03.803336Z",
     "shell.execute_reply.started": "2023-02-07T12:30:03.800527Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best validation accuracy:  0.6012279987335205\n"
     ]
    }
   ],
   "source": [
    "print(\"Best validation accuracy: \", np.max(h.history['val_acc']));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0829bc2-7aea-4c8b-85bc-f4f9041379c9",
   "metadata": {},
   "source": [
    "# Validation accuracy of MLP / Ensemble distinguisher = 60.1% "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cc8b08e-2b08-48e6-b578-5ed2d341244b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "462.583px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "toc-autonumbering": false,
  "toc-showtags": false,
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}