1.profile.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply an image-based profiling pipeline using pycytominer\n",
    "\n",
    "As described fully in [Caicedo et al. 2017](https://doi.org/10.1038/nmeth.4397), an image-based profiling pipeline consists of three core steps:\n",
    "\n",
    "1. Aggregation\n",
    "2. Normalization\n",
    "3. Feature selection\n",
    "\n",
    "[Pycytominer](https://github.com/cytomining/pycytominer) is a python package, built on top of pandas, that facilitates all of these steps and more.\n",
    "\n",
    "### Data levels\n",
    "\n",
    "The concept of \"data levels\" is important to understand when implementing an image-based profiling pipeline.\n",
    "\n",
    "| Data | Level |\n",
    "| :---- | :---- |\n",
    "| Images | Level 1 |\n",
    "| Single cell profiles (SQLite) | Level 2 |\n",
    "| Aggregated profiles with metadata information | Level 3 |\n",
    "| Normalized aggregated profiles | Level 4a |\n",
    "| Normalized and feature selected profiles | Level 4b |\n",
    "| Consensus profiles | Level 5 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "import pandas as pd\n",
    "\n",
    "from pycytominer import annotate, normalize, feature_select, consensus\n",
    "from pycytominer.cyto_utils import cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 0 - Initialize data and options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sqlite:///data/218360.sqlite'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dir = \"data\"\n",
    "plate_id = \"218360\"\n",
    "platemap = f\"{plate_id}.txt\"\n",
    "\n",
    "sqlite_file = f\"sqlite:///{data_dir}/{plate_id}.sqlite\"\n",
    "sqlite_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_dir = pathlib.Path(\"profiles\")\n",
    "output_dir.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60, 8)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>plate_map_name</th>\n",
       "      <th>well_position</th>\n",
       "      <th>clone_number</th>\n",
       "      <th>plate_ID</th>\n",
       "      <th>plate_filename</th>\n",
       "      <th>treatment</th>\n",
       "      <th>treatment_time</th>\n",
       "      <th>cell_density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>218360</td>\n",
       "      <td>B02</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>0.1% DMSO</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>218360</td>\n",
       "      <td>B03</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>2.1 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>218360</td>\n",
       "      <td>B04</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>21 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   plate_map_name well_position clone_number  plate_ID     plate_filename  \\\n",
       "0          218360           B02  WT parental    218360  20200626-WTpAE-Lo   \n",
       "1          218360           B03  WT parental    218360  20200626-WTpAE-Lo   \n",
       "2          218360           B04  WT parental    218360  20200626-WTpAE-Lo   \n",
       "\n",
       "           treatment treatment_time         cell_density  \n",
       "0          0.1% DMSO          13 hr  2.5x10^3 cells/well  \n",
       "1  2.1 nM bortezomib          13 hr  2.5x10^3 cells/well  \n",
       "2   21 nM bortezomib          13 hr  2.5x10^3 cells/well  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load platemap file\n",
    "platemap_file = pathlib.Path(f\"{data_dir}/{platemap}\")\n",
    "platemap_df = pd.read_csv(platemap_file, sep=\"\\t\")\n",
    "\n",
    "print(platemap_df.shape)\n",
    "platemap_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output options\n",
    "compression_options = {\"method\": \"gzip\", \"mtime\": 1}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1 - Single cell aggregation\n",
    "\n",
    "In this step, **level 2 profiles** (single cells) are processed to **level 3 profiles** (well-level profiles)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the single cells class\n",
    "sc_data = cells.SingleCells(\n",
    "    sqlite_file,\n",
    "    strata=[\"Metadata_Plate\", \"Metadata_Well\"],\n",
    "    features=\"infer\",\n",
    "    compartments=[\"cells\", \"cytoplasm\", \"nuclei\"],\n",
    "    merge_cols=[\"TableNumber\", \"ImageNumber\"],\n",
    "    aggregation_operation=\"median\",\n",
    "    load_image_data=True,\n",
    "    subsample_frac=1,\n",
    "    subsample_n=\"all\",\n",
    "    subsampling_random_state=\"none\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60, 11)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metadata_Plate</th>\n",
       "      <th>Metadata_Well</th>\n",
       "      <th>cell_count</th>\n",
       "      <th>plate_map_name</th>\n",
       "      <th>well_position</th>\n",
       "      <th>clone_number</th>\n",
       "      <th>plate_ID</th>\n",
       "      <th>plate_filename</th>\n",
       "      <th>treatment</th>\n",
       "      <th>treatment_time</th>\n",
       "      <th>cell_density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>218360</td>\n",
       "      <td>B02</td>\n",
       "      <td>1386</td>\n",
       "      <td>218360</td>\n",
       "      <td>B02</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>0.1% DMSO</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>218360</td>\n",
       "      <td>B03</td>\n",
       "      <td>793</td>\n",
       "      <td>218360</td>\n",
       "      <td>B03</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>2.1 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>218360</td>\n",
       "      <td>B04</td>\n",
       "      <td>456</td>\n",
       "      <td>218360</td>\n",
       "      <td>B04</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>21 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>218360</td>\n",
       "      <td>B05</td>\n",
       "      <td>258</td>\n",
       "      <td>218360</td>\n",
       "      <td>B05</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>210 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>218360</td>\n",
       "      <td>B06</td>\n",
       "      <td>1094</td>\n",
       "      <td>218360</td>\n",
       "      <td>B06</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>0.1% DMSO</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Metadata_Plate Metadata_Well  cell_count  plate_map_name well_position  \\\n",
       "0          218360           B02        1386          218360           B02   \n",
       "1          218360           B03         793          218360           B03   \n",
       "2          218360           B04         456          218360           B04   \n",
       "3          218360           B05         258          218360           B05   \n",
       "4          218360           B06        1094          218360           B06   \n",
       "\n",
       "  clone_number  plate_ID     plate_filename          treatment treatment_time  \\\n",
       "0  WT parental    218360  20200626-WTpAE-Lo          0.1% DMSO          13 hr   \n",
       "1  WT parental    218360  20200626-WTpAE-Lo  2.1 nM bortezomib          13 hr   \n",
       "2  WT parental    218360  20200626-WTpAE-Lo   21 nM bortezomib          13 hr   \n",
       "3  WT parental    218360  20200626-WTpAE-Lo  210 nM bortezomib          13 hr   \n",
       "4  WT parental    218360  20200626-WTpAE-Lo          0.1% DMSO          13 hr   \n",
       "\n",
       "          cell_density  \n",
       "0  2.5x10^3 cells/well  \n",
       "1  2.5x10^3 cells/well  \n",
       "2  2.5x10^3 cells/well  \n",
       "3  2.5x10^3 cells/well  \n",
       "4  2.5x10^3 cells/well  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Count cells\n",
    "cell_count_df = sc_data.count_cells()\n",
    "\n",
    "cell_count_df = (\n",
    "    cell_count_df.merge(\n",
    "        platemap_df,\n",
    "        left_on=\"Metadata_Well\",\n",
    "        right_on=\"well_position\"\n",
    "    )\n",
    ")\n",
    "\n",
    "# Output cell count to file\n",
    "cell_count_file = pathlib.Path(f\"{output_dir}/cell_counts.tsv\")\n",
    "cell_count_df.to_csv(cell_count_file, sep=\"\\t\", index=False)\n",
    "\n",
    "print(cell_count_df.shape)\n",
    "cell_count_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('profiles/218360.csv.gz')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform the aggregation - output well level profiles\n",
    "output_file = pathlib.Path(f\"{output_dir}/{plate_id}.csv.gz\")\n",
    "\n",
    "# Aggregate profiles can output a file, or save the result to a variable\n",
    "# Here, we output the intermediate result to a file\n",
    "sc_data.aggregate_profiles(\n",
    "    output_file=output_file,\n",
    "    compute_subsample=True,\n",
    "    compression_options=compression_options,\n",
    "    float_format=None\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60, 3530)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metadata_Plate</th>\n",
       "      <th>Metadata_Well</th>\n",
       "      <th>Metadata_Site_Count</th>\n",
       "      <th>Metadata_Object_Count</th>\n",
       "      <th>Cells_AreaShape_Area</th>\n",
       "      <th>Cells_AreaShape_Center_X</th>\n",
       "      <th>Cells_AreaShape_Center_Y</th>\n",
       "      <th>Cells_AreaShape_Center_Z</th>\n",
       "      <th>Cells_AreaShape_Compactness</th>\n",
       "      <th>Cells_AreaShape_Eccentricity</th>\n",
       "      <th>...</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_10_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_10_03</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_00</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_01</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_03</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_00</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_01</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_03</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>218360</td>\n",
       "      <td>B02</td>\n",
       "      <td>16</td>\n",
       "      <td>1386</td>\n",
       "      <td>3103.0</td>\n",
       "      <td>1185.5</td>\n",
       "      <td>1134.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.176543</td>\n",
       "      <td>0.762628</td>\n",
       "      <td>...</td>\n",
       "      <td>134.630929</td>\n",
       "      <td>136.168955</td>\n",
       "      <td>130.178186</td>\n",
       "      <td>143.230289</td>\n",
       "      <td>130.452067</td>\n",
       "      <td>139.230146</td>\n",
       "      <td>135.932245</td>\n",
       "      <td>133.168297</td>\n",
       "      <td>137.391985</td>\n",
       "      <td>135.095233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>218360</td>\n",
       "      <td>B03</td>\n",
       "      <td>16</td>\n",
       "      <td>793</td>\n",
       "      <td>3365.0</td>\n",
       "      <td>1067.0</td>\n",
       "      <td>1137.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.148519</td>\n",
       "      <td>0.738471</td>\n",
       "      <td>...</td>\n",
       "      <td>143.936790</td>\n",
       "      <td>150.720200</td>\n",
       "      <td>155.195390</td>\n",
       "      <td>182.867025</td>\n",
       "      <td>153.439750</td>\n",
       "      <td>162.839254</td>\n",
       "      <td>146.040203</td>\n",
       "      <td>143.993466</td>\n",
       "      <td>147.316124</td>\n",
       "      <td>147.909953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>218360</td>\n",
       "      <td>B04</td>\n",
       "      <td>16</td>\n",
       "      <td>456</td>\n",
       "      <td>3360.5</td>\n",
       "      <td>1141.0</td>\n",
       "      <td>1166.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.152265</td>\n",
       "      <td>0.735413</td>\n",
       "      <td>...</td>\n",
       "      <td>139.837458</td>\n",
       "      <td>147.532470</td>\n",
       "      <td>143.564510</td>\n",
       "      <td>152.944447</td>\n",
       "      <td>143.430507</td>\n",
       "      <td>147.967005</td>\n",
       "      <td>140.323478</td>\n",
       "      <td>141.193128</td>\n",
       "      <td>140.788792</td>\n",
       "      <td>140.561236</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 3530 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Metadata_Plate Metadata_Well  Metadata_Site_Count  Metadata_Object_Count  \\\n",
       "0          218360           B02                   16                   1386   \n",
       "1          218360           B03                   16                    793   \n",
       "2          218360           B04                   16                    456   \n",
       "\n",
       "   Cells_AreaShape_Area  Cells_AreaShape_Center_X  Cells_AreaShape_Center_Y  \\\n",
       "0                3103.0                    1185.5                    1134.5   \n",
       "1                3365.0                    1067.0                    1137.0   \n",
       "2                3360.5                    1141.0                    1166.5   \n",
       "\n",
       "   Cells_AreaShape_Center_Z  Cells_AreaShape_Compactness  \\\n",
       "0                       1.0                     1.176543   \n",
       "1                       1.0                     1.148519   \n",
       "2                       1.0                     1.152265   \n",
       "\n",
       "   Cells_AreaShape_Eccentricity  ...  Nuclei_Texture_Variance_RNA_10_02  \\\n",
       "0                      0.762628  ...                         134.630929   \n",
       "1                      0.738471  ...                         143.936790   \n",
       "2                      0.735413  ...                         139.837458   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_10_03  Nuclei_Texture_Variance_RNA_20_00  \\\n",
       "0                         136.168955                         130.178186   \n",
       "1                         150.720200                         155.195390   \n",
       "2                         147.532470                         143.564510   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_20_01  Nuclei_Texture_Variance_RNA_20_02  \\\n",
       "0                         143.230289                         130.452067   \n",
       "1                         182.867025                         153.439750   \n",
       "2                         152.944447                         143.430507   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_20_03  Nuclei_Texture_Variance_RNA_5_00  \\\n",
       "0                         139.230146                        135.932245   \n",
       "1                         162.839254                        146.040203   \n",
       "2                         147.967005                        140.323478   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_5_01  Nuclei_Texture_Variance_RNA_5_02  \\\n",
       "0                        133.168297                        137.391985   \n",
       "1                        143.993466                        147.316124   \n",
       "2                        141.193128                        140.788792   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_5_03  \n",
       "0                        135.095233  \n",
       "1                        147.909953  \n",
       "2                        140.561236  \n",
       "\n",
       "[3 rows x 3530 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in and preview what was output in the previous step\n",
    "# Note: This is not necessary to perform in automated pipelines, but is a nice check.\n",
    "aggregated_df = pd.read_csv(output_file)\n",
    "\n",
    "print(aggregated_df.shape)\n",
    "aggregated_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Annotate wells using the platemap file\n",
    "\n",
    "In this step, **level 3 profiles** (well-level profiles) are annotated with platemap metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Annotate profiles\n",
    "annotate_file = pathlib.Path(f\"{output_dir}/{plate_id}_augmented.csv.gz\")\n",
    "\n",
    "annotate(\n",
    "    profiles=output_file,\n",
    "    platemap=platemap_df,\n",
    "    join_on=[\"Metadata_well_position\", \"Metadata_Well\"],\n",
    "    output_file=annotate_file,\n",
    "    compression_options=compression_options,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60, 3537)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metadata_plate_map_name</th>\n",
       "      <th>Metadata_clone_number</th>\n",
       "      <th>Metadata_plate_ID</th>\n",
       "      <th>Metadata_plate_filename</th>\n",
       "      <th>Metadata_treatment</th>\n",
       "      <th>Metadata_treatment_time</th>\n",
       "      <th>Metadata_cell_density</th>\n",
       "      <th>Metadata_Plate</th>\n",
       "      <th>Metadata_Well</th>\n",
       "      <th>Metadata_Site_Count</th>\n",
       "      <th>...</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_10_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_10_03</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_00</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_01</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_03</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_00</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_01</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_03</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>0.1% DMSO</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B02</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>134.630929</td>\n",
       "      <td>136.168955</td>\n",
       "      <td>130.178186</td>\n",
       "      <td>143.230289</td>\n",
       "      <td>130.452067</td>\n",
       "      <td>139.230146</td>\n",
       "      <td>135.932245</td>\n",
       "      <td>133.168297</td>\n",
       "      <td>137.391985</td>\n",
       "      <td>135.095233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>2.1 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B03</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>143.936790</td>\n",
       "      <td>150.720200</td>\n",
       "      <td>155.195390</td>\n",
       "      <td>182.867025</td>\n",
       "      <td>153.439750</td>\n",
       "      <td>162.839254</td>\n",
       "      <td>146.040203</td>\n",
       "      <td>143.993466</td>\n",
       "      <td>147.316124</td>\n",
       "      <td>147.909953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>21 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B04</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>139.837458</td>\n",
       "      <td>147.532470</td>\n",
       "      <td>143.564510</td>\n",
       "      <td>152.944447</td>\n",
       "      <td>143.430507</td>\n",
       "      <td>147.967005</td>\n",
       "      <td>140.323478</td>\n",
       "      <td>141.193128</td>\n",
       "      <td>140.788792</td>\n",
       "      <td>140.561236</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 3537 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Metadata_plate_map_name Metadata_clone_number  Metadata_plate_ID  \\\n",
       "0                   218360           WT parental             218360   \n",
       "1                   218360           WT parental             218360   \n",
       "2                   218360           WT parental             218360   \n",
       "\n",
       "  Metadata_plate_filename Metadata_treatment Metadata_treatment_time  \\\n",
       "0       20200626-WTpAE-Lo          0.1% DMSO                   13 hr   \n",
       "1       20200626-WTpAE-Lo  2.1 nM bortezomib                   13 hr   \n",
       "2       20200626-WTpAE-Lo   21 nM bortezomib                   13 hr   \n",
       "\n",
       "  Metadata_cell_density  Metadata_Plate Metadata_Well  Metadata_Site_Count  \\\n",
       "0   2.5x10^3 cells/well          218360           B02                   16   \n",
       "1   2.5x10^3 cells/well          218360           B03                   16   \n",
       "2   2.5x10^3 cells/well          218360           B04                   16   \n",
       "\n",
       "   ...  Nuclei_Texture_Variance_RNA_10_02  Nuclei_Texture_Variance_RNA_10_03  \\\n",
       "0  ...                         134.630929                         136.168955   \n",
       "1  ...                         143.936790                         150.720200   \n",
       "2  ...                         139.837458                         147.532470   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_20_00  Nuclei_Texture_Variance_RNA_20_01  \\\n",
       "0                         130.178186                         143.230289   \n",
       "1                         155.195390                         182.867025   \n",
       "2                         143.564510                         152.944447   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_20_02  Nuclei_Texture_Variance_RNA_20_03  \\\n",
       "0                         130.452067                         139.230146   \n",
       "1                         153.439750                         162.839254   \n",
       "2                         143.430507                         147.967005   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_5_00  Nuclei_Texture_Variance_RNA_5_01  \\\n",
       "0                        135.932245                        133.168297   \n",
       "1                        146.040203                        143.993466   \n",
       "2                        140.323478                        141.193128   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_5_02  Nuclei_Texture_Variance_RNA_5_03  \n",
       "0                        137.391985                        135.095233  \n",
       "1                        147.316124                        147.909953  \n",
       "2                        140.788792                        140.561236  \n",
       "\n",
       "[3 rows x 3537 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in and preview what was output in the previous step \n",
    "annotated_df = pd.read_csv(annotate_file)\n",
    "\n",
    "print(annotated_df.shape)\n",
    "annotated_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Normalize well-level profiles\n",
    "\n",
    "In this step, **level 3 profiles** (well-level profiles) are normalized to form **level 4a profiles**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize profiles\n",
    "normalize_file = pathlib.Path(f\"{output_dir}/{plate_id}_normalized.csv.gz\")\n",
    "\n",
    "normalize(\n",
    "    profiles=annotate_file,\n",
    "    features=\"infer\",\n",
    "    meta_features=\"infer\",\n",
    "    samples=\"Metadata_treatment == '0.1% DMSO'\",\n",
    "    method=\"standardize\",\n",
    "    output_file=normalize_file,\n",
    "    compression_options=compression_options,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60, 3537)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metadata_plate_map_name</th>\n",
       "      <th>Metadata_clone_number</th>\n",
       "      <th>Metadata_plate_ID</th>\n",
       "      <th>Metadata_plate_filename</th>\n",
       "      <th>Metadata_treatment</th>\n",
       "      <th>Metadata_treatment_time</th>\n",
       "      <th>Metadata_cell_density</th>\n",
       "      <th>Metadata_Plate</th>\n",
       "      <th>Metadata_Well</th>\n",
       "      <th>Metadata_Site_Count</th>\n",
       "      <th>...</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_10_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_10_03</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_00</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_01</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_20_03</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_00</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_01</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_02</th>\n",
       "      <th>Nuclei_Texture_Variance_RNA_5_03</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>0.1% DMSO</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B02</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>1.994853</td>\n",
       "      <td>1.918636</td>\n",
       "      <td>2.082421</td>\n",
       "      <td>1.807392</td>\n",
       "      <td>2.111036</td>\n",
       "      <td>2.086668</td>\n",
       "      <td>1.950899</td>\n",
       "      <td>1.943007</td>\n",
       "      <td>1.988028</td>\n",
       "      <td>1.987186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>2.1 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B03</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>2.343617</td>\n",
       "      <td>2.448316</td>\n",
       "      <td>3.071418</td>\n",
       "      <td>3.091680</td>\n",
       "      <td>3.011963</td>\n",
       "      <td>2.951846</td>\n",
       "      <td>2.326277</td>\n",
       "      <td>2.351921</td>\n",
       "      <td>2.354838</td>\n",
       "      <td>2.466231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>21 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B04</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>2.189983</td>\n",
       "      <td>2.332280</td>\n",
       "      <td>2.611618</td>\n",
       "      <td>2.122145</td>\n",
       "      <td>2.619684</td>\n",
       "      <td>2.406839</td>\n",
       "      <td>2.113976</td>\n",
       "      <td>2.246140</td>\n",
       "      <td>2.113579</td>\n",
       "      <td>2.191518</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 3537 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Metadata_plate_map_name Metadata_clone_number  Metadata_plate_ID  \\\n",
       "0                   218360           WT parental             218360   \n",
       "1                   218360           WT parental             218360   \n",
       "2                   218360           WT parental             218360   \n",
       "\n",
       "  Metadata_plate_filename Metadata_treatment Metadata_treatment_time  \\\n",
       "0       20200626-WTpAE-Lo          0.1% DMSO                   13 hr   \n",
       "1       20200626-WTpAE-Lo  2.1 nM bortezomib                   13 hr   \n",
       "2       20200626-WTpAE-Lo   21 nM bortezomib                   13 hr   \n",
       "\n",
       "  Metadata_cell_density  Metadata_Plate Metadata_Well  Metadata_Site_Count  \\\n",
       "0   2.5x10^3 cells/well          218360           B02                   16   \n",
       "1   2.5x10^3 cells/well          218360           B03                   16   \n",
       "2   2.5x10^3 cells/well          218360           B04                   16   \n",
       "\n",
       "   ...  Nuclei_Texture_Variance_RNA_10_02  Nuclei_Texture_Variance_RNA_10_03  \\\n",
       "0  ...                           1.994853                           1.918636   \n",
       "1  ...                           2.343617                           2.448316   \n",
       "2  ...                           2.189983                           2.332280   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_20_00  Nuclei_Texture_Variance_RNA_20_01  \\\n",
       "0                           2.082421                           1.807392   \n",
       "1                           3.071418                           3.091680   \n",
       "2                           2.611618                           2.122145   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_20_02  Nuclei_Texture_Variance_RNA_20_03  \\\n",
       "0                           2.111036                           2.086668   \n",
       "1                           3.011963                           2.951846   \n",
       "2                           2.619684                           2.406839   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_5_00  Nuclei_Texture_Variance_RNA_5_01  \\\n",
       "0                          1.950899                          1.943007   \n",
       "1                          2.326277                          2.351921   \n",
       "2                          2.113976                          2.246140   \n",
       "\n",
       "   Nuclei_Texture_Variance_RNA_5_02  Nuclei_Texture_Variance_RNA_5_03  \n",
       "0                          1.988028                          1.987186  \n",
       "1                          2.354838                          2.466231  \n",
       "2                          2.113579                          2.191518  \n",
       "\n",
       "[3 rows x 3537 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in and preview what was output in the previous step \n",
    "normalized_df = pd.read_csv(normalize_file)\n",
    "\n",
    "print(normalized_df.shape)\n",
    "normalized_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Apply feature selection\n",
    "\n",
    "In this step, we apply a series of feature selection steps to **level 4a profiles** (normalized well-level profiles) to form **level 4b profiles**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply feature selection\n",
    "feature_select_file = pathlib.Path(f\"{output_dir}/{plate_id}_normalized_feature_select.csv.gz\")\n",
    "\n",
    "feature_select_opts = [\n",
    "    \"variance_threshold\",\n",
    "    \"drop_na_columns\",\n",
    "    \"correlation_threshold\",\n",
    "    \"blocklist\",\n",
    "    \"drop_outliers\"\n",
    "]\n",
    "\n",
    "feature_select(\n",
    "    profiles=normalize_file,\n",
    "    features=\"infer\",\n",
    "    samples=\"all\",\n",
    "    operation=feature_select_opts,\n",
    "    output_file=feature_select_file,\n",
    "    compression_options=compression_options,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60, 401)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metadata_plate_map_name</th>\n",
       "      <th>Metadata_clone_number</th>\n",
       "      <th>Metadata_plate_ID</th>\n",
       "      <th>Metadata_plate_filename</th>\n",
       "      <th>Metadata_treatment</th>\n",
       "      <th>Metadata_treatment_time</th>\n",
       "      <th>Metadata_cell_density</th>\n",
       "      <th>Metadata_Plate</th>\n",
       "      <th>Metadata_Well</th>\n",
       "      <th>Metadata_Site_Count</th>\n",
       "      <th>...</th>\n",
       "      <th>Nuclei_Texture_InfoMeas2_Mito_20_03</th>\n",
       "      <th>Nuclei_Texture_InfoMeas2_RNA_5_02</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_AGP_20_01</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_DNA_20_03</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_ER_20_03</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_Mito_20_03</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_RNA_20_00</th>\n",
       "      <th>Nuclei_Texture_SumVariance_ER_5_01</th>\n",
       "      <th>Nuclei_Texture_SumVariance_Mito_10_00</th>\n",
       "      <th>Nuclei_Texture_SumVariance_RNA_10_02</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>0.1% DMSO</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B02</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>0.666380</td>\n",
       "      <td>1.640270</td>\n",
       "      <td>-1.123220</td>\n",
       "      <td>-0.511368</td>\n",
       "      <td>-1.301264</td>\n",
       "      <td>-0.172448</td>\n",
       "      <td>-1.220708</td>\n",
       "      <td>1.750259</td>\n",
       "      <td>0.405271</td>\n",
       "      <td>1.982705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>2.1 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B03</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>1.022193</td>\n",
       "      <td>0.938770</td>\n",
       "      <td>-0.321421</td>\n",
       "      <td>-1.862488</td>\n",
       "      <td>-0.175040</td>\n",
       "      <td>0.309052</td>\n",
       "      <td>-0.832363</td>\n",
       "      <td>2.854113</td>\n",
       "      <td>2.696279</td>\n",
       "      <td>2.137837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>218360</td>\n",
       "      <td>WT parental</td>\n",
       "      <td>218360</td>\n",
       "      <td>20200626-WTpAE-Lo</td>\n",
       "      <td>21 nM bortezomib</td>\n",
       "      <td>13 hr</td>\n",
       "      <td>2.5x10^3 cells/well</td>\n",
       "      <td>218360</td>\n",
       "      <td>B04</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>1.378143</td>\n",
       "      <td>0.603039</td>\n",
       "      <td>-0.322174</td>\n",
       "      <td>-2.405300</td>\n",
       "      <td>0.392906</td>\n",
       "      <td>-1.128065</td>\n",
       "      <td>-0.716326</td>\n",
       "      <td>1.869677</td>\n",
       "      <td>8.215960</td>\n",
       "      <td>1.554196</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 401 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Metadata_plate_map_name Metadata_clone_number  Metadata_plate_ID  \\\n",
       "0                   218360           WT parental             218360   \n",
       "1                   218360           WT parental             218360   \n",
       "2                   218360           WT parental             218360   \n",
       "\n",
       "  Metadata_plate_filename Metadata_treatment Metadata_treatment_time  \\\n",
       "0       20200626-WTpAE-Lo          0.1% DMSO                   13 hr   \n",
       "1       20200626-WTpAE-Lo  2.1 nM bortezomib                   13 hr   \n",
       "2       20200626-WTpAE-Lo   21 nM bortezomib                   13 hr   \n",
       "\n",
       "  Metadata_cell_density  Metadata_Plate Metadata_Well  Metadata_Site_Count  \\\n",
       "0   2.5x10^3 cells/well          218360           B02                   16   \n",
       "1   2.5x10^3 cells/well          218360           B03                   16   \n",
       "2   2.5x10^3 cells/well          218360           B04                   16   \n",
       "\n",
       "   ...  Nuclei_Texture_InfoMeas2_Mito_20_03  \\\n",
       "0  ...                             0.666380   \n",
       "1  ...                             1.022193   \n",
       "2  ...                             1.378143   \n",
       "\n",
       "   Nuclei_Texture_InfoMeas2_RNA_5_02  \\\n",
       "0                           1.640270   \n",
       "1                           0.938770   \n",
       "2                           0.603039   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_AGP_20_01  \\\n",
       "0                                         -1.123220   \n",
       "1                                         -0.321421   \n",
       "2                                         -0.322174   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_DNA_20_03  \\\n",
       "0                                         -0.511368   \n",
       "1                                         -1.862488   \n",
       "2                                         -2.405300   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_ER_20_03  \\\n",
       "0                                        -1.301264   \n",
       "1                                        -0.175040   \n",
       "2                                         0.392906   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_Mito_20_03  \\\n",
       "0                                          -0.172448   \n",
       "1                                           0.309052   \n",
       "2                                          -1.128065   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_RNA_20_00  \\\n",
       "0                                         -1.220708   \n",
       "1                                         -0.832363   \n",
       "2                                         -0.716326   \n",
       "\n",
       "   Nuclei_Texture_SumVariance_ER_5_01  Nuclei_Texture_SumVariance_Mito_10_00  \\\n",
       "0                            1.750259                               0.405271   \n",
       "1                            2.854113                               2.696279   \n",
       "2                            1.869677                               8.215960   \n",
       "\n",
       "   Nuclei_Texture_SumVariance_RNA_10_02  \n",
       "0                              1.982705  \n",
       "1                              2.137837  \n",
       "2                              1.554196  \n",
       "\n",
       "[3 rows x 401 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in and preview what was output in the previous step \n",
    "feature_select_df = pd.read_csv(feature_select_file)\n",
    "\n",
    "print(feature_select_df.shape)\n",
    "feature_select_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Form consensus signatures\n",
    "\n",
    "In this step, we collapse replicates (**level 4 profiles**) into a single profile.\n",
    "This forms **level 5 profiles**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate consensus profiles\n",
    "consensus_file = pathlib.Path(f\"{output_dir}/{plate_id}_consensus.csv.gz\")\n",
    "\n",
    "consensus(\n",
    "    profiles=feature_select_df,\n",
    "    replicate_columns=[\"Metadata_clone_number\", \"Metadata_treatment\"],\n",
    "    operation=\"modz\",\n",
    "    features=\"infer\",\n",
    "    output_file=consensus_file,\n",
    "    modz_args={\n",
    "        \"method\": \"spearman\",\n",
    "        \"min_weight\": 0.01,\n",
    "        \"precision\": 4,\n",
    "    },\n",
    "    compression_options=compression_options,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(12, 392)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metadata_clone_number</th>\n",
       "      <th>Metadata_treatment</th>\n",
       "      <th>Cells_AreaShape_FormFactor</th>\n",
       "      <th>Cells_AreaShape_Orientation</th>\n",
       "      <th>Cells_AreaShape_Zernike_0_0</th>\n",
       "      <th>Cells_AreaShape_Zernike_1_1</th>\n",
       "      <th>Cells_AreaShape_Zernike_2_0</th>\n",
       "      <th>Cells_AreaShape_Zernike_3_1</th>\n",
       "      <th>Cells_AreaShape_Zernike_3_3</th>\n",
       "      <th>Cells_AreaShape_Zernike_4_0</th>\n",
       "      <th>...</th>\n",
       "      <th>Nuclei_Texture_InfoMeas2_Mito_20_03</th>\n",
       "      <th>Nuclei_Texture_InfoMeas2_RNA_5_02</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_AGP_20_01</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_DNA_20_03</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_ER_20_03</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_Mito_20_03</th>\n",
       "      <th>Nuclei_Texture_InverseDifferenceMoment_RNA_20_00</th>\n",
       "      <th>Nuclei_Texture_SumVariance_ER_5_01</th>\n",
       "      <th>Nuclei_Texture_SumVariance_Mito_10_00</th>\n",
       "      <th>Nuclei_Texture_SumVariance_RNA_10_02</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Clone A</td>\n",
       "      <td>0.1% DMSO</td>\n",
       "      <td>-0.012822</td>\n",
       "      <td>0.039280</td>\n",
       "      <td>-0.943973</td>\n",
       "      <td>-0.571392</td>\n",
       "      <td>-0.719932</td>\n",
       "      <td>0.448598</td>\n",
       "      <td>0.672615</td>\n",
       "      <td>0.166382</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.022600</td>\n",
       "      <td>-0.544882</td>\n",
       "      <td>1.048027</td>\n",
       "      <td>0.717373</td>\n",
       "      <td>0.690795</td>\n",
       "      <td>0.403234</td>\n",
       "      <td>0.838370</td>\n",
       "      <td>-0.635526</td>\n",
       "      <td>0.061329</td>\n",
       "      <td>-0.631957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Clone A</td>\n",
       "      <td>2.1 nM bortezomib</td>\n",
       "      <td>-1.800890</td>\n",
       "      <td>-0.281891</td>\n",
       "      <td>-1.001484</td>\n",
       "      <td>-0.746903</td>\n",
       "      <td>-0.355596</td>\n",
       "      <td>0.241331</td>\n",
       "      <td>-0.811865</td>\n",
       "      <td>0.549042</td>\n",
       "      <td>...</td>\n",
       "      <td>0.471521</td>\n",
       "      <td>-0.615206</td>\n",
       "      <td>0.592792</td>\n",
       "      <td>-0.401117</td>\n",
       "      <td>0.165698</td>\n",
       "      <td>-0.366404</td>\n",
       "      <td>0.488193</td>\n",
       "      <td>-0.067002</td>\n",
       "      <td>0.768437</td>\n",
       "      <td>-0.597989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Clone A</td>\n",
       "      <td>21 nM bortezomib</td>\n",
       "      <td>-3.368059</td>\n",
       "      <td>-0.130583</td>\n",
       "      <td>-1.832667</td>\n",
       "      <td>-0.117159</td>\n",
       "      <td>0.443206</td>\n",
       "      <td>1.054792</td>\n",
       "      <td>-3.051336</td>\n",
       "      <td>1.910167</td>\n",
       "      <td>...</td>\n",
       "      <td>1.247188</td>\n",
       "      <td>-0.767633</td>\n",
       "      <td>0.969311</td>\n",
       "      <td>-1.975002</td>\n",
       "      <td>0.059089</td>\n",
       "      <td>-1.450205</td>\n",
       "      <td>0.349720</td>\n",
       "      <td>2.162809</td>\n",
       "      <td>6.807487</td>\n",
       "      <td>-0.077198</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 392 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  Metadata_clone_number Metadata_treatment  Cells_AreaShape_FormFactor  \\\n",
       "0               Clone A          0.1% DMSO                   -0.012822   \n",
       "1               Clone A  2.1 nM bortezomib                   -1.800890   \n",
       "2               Clone A   21 nM bortezomib                   -3.368059   \n",
       "\n",
       "   Cells_AreaShape_Orientation  Cells_AreaShape_Zernike_0_0  \\\n",
       "0                     0.039280                    -0.943973   \n",
       "1                    -0.281891                    -1.001484   \n",
       "2                    -0.130583                    -1.832667   \n",
       "\n",
       "   Cells_AreaShape_Zernike_1_1  Cells_AreaShape_Zernike_2_0  \\\n",
       "0                    -0.571392                    -0.719932   \n",
       "1                    -0.746903                    -0.355596   \n",
       "2                    -0.117159                     0.443206   \n",
       "\n",
       "   Cells_AreaShape_Zernike_3_1  Cells_AreaShape_Zernike_3_3  \\\n",
       "0                     0.448598                     0.672615   \n",
       "1                     0.241331                    -0.811865   \n",
       "2                     1.054792                    -3.051336   \n",
       "\n",
       "   Cells_AreaShape_Zernike_4_0  ...  Nuclei_Texture_InfoMeas2_Mito_20_03  \\\n",
       "0                     0.166382  ...                            -0.022600   \n",
       "1                     0.549042  ...                             0.471521   \n",
       "2                     1.910167  ...                             1.247188   \n",
       "\n",
       "   Nuclei_Texture_InfoMeas2_RNA_5_02  \\\n",
       "0                          -0.544882   \n",
       "1                          -0.615206   \n",
       "2                          -0.767633   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_AGP_20_01  \\\n",
       "0                                          1.048027   \n",
       "1                                          0.592792   \n",
       "2                                          0.969311   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_DNA_20_03  \\\n",
       "0                                          0.717373   \n",
       "1                                         -0.401117   \n",
       "2                                         -1.975002   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_ER_20_03  \\\n",
       "0                                         0.690795   \n",
       "1                                         0.165698   \n",
       "2                                         0.059089   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_Mito_20_03  \\\n",
       "0                                           0.403234   \n",
       "1                                          -0.366404   \n",
       "2                                          -1.450205   \n",
       "\n",
       "   Nuclei_Texture_InverseDifferenceMoment_RNA_20_00  \\\n",
       "0                                          0.838370   \n",
       "1                                          0.488193   \n",
       "2                                          0.349720   \n",
       "\n",
       "   Nuclei_Texture_SumVariance_ER_5_01  Nuclei_Texture_SumVariance_Mito_10_00  \\\n",
       "0                           -0.635526                               0.061329   \n",
       "1                           -0.067002                               0.768437   \n",
       "2                            2.162809                               6.807487   \n",
       "\n",
       "   Nuclei_Texture_SumVariance_RNA_10_02  \n",
       "0                             -0.631957  \n",
       "1                             -0.597989  \n",
       "2                             -0.077198  \n",
       "\n",
       "[3 rows x 392 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in and preview what was output in the previous step \n",
    "consensus_df = pd.read_csv(consensus_file)\n",
    "\n",
    "print(consensus_df.shape)\n",
    "consensus_df.head(3)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pycytominer-example] *",
   "language": "python",
   "name": "conda-env-pycytominer-example-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}