Microplant_Mystery_Zooniverse_Processing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/emcdona1/fmnh_scripts/blob/main/Microplant_Mystery_Zooniverse_Processing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IGnWx_ka-xhK"
      },
      "source": [
        "## Processing Your Zooniverse Project\n",
        "\n",
        "This Colab script is designed specifically for the [Microplant Mystery Zooniverse Project](https://www.zooniverse.org/projects/nvuitton/unfolding-of-microplant-mysteries/).  It _may_ work for other projects, but I would strongly suggest creating a new copy of this script for each project (that way you can modify it and not break this one!).\n",
        "\n",
        "### Preparation steps:\n",
        "\n",
        "  1. Download your Classifications CSV from Zooniverse.\n",
        "  2. Go through this file and remove any old versions of your workflows.  (e.g. use Excel or Google Sheets)  (see section below for details)\n",
        "  3.  Upload the new CSV file to your Google Drive.\n",
        "\n",
        "-----\n",
        "\n",
        "### Specific preprocessing suggestions for Microplant Mystery:\n",
        "  1. Open the raw CSV file in Excel.\n",
        "  2. Select all cells (ctrl+A), then turn on filtering (a funnel-shaped icon).\n",
        "  3. In the `workflow_name` column, select ONLY \"Microplant shapes of Inflated Sacs.\"  Hit OK.\n",
        "  4. Delete all rows of that workflow.  (Click the row number on the left side to select the whole row, right-click to delete.)\n",
        "  5. Turn off filtering (click the icon again).\n",
        "  6. Determine the start date you'd like to use (e.g. the date the project officially launched).\n",
        "  7. Looking at the \"created_at\" column, delete all rows BEFORE your start date.\n",
        "  8. Save the file as a new CSV.\n",
        "  9. Upload file to your Google Drive.\n",
        "\n",
        "-----\n",
        "\n",
        "### Google Colab steps:\n",
        "\n",
        "**1. Press the play button on the cell below to start up Google Colab and connect it to your Google Drive.**\n",
        "\n",
        "   1. You may be prompted to open up a URL (blue link) - click the link and follow the prompts to authorize your Google account.\n",
        "   2. After you connect, you'll see an Authorization Code (long string of gibberish), copy and paste this text into the Colab prompt box (below).\n",
        "   3. Click Enter and wait 5-10 seconds while Google Drive connects."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Aa_4k-x978L3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bd811708-d6b6-4dfd-9fae-eb9b0f8bdeca"
      },
      "source": [
        "import os\n",
        "import sys\n",
        "import io\n",
        "import shutil\n",
        "import ast\n",
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "from statistics import mode, StatisticsError\n",
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/drive')\n",
        "drive_folder = Path('/content/drive/MyDrive')"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F2bAF2p5-u1b"
      },
      "source": [
        "**2. Upload your Zooniverse classification file:**\n",
        "\n",
        "When prompted, type the name of the CSV file in your Google Drive -- be careful to type the name exactly!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "ouYvywb_LDOe",
        "outputId": "5cffd23e-6040-4308-dfd7-fb96005ec5ef"
      },
      "source": [
        "csv_file = input('Type the name of your CSV file: ')\n",
        "input_filename = Path(csv_file)\n",
        "csv_file_path = Path(drive_folder, csv_file)\n",
        "output_filename = f'{input_filename.stem}-processed'\n",
        "if not os.path.exists(csv_file_path):\n",
        "    print(f'WARNING!  The file \"{csv_file}\" is not in your Google Drive!  Did you type the name correctly?  Did it finish uploading?')"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Type the name of your CSV file: unfolding-of-microplant-mysteries-classifications (1).csv\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'unfolding-of-microplant-mysteries-classifications (1)-processed'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JeW35BAsVHiX"
      },
      "source": [
        "**3. Apply some basic formatting to the CSV file, extract the metadata, and identify the workflows.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1XoYOCIG-5hM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "764011a0-0e3c-4438-e3a1-09d444dab94f"
      },
      "source": [
        "zooniverse_classifications = pd.read_csv(csv_file_path)\n",
        "def clean_formatting_and_parse(data: str):\n",
        "    data = data.replace('null', 'None')\n",
        "    data = data.replace('true', 'True')\n",
        "    data = data.replace('false', 'False')\n",
        "    return ast.literal_eval(data)\n",
        "\n",
        "zooniverse_classifications.loc[:, 'annotations'] = zooniverse_classifications['annotations'].apply(clean_formatting_and_parse)\n",
        "zooniverse_classifications.loc[:, 'metadata'] = zooniverse_classifications['metadata'].apply(clean_formatting_and_parse)\n",
        "zooniverse_classifications.loc[:, 'subject_data'] = zooniverse_classifications['subject_data'].apply(clean_formatting_and_parse)\n",
        "\n",
        "zooniverse_classifications['metadata-user_agent'] = zooniverse_classifications['metadata'].apply(lambda d: d['user_agent'])\n",
        "zooniverse_classifications['image_filename'] = zooniverse_classifications['subject_data'].apply(lambda d: d[list(d.keys())[0]]['Filename'])\n",
        "\n",
        "workflows = list(set(zooniverse_classifications.loc[:, 'workflow_name']))\n",
        "print('Zooniverse Workflows found:')\n",
        "for workflow in workflows:\n",
        "  print(f'-- {workflow}')"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Zooniverse Workflows found:\n",
            "-- Determining the Reproductive Structure of a Liverwort\n",
            "-- Stem and Branching Patterns \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9nbmscJLVG0A"
      },
      "source": [
        "**4. Parse the tasks (from annotations column) into separate columns.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oSGVYY6XFrFr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "511ba1c9-1fc5-48e5-9634-1da5fb9da56b"
      },
      "source": [
        "results_by_workflow = list()\n",
        "for workflow_name in workflows:\n",
        "    results = zooniverse_classifications[zooniverse_classifications['workflow_name'] == workflow_name].copy()\n",
        "    max_count = 0\n",
        "    for idx, row in results.iterrows():\n",
        "        tasks = row['annotations']\n",
        "        for count, task in enumerate(tasks):\n",
        "            name = f'Task{count+1}'\n",
        "            results.loc[idx, f'{name}-id'] = task['task']\n",
        "            results.loc[idx, f'{name}-label'] = task['task_label']\n",
        "            results.loc[idx, f'{name}-value'] = str(task['value'])\n",
        "        if count > max_count: max_count = count\n",
        "    results_by_workflow.append(results)\n",
        "    print(f'{workflow_name.strip()} expanded into {max_count+1} task(s).')"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Determining the Reproductive Structure of a Liverwort expanded into 2 task(s).\n",
            "Stem and Branching Patterns expanded into 1 task(s).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z04XXlZoHpmi"
      },
      "source": [
        "**5. Extract the (x, y) coordinates.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LywBUkNBVrOi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "de2436a5-313c-4557-f8c6-80657c834efd"
      },
      "source": [
        "def extract_coordinate_info(workflow, idx, pt_info, colname):\n",
        "    workflow.loc[idx, f'{colname}-x'] = pt_info['x']\n",
        "    workflow.loc[idx, f'{colname}-y'] = pt_info['y']\n",
        "    workflow.loc[idx, f'{colname}-width'] = pt_info['width']\n",
        "    workflow.loc[idx, f'{colname}-height'] = pt_info['height']\n",
        "    workflow.loc[idx, f'{colname}-value_label'] = pt_info['tool_label']\n",
        "\n",
        "for workflow_name, workflow in zip(workflows, results_by_workflow):\n",
        "    value_columns = [val for val in list(workflow.columns) if 'value' in val]\n",
        "    for idx, row in workflow.iterrows():\n",
        "        for col in value_columns:\n",
        "            try:\n",
        "                points = ast.literal_eval(row[col])\n",
        "                if type(points) == list:\n",
        "                    for pt_idx, point in enumerate(points):\n",
        "                        colname = col.split('-')[0] + '-Pt' + str(pt_idx + 1)\n",
        "                        extract_coordinate_info(workflow, idx, points[0], colname)\n",
        "                workflow.loc[idx, col] = None\n",
        "            except: pass\n",
        "    found_points = list(set([a.split('-')[0]+'-'+a.split('-')[1] for a in list(workflow.columns) if 'Pt' in a]))\n",
        "    print(f'Workflow: {workflow_name}')\n",
        "    if len(found_points) > 0:\n",
        "        print(f'-- Point info found:', found_points)\n",
        "    else:\n",
        "        print('-- No point data found.')\n",
        "    print()"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/pandas/core/indexing.py:1684: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead.  To get a de-fragmented frame, use `newframe = frame.copy()`\n",
            "  self.obj[key] = infer_fill_value(value)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Workflow: Determining the Reproductive Structure of a Liverwort\n",
            "-- Point info found: ['Task2-Pt1', 'Task2-Pt33', 'Task2-Pt39', 'Task2-Pt45', 'Task2-Pt5', 'Task2-Pt26', 'Task2-Pt11', 'Task2-Pt7', 'Task2-Pt28', 'Task2-Pt23', 'Task2-Pt16', 'Task2-Pt30', 'Task2-Pt20', 'Task2-Pt25', 'Task2-Pt27', 'Task2-Pt40', 'Task2-Pt43', 'Task2-Pt31', 'Task2-Pt29', 'Task2-Pt42', 'Task2-Pt24', 'Task2-Pt17', 'Task2-Pt48', 'Task2-Pt55', 'Task2-Pt53', 'Task2-Pt49', 'Task2-Pt2', 'Task2-Pt13', 'Task2-Pt21', 'Task2-Pt32', 'Task2-Pt12', 'Task2-Pt22', 'Task2-Pt50', 'Task2-Pt14', 'Task2-Pt54', 'Task2-Pt59', 'Task2-Pt9', 'Task2-Pt4', 'Task2-Pt35', 'Task2-Pt57', 'Task2-Pt60', 'Task2-Pt62', 'Task2-Pt37', 'Task2-Pt44', 'Task2-Pt10', 'Task2-Pt46', 'Task2-Pt56', 'Task2-Pt36', 'Task2-Pt8', 'Task2-Pt38', 'Task2-Pt34', 'Task2-Pt47', 'Task2-Pt58', 'Task2-Pt6', 'Task2-Pt19', 'Task2-Pt41', 'Task2-Pt51', 'Task2-Pt52', 'Task2-Pt3', 'Task2-Pt61', 'Task2-Pt15', 'Task2-Pt18']\n",
            "\n",
            "Workflow: Stem and Branching Patterns \n",
            "-- No point data found.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "input_filename.parent"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rsjdtI6PY69m",
        "outputId": "33d21b4b-2130-4f74-e5a5-ab6b8f03dbc7"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('.')"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IUo5wQGIVYPf"
      },
      "source": [
        "**6. Save the files to your Google Drive in a new folder.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FI0B-yAl8WT_",
        "outputId": "d97e675d-75bb-4f48-de4a-dd21c4757a24",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "results_folder = Path(drive_folder, input_filename.parent, f'{input_filename.stem}_processed')\n",
        "if not os.path.exists(results_folder):\n",
        "  os.makedirs(results_folder)\n",
        "for (workflow_name, workflow) in zip(workflows, results_by_workflow):\n",
        "  file_location = Path(results_folder, f'{workflow_name.strip()}.csv')\n",
        "  workflow.to_csv(file_location, index=False, encoding='UTF-8')\n",
        "\n",
        "# save_location = shutil.make_archive(output_filename, 'zip', results_folder)\n",
        "print(f'Saved to {results_folder} in your Google Drive.')"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saved to /content/drive/MyDrive/unfolding-of-microplant-mysteries-classifications (1)_processed in your Google Drive.\n"
          ]
        }
      ]
    }
  ]
}