b_data_collection_methods/1_api_harvesting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "26acf247",
   "metadata": {},
   "source": [
    "<img src='images/header.png' style='height: 50px; float: left'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "304c55db",
   "metadata": {},
   "source": [
    "## Introduction to Computational Social Science methods with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cd4ea4a",
   "metadata": {},
   "source": [
    "# Session B1: API harvesting\n",
    "\n",
    "In January 2023, the most popular social media platforms were Facebook, YouTube, WhatsApp, Instagram, and WeChat (Statista, 2023). The web and its **online platforms** provide vast amounts of data that are highly relevant and interesting for social research. Whether generated in social media platforms, search engines (*e.g.*, Google), or knowledge production platforms (*e.g.*, Wikipedia, GitHub), the data resembles **digital traces** of behavior that are, as a first approximation, unobtrusive (*i.e.*, not influenced by observational or memory effects), complete (*i.e.*, a non-probabilistic sample), and highly resolved (*i.e.*, in real time and at scale). This provides a unique opportunity to study human behaviour in naturalistic settings (<a href='#lazer_computational_2009'>Lazer *et al.*, 2009</a>).\n",
    "\n",
    "Obtaining Digital Behavioral Data (DBD) from online platforms, however, is a whole different story. If you are lucky, then the dataset you need for your research has already been collected and dumped on a website or stored in a data archive. If your dream dataset has not been pre-collected, you must do it yourself. For those purposes, you can use the [APIs](https://en.wikipedia.org/wiki/API) which large online platforms typically provide. In general, an **Application Programming Interface** (API) is a piece of software that helps you facilitate communication among computer programs. When you type a domain name in your browser, you use an API that helpy you obtain information from a computer far away and spares you of having to type its IP address. Even Python packages like Pandas are APIs because they assist programmers in speeding up their coding by providing a set of pre-programmed functions that perform commonly needed operations without the need to write them from scratch.\n",
    "\n",
    "Providers of large digital platforms typically provide APIs for the use of which you can apply as a researcher. In many cases, users must undergo a vetting procedure in which the goals and procedures of the project are described to the providers. Once you have access, APIs are usually accessed through wrappers that facilitate the interaction with the API through a programming language like Python. Essentially, **wrappers** are overlays that communicate with the API for us but are more convenient to the users due to easier implementations of automating requests. There are two main downsides to using APIs. First, most APIs have restrictions on data types and how much data they provide. Such restrictions are also often tiered, in which free access provides the least amount of data, while higher tiers provide a wider variety of data types and larger amounts of data. Second, APIs change (<a href='#junger_a_2022'>Jünger, 2022</a>; <a href='#mclevey_doing_2022'>McLevey, 2022</a>, ch. 4).\n",
    "\n",
    "Recently, API change has become a tremendous problem for social research. In 2009, it still seemed that collaborations between platform operators and academic institutions could guarantee both data access and user privacy (<a href='#lazer_computational_2009'>Lazer *et al.*, 2009</a>). The two giants Meta (the company running Facebook) and Twitter had both set up **academic APIs**, but after years of experiments both terminated them in 2022 and 2023, respectively. We have definitely arrived in the \"Post-API Age\" (Freelon, 2018). Platform operators are private companies whose business models do not align well with free data access for researchers, journalists, or those who openly work in the public interest. Twitter has since been renamed X and now charges \\\\$42,000 for 50M requests. These developments have plummeted Computational Social Science into a **reproducibility crisis** (<a href='#davidson_social_2023'>Davidson *et al.*, 2023</a>) and are causing the field to invest more into other data collection methods like data scraping techniques, browser extensions, or user data donations, potentially provided by centralized infrastructures (<a href='#lazer_computational_2020'>Lazer *et al.*, 2020</a>).\n",
    "\n",
    "Nevertheless, given the many digital platforms and APIs out there, API harvesting stays an important data collection method. This is particularly the case for non-commercial platforms like Wikipedia. To ensure reproducibility and **data quality**, the characteristics of collected – not just harvested – datasets should be transparently documented and communicated. Just like in survey research, computational scientists are advised to reflect upon the challenges associated with the collection of digital traces, the underlying population that produced them, the meaning encoded in these traces, and the role of the platform in the trace generation process (<a href='#sen_a_2021'>Sen *et al.*, 2021</a>). DBD can only be complete with respect to the trace-producing population, and platform effects render it obtrusive in its very own meaning. The Total Error Sheets for Datasets (TES-D) framework is a critical guide to documenting online platform datasets (<a href='#frohling_total_2023'>Fröhling *et al.*, 2023</a>).\n",
    "\n",
    "<div class='alert alert-block alert-success'>\n",
    "<b>In this session</b>, \n",
    "\n",
    "you will learn how to collect Digital Behavioral Data via API harvesting. In subsession **B1.1**, we will list resources for social media APIs and for datasets that have already been collected. In subsession **B1.2**, we will dive into harvesting Wikipedia, introducing a few APIs that help with collecting various parts of Wikipedia pages. Finally, in subsession **B1.3**, we will discuss the Total Error Sheets for Datasets (TES-D) framework to document a Twitter dataset.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56c76de5",
   "metadata": {},
   "source": [
    "## B1.1. APIs and precollected datasets\n",
    "\n",
    "<img src=\"./images/datasets.jpg\" width=\"500\" height = \"900\" align=\"left\"/>  \n",
    "\n",
    "- __Awesome list__\n",
    "- __More APIs__\n",
    "\n",
    "    [Facebook for Developers](https://developers.facebook.com/)  \n",
    "    [Facebook Ads API](https://developers.facebook.com/docs/marketing-apis/)  \n",
    "    [Instagram Developer](https://developers.facebook.com/docs/instagram-basic-display-api)  \n",
    "    [YouTube Developers](https://developers.google.com/youtube/)  \n",
    "    [Weibo API](http://open.weibo.com/wiki/API%E6%96%87%E6%A1%A3/en)  \n",
    "    [CrowdTangle](https://www.crowdtangle.com/request)  \n",
    "    [4chan](https://github.com/4chan/4chan-API)  \n",
    "    [Gab](https://github.com/a-tal/gab)  \n",
    "    [Github REST API](https://docs.github.com/en/rest)  \n",
    "    [Github GraphQL](https://docs.github.com/en/graphql)  \n",
    "    [Stackoverflow](https://api.stackexchange.com/docs)  \n",
    "    [Facepager](https://github.com/strohne/Facepager)  \n",
    "\n",
    "\n",
    "- __Precollected datasets__  \n",
    "    https://datasetsearch.research.google.com  \n",
    "    https://www.kaggle.com/datasets  \n",
    "    https://data.gesis.org/sharing/#!Search  \n",
    "\n",
    "\n",
    "- __Locating or Requesting Social Media Data__\n",
    "    https://www.programmableweb.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2ac0efa",
   "metadata": {},
   "source": [
    "## B1.2. Harvesting Wikipedia\n",
    "\n",
    "<img src='./images/wikipedia_logo.png' style='height: 190px; float: right; margin-left: 50px' >\n",
    "\n",
    "Wikipedia is a rich source of data for social science research. Although we can access its data through other techniques like web scraping, there are also useful APIs that could ease collecting data from the website.\n",
    "\n",
    "Since Wikipedia is built on [MediaWiki](https://en.wikipedia.org/wiki/MediaWiki), we will be using python wrappers written for its API,\n",
    "[Mediawiki Action API](https://www.mediawiki.org/wiki/API:Main_page). Each of these wrappers provide some useful methods, and we will try to go through the ones that are the most important to our data collection tasks.\n",
    "\n",
    "We will also introduce two useful parsers for the Wikipedia markup language, and will see how they could be used for extracting clean data from the raw markup code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44e00226",
   "metadata": {},
   "source": [
    "### B1.2.1. wikipedia\n",
    "\n",
    "The first wrapper we introduce here is simply called [wikipedia](https://wikipedia.readthedocs.io/en/latest/code.html#api)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3f28f71",
   "metadata": {},
   "outputs": [],
   "source": [
    "import wikipedia as wp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb7c2a9e",
   "metadata": {},
   "source": [
    "Searching a query with `wikipedia` can be done using the [`search()`](https://wikipedia.readthedocs.io/en/latest/code.html#api) function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9515cd52",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp.search(\"seattle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45a31e0b",
   "metadata": {},
   "source": [
    "You can get fewer or more results with a specific number like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f09823a",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp.search(\"seattle\", results=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "255241bd",
   "metadata": {},
   "source": [
    "Wikipedia's suggested query can be accessed with the [`suggest()`](https://wikipedia.readthedocs.io/en/latest/code.html#api) function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be845c05",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp.suggest(\"seattle\") # what does it do?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07f2e934",
   "metadata": {},
   "source": [
    "For getting the summary of an article, you can use the [`summary()`](https://wikipedia.readthedocs.io/en/latest/code.html#api) function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c86bc0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wp.summary(\"Chief Seattle\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cad19a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wp.summary(\"Chief Seattle\", sentences=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23126943",
   "metadata": {},
   "source": [
    "`summary()` will raise a `DisambiguationError` if the page is a disambiguation page, or a `PageError` if the page doesn’t exist (although by default, it tries to find the page you meant with suggest and search.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63b34048",
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(wp.summary(\"Mercury\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "328ba9ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    wp_summary = print(wp.summary(\"Mercury\"))\n",
    "except wp.exceptions.DisambiguationError as e:\n",
    "    print(e.options)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c68225c",
   "metadata": {},
   "source": [
    "The [`page()`](https://wikipedia.readthedocs.io/en/latest/code.html#api)function enables you to load and access data from full Wikipedia pages. Initialize with a page title (keep in mind the errors listed above), then you can easily access most properties of the page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecc46d21",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_page = wp.page(\"Chief Seattle\")\n",
    "wp_page"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48a6cc06",
   "metadata": {},
   "source": [
    "HTML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9872def1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import HTML\n",
    "\n",
    "HTML(wp_page.html())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d60b78ad",
   "metadata": {},
   "source": [
    "You can get information like title of the page, its url etc. In order to get the title of the page, you can use the `title` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "053d27d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_page.title"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ee909db",
   "metadata": {},
   "source": [
    "Using the `url` attribute, you can get the url of the page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa3f6b52",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_page.url"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea8d8d41",
   "metadata": {},
   "source": [
    "To get the full text of the page, you can use the `content` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad0a9a7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wp_page.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a76438a",
   "metadata": {},
   "source": [
    "In order to access the plain text content of a section in the page, you can use the `sections` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c238130",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_page.sections # should work but doesn't"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05dd5424",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wp_page.section('Biography'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1ae2bb0",
   "metadata": {},
   "source": [
    "You can access the images in the page using `.images`. The URLs of the first five images are retrieved like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5672abf",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_page.images[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee8b7237",
   "metadata": {},
   "source": [
    "In order to get the URLs of the external links of the page, you can use `.references`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88e25499",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_page.references[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "582b4d7c",
   "metadata": {},
   "source": [
    "You can get the texts of the links in the page using `.links`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28a7b7c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_page.links[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "030c9b82",
   "metadata": {},
   "source": [
    "Categories (where from?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7760af3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_page.categories[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c38f2549",
   "metadata": {},
   "source": [
    "Dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dfa7f18",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cdea228",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(\n",
    "    data = [[wp_page.title, wp_page.url, wp_page.content, wp_page.images, wp_page.references, wp_page.links, wp_page.categories]], \n",
    "    columns = ['Title', 'URL', 'Content', 'Images', 'References', 'Links', 'Categories']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a738f891",
   "metadata": {},
   "source": [
    "In order to change the language of the Wikipedia pages you are accessing, you can use the [`set_lang()`](https://wikipedia.readthedocs.io/en/latest/code.html#api) function. Remember to search for page titles in the language that you have set, not English:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf8edbe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp.set_lang(\"es\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5b4d72e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wp.summary(\"Chief Seattle\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67f437b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "wp.set_lang(\"en\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d12e4cdb",
   "metadata": {},
   "source": [
    "### B1.2.2. Harvesting tables\n",
    "\n",
    "The `wikipedia` package that we introduced in B1.2.1 cannot always help us with all the tasks we may want to do in order to collect data from Wikipedia.\n",
    "\n",
    "For getting data other than what `wikipedia` can give us, we can use other libraries to access the markup code of Wikipedia, and then parse it to get the information we want. We will introduce [pywikibot](https://doc.wikimedia.org/pywikibot/stable/), a wrapper that can give us the markup, together with two parsers [mwparserfromhell](https://mwparserfromhell.readthedocs.io/en/latest/index.html) and [wikitextparser](https://wikitextparser.readthedocs.io/en/latest/), in order to parse the markup code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52914658",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pywikibot as pwb\n",
    "import wikitextparser as wtp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3de06950",
   "metadata": {},
   "source": [
    "We will begin with an example page: [List of political parties in Germany](https://en.wikipedia.org/wiki/List_of_political_parties_in_Germany). We want to extract the tables data in that page. Using pywikibot, we can get the markup code of the page, and then parse it with wikitextparser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b199e51",
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd_site = pwb.Site('en', 'wikipedia') # The site we want to run our bot on\n",
    "pwb_page = pwb.Page(pwd_site, \"List of political parties in Germany\")\n",
    "pwb_text = pwb_page.text\n",
    "print(pwb_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27bed284",
   "metadata": {},
   "source": [
    "In order to parse the table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebab4dd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "wtp_text = wtp.parse(pwb_text)\n",
    "wtp_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4be93c7f",
   "metadata": {},
   "source": [
    "We can get the tables data with `page.tables`. Let's say we want to get the first table's data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "311d823b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "wtp_first_table = wtp_text.tables[0].data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79af55af",
   "metadata": {},
   "source": [
    "By putting the data in a dataframe, we can have a better overview of it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c30e4e02",
   "metadata": {},
   "outputs": [],
   "source": [
    "first_table = pd.DataFrame(wtp_first_table[1:])\n",
    "first_table.columns = wtp_first_table[0]\n",
    "first_table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bb05f3f",
   "metadata": {},
   "source": [
    "As you can see, the cells data are not shown in a clean way, like the way they are in the original Wikipedia page. We can parse each cell's data with mwparserfromhell, and then create the dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "535d8d6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mwparserfromhell as mwp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02f2104b",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(wtp_first_table)):\n",
    "    for j in range(len(wtp_first_table[i])):\n",
    "        wikicode = mwp.parse(wtp_first_table[i][j])\n",
    "        wtp_first_table[i][j] = wikicode.strip_code(wtp_first_table[i][j])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef8f925f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "first_table = pd.DataFrame(wtp_first_table[1:])\n",
    "first_table.columns = wtp_first_table[0]\n",
    "first_table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed4f08c7",
   "metadata": {},
   "source": [
    "Now the table looks pretty much the same as the table in the original page."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d020a393",
   "metadata": {},
   "source": [
    "#### An alternative for extracting tables data: wikitables library\n",
    "\n",
    "In order to get table's data, you can also get help from `wikitables` library. It eases some steps of accessing the tables data, but you need to be careful with small bugs or mistakes in the resulting tables. Let's say we want to extract the second table's data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fef55693",
   "metadata": {},
   "outputs": [],
   "source": [
    "from wikitables import import_tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c553191",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tables = import_tables('List of political parties in Germany')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf3603a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "first_table_wt = pd.DataFrame(tables[0].rows)\n",
    "first_table_wt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb94b022",
   "metadata": {},
   "source": [
    "As you can see, ... This needs to be taken care of, in case you want to use `wikitables`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3783843f",
   "metadata": {},
   "source": [
    "### B1.2.3. Extracting main text of different revisions\n",
    "\n",
    "There may be multiple different revisions available for each Wikipedia page. In this section, we will demonstrate how you can extract the main text of the first revision of an article in each year since the beginning, using `pywikibot` and `mwparserfromhell`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c3a00da",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pywikibot\n",
    "import mwparserfromhell"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9620e76",
   "metadata": {},
   "source": [
    "Like before, you can first get the page using pwwikibot's [`.Site()`](https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.site.html#module-site) and [`.Page()`](https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.site.html#module-site):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a45428be",
   "metadata": {},
   "outputs": [],
   "source": [
    "site = pywikibot.Site('en', 'wikipedia')\n",
    "page = pywikibot.Page(site, \"Koç University\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "321674b6",
   "metadata": {},
   "source": [
    "Then, you can get all the revisions of the page using `page.revisions()`. Depending on how old/rich the page is, this may take a few seconds:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3b8a09f",
   "metadata": {},
   "outputs": [],
   "source": [
    "revisions = page.revisions(content=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4d738dc",
   "metadata": {},
   "source": [
    "Now we can make a list of all of the revisions, and put the **year** in which each revision has been written into a `years` list. Each revision is in the form of a dictionary, and we can get the *years* using the `timestamp` key in those dictionaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73d99ea6",
   "metadata": {},
   "outputs": [],
   "source": [
    "revisions_list = []\n",
    "years = []\n",
    "\n",
    "for i in revisions:\n",
    "    revisions_list.append(i)\n",
    "    years.append(int(str(i['timestamp'])[:4]))\n",
    "years.reverse()\n",
    "revisions_list.reverse()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e2ea51",
   "metadata": {},
   "source": [
    "Since revisions are sorted from the newest to the eldest, we have to reverse the `years` and `revisions_list` lists to have their items in an ascending order. By printing the `years` list, you can see an overview of how many revisions in each year there are for the page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6edb982e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(years)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30bd00dd",
   "metadata": {},
   "source": [
    "We want to put the first revision of each year into a `yearly_revisions` list. In order to do that, we first get the indices of the first appearances of each year in the `years` list, and get the revisions with those indices in the `revisions_list` list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f4963af",
   "metadata": {},
   "outputs": [],
   "source": [
    "yearly_revisions = []\n",
    "for i in range(years[0], years[-1]+1):\n",
    "    index = years.index(i)\n",
    "    yearly_revisions.append(revisions_list[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "394409f4",
   "metadata": {},
   "source": [
    "In order to get the clean main text of each revision, we can use the `text` attribute of the revisions, and have the result parsed using `mwparserfromhell`. Take the last revision as an example; we first put the un-parsed code into the `text` variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dafeb949",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = yearly_revisions[-1].text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "642fbc75",
   "metadata": {},
   "source": [
    "Now we can parse it with `mwparserfromhell` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "899c8dec",
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed = mwparserfromhell.parse(text)\n",
    "print(parsed.strip_code())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9b9991f",
   "metadata": {},
   "source": [
    "## B1.3. Documentation of datasets collected from online platforms\n",
    "\n",
    "In the following we would like to show you how to describe systematically digital behavioral data. For this purpose we will utilize TES-D template (ADD citation; <a href='#Fröhling'>Fröhling et al., 2023</a>; <a href='#Sen'>Sen et al., 2021</a>). For more details you can refer to TES-D Manual (ADD citation).\n",
    "\n",
    "**TES-D “Computational Social Science Turkey Tweets 2008-2023”**\n",
    "\n",
    "**General Characteristics** \n",
    "\n",
    "1. *Who collected the dataset and who funded the process?*\n",
    "\n",
    "The dataset have been collected by \"Social ComQuant\" Project team (Gizem Bacaksizlar Turbic, Haiko Lietz, Pouria Mirelmi, Olga Zagovora) at GESIS - Leibniz Institute for the Social Sciences, Computational Social Science department. The dataset collection was funded by a European Commission as a part of [the Social ComQuant Project](https://socialcomquant.ku.edu.tr/).\n",
    "\n",
    "2. *Where is the dataset hosted? Is the dataset distributed under a copyright or license?* \n",
    "\n",
    "The dataset is hosted on open access [github repository](https://github.com/gesiscss/css_methods_python) of CSS department at GESIS. ADD LICENSE   \n",
    "\n",
    "3. *What do the instances that comprise the dataset represent? What data does each instance consist of?*\n",
    "\n",
    "Each line of dataset reprents a distinct Tweet posted on Twitter in the period between 5th January 2008 and 8th January 2023. Each instance consist of: the unique identifier of the Tweet, the unique identifier of the User who posted this Tweet, creation time of the Tweet (in ISO 8601 format), the actual UTF-8 text of the Tweet, language of the Tweet, if detected by Twitter (it is returned as a BCP47 language tag). Data was not prerocessed and is represented in formats provided by API. \n",
    "\n",
    "4. *How many instances are there in total in each category (as defined by the instances’ label), and - if applicable - in each recommended data split?*\n",
    "\n",
    "There are 105 instances on the dataset. Instances are homogen, i.e., each of them is representing a Tweet. \n",
    "\n",
    "5. *In which contexts and publications has the dataset been used already?* \n",
    "\n",
    "The dataset have been used in the online materials of [the Introduction to Computational Social Science methods with Python](https://github.com/gesiscss/css_methods_python) Course. \n",
    "\n",
    "6. *Are there alternative datasets that could be used for the measurement of the same or similar constructs? Could they be a better fit? How do they differ?* \n",
    "\n",
    "The dataset have been created for teaching purpose, namely, exercise on getting data using API. Any similar dataset is unknown. \n",
    "\n",
    "7. *Can the dataset collection be readily reproduced given the current data access, the general context and other potentially interfering developments?*\n",
    "\n",
    "[Jupyter Notebook](https://github.com/gesiscss/css_methods_python/blob/main/b_data_collection_methods/1_API_harvesting.ipynb), subsection B1.2.4 provides code in Python that explain how to obtain the dataset. Be aware that Twitter API might be depricated due to changes in Policies on free Access to the API. All the relevant informatiom one can find in the [documentation](https://developer.twitter.com/en/docs) or in this news article [Why Twitter ending free access to its APIs should be a ‘wake-up call’](https://www.theguardian.com/technology/2023/feb/07/techscape-elon-musk-twitter-api).   \n",
    "\n",
    "8. *Were any ethical review processes conducted?* \n",
    "\n",
    "No thical review processes have been conducted. Dataset do not consist of any Private Data.    \n",
    "\n",
    "9. *Did any ethical considerations limit the dataset creation?* \n",
    "\n",
    "We have not stored any data related to user accounts that have been posting relevant Tweets. Storage of this data can cause additional ethical considerations. \n",
    "\n",
    "10. *Are there any potential risks for individuals using the data?* \n",
    "\n",
    "Theoretical, some Tweets' texts can include usernames. Thus, to achive complete anonymisation one might need to postprocess data and remove these names.    \n",
    "\n",
    "**Construct Definition** \n",
    "\n",
    "Validity \n",
    "\n",
    "1. For the measurement of what construct was the dataset created? \n",
    "\n",
    " \n",
    "\n",
    "2. How is the construct operationalized? Can the dataset fully grasp the construct? If not, what dimensions are left out? Have there been any attempts to evaluate the validity of the construct's operationalization? \n",
    "\n",
    " \n",
    "\n",
    "3. What related constructs could (not) be measured through the dataset? What should be considered when measuring other constructs with the dataset? \n",
    "\n",
    " \n",
    "\n",
    "4. What is the target population? \n",
    "\n",
    " \n",
    "\n",
    "5. How does the dataset handle subpopulations? \n",
    "\n",
    "\n",
    "\n",
    "**Platform Selection**\n",
    "\n",
    "Platform Affordances Error \n",
    "\n",
    "1. What are the key characteristics of the platform at the time of data collection? \n",
    "\n",
    " \n",
    "\n",
    "2. What are the effects of the platform's ToS on the collected data? \n",
    "\n",
    " \n",
    "\n",
    "3. What are the effects of the platform's sociocultural norms on the collected data? \n",
    "\n",
    " \n",
    "\n",
    "4. How were the relevant traces collected from the platform? Are there any technical constraints of the data collection method? If yes, how did those limit the dataset design? \n",
    "\n",
    " \n",
    "\n",
    "5. In case multiple data sources were used, what errors might occur through their merger or combination? \n",
    "\n",
    "\n",
    "Platform Coverage Error \n",
    "\n",
    "1. What is known about the platform/s population? \n",
    "\n",
    "**Data Collection** \n",
    "\n",
    "Trace Selection Error \n",
    "\n",
    "1. How was the data associated with each instance acquired? On what basis were the trace selection criteria chosen? \n",
    "\n",
    " \n",
    "\n",
    "2. Was there any data that could not be adequately collected? \n",
    "\n",
    " \n",
    "\n",
    "3. Is any information missing from individual instances? Could there be a systematic bias? \n",
    "\n",
    " \n",
    "\n",
    "4. Does the dataset include sensitive or confidential information? \n",
    "\n",
    "User Selection Error \n",
    "\n",
    "1. Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample from a larger set, what was the sampling strategy? \n",
    "\n",
    " \n",
    "\n",
    "2. What is known about the dataset population? Are there user groups systematically in- or excluded in/from the dataset in direct consequence of the trace selection criteria? \n",
    "\n",
    " \n",
    "\n",
    "3. Over what timeframe was the data collected, and how might that timeframe have affected the collected data? \n",
    "\n",
    " \n",
    "\n",
    "4. If the dataset relates to people, how did they consent to collecting and using their data? \n",
    "\n",
    " \n",
    "\n",
    "5. Does the data include information on minors? \n",
    "\n",
    "**Data Preprocessing and Data Augmentation**\n",
    "\n",
    "Trace Augmentation and Trace Measurement Error \n",
    "\n",
    "Is there a label or target associated with each instance? If so, how were the labels or targets generated? \n",
    "\n",
    " \n",
    "\n",
    "If automated methods were used, how does the methods’ performance impact the correctness of the augmentations? \n",
    "\n",
    " \n",
    "\n",
    "If human annotations were used, who were the annotators that created the labels? How were they recruited or chosen? How were they instructed? \n",
    "\n",
    " \n",
    "\n",
    "If the final gold label was derived from different annotations, how was this done? \n",
    "\n",
    " \n",
    "\n",
    "Have there been anCan date the labels? \n",
    "\n",
    " \n",
    "How could the data be misused? \n",
    "\n",
    " \n",
    "Can the dataset in any way unintendedly contribute to the reinforcement of social inequality? \n",
    "\n",
    "User Augmentation Error \n",
    "\n",
    "Have attributes and characteristics of individuals been inferred? \n",
    "\n",
    " \n",
    "\n",
    "Is it possible to identify individuals either directly or indirectly from the data? \n",
    "\n",
    "Trace Reduction Error \n",
    "\n",
    "Have traces been excluded? Why and by what criteria? \n",
    "\n",
    "User Reduction Error \n",
    "\n",
    "Have users been excluded? Why and by what criteria? \n",
    "\n",
    "Adjustment Error \n",
    "\n",
    "Does the dataset provide information to adjust the results to a target population? If so, is this information inferred or self-reported? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fe29c81",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "### Recommended readings\n",
    "\n",
    "<a id='junger_a_2022'></a>\n",
    "Jünger, J. (2022) \"A brief history of APIs: Limitations and opportunities for online\n",
    "research.\" In: Engel, U. & Quan-Haase, A. (eds), *Handbook of Computational Social\n",
    "Science*, vol. 2, p. 17–32. Abingdon: Routledge. https://doi.org/10.4324/9781003025245.\n",
    "\n",
    "<a id='mclevey_doing_2022'></a>\n",
    "McLevey, J. (2022). *Doing Computational Social Science: A Practical Introduction*. SAGE. https://us.sagepub.com/en-us/nam/doing-computational-social-science/book266031. *A rather complete introduction to the field with well-structured and insightful chapters also on using Pandas. The [website](https://github.com/UWNETLAB/dcss_supplementary) offers the code used in the book.*\n",
    "\n",
    "### Complementary readings\n",
    "\n",
    "<a id='davidson_social_2023'></a>\n",
    "Davidson, B. I., Wischerath, D., Racek, D., Parry, D. A., Godwin, E., Hinds, J., Linden, D. v. d., Roscoe, J. F., & Ayravainen, L. (2023). \"Social media APIs: A quiet threat to the advancement of science.\" *PsyArXiv*:ps32z. https://doi.org/10.31234/osf.io/ps32z.\n",
    "\n",
    "<a id='freelon_computational_2018'></a>\n",
    "Freelon, D. (2018). \"Computational Research in the Post-API Age.\" *Political\n",
    "Communication* 35:665–668. https://doi.org/10.1080/10584609.2018.1477506.\n",
    "\n",
    "<a id='frohling_total_2023'></a>\n",
    "Fröhling, L., Sen, I., Soldner, F., Steinbrinker, L., Zens, M., & Weller, K. (2023). \"Total Error Sheets for Datasets (TES-D) -- A critical guide to documenting online platform datasets.\" *arXiv*:2306.14219. https://doi.org/10.48550/arXiv.2306.14219.\n",
    "\n",
    "<a id='lazer_computational_2009'></a>\n",
    "Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., Jebara, T., King, G., Macy, M., Roy, D., Van Alstyne, M. (2009). \"Computational Social Science.\" *Science* 323:721–723. https://doi.org/10.1126/science.1167742.\n",
    "\n",
    "<a id='lazer_computational_2020'></a>\n",
    "Lazer, D. M. J., Pentland, A., Watts, D. J., Aral, S., Athey, S., Contractor, N., Freelon, D., Gonzalez-Bailon, S., King, G., Margetts, H., Nelson, A., Salganik, M. J., Strohmaier, M., Vespignani, A., & Wagner, C. (2020). \"Computational Social Science: Obstacles and opportunities.\" *Science* 369:1060–1062. https://doi.org/10.1126/science.aaz8170.\n",
    "\n",
    "<a id='sen_a_2021'></a>\n",
    "Sen, I., Flöck, F., Weller, K., Weiß, B., & Wagner, C. (2021). \"A total error framework for digital traces of human behavior on online platforms.\" *Public Opinion Quarterly* 85:399–422. https://doi.org/10.1093/poq/nfab018.\n",
    "\n",
    "<a id='statista_most_2023'></a>\n",
    "Statista (2023). \"Most popular social networks worldwide as of January 2023, ranked by number of monthly active users.\" https://www.statista.com/statistics/272014/global-social-networks-ranked-by-number-of-users/. Retrieved 18 August 2023."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e36a4b31",
   "metadata": {},
   "source": [
    "<div class='alert alert-block alert-success'>\n",
    "<b>Document information</b>\n",
    "\n",
    "Contact and main author: N. Gizem Bacaksizlar Turbic & Pouria Mirelmi \n",
    "\n",
    "Contributors: Felix Beck-Soldner & Haiko Lietz\n",
    "\n",
    "Version date: 18 August 2023\n",
    "\n",
    "License: Creative Commons Attribution 4.0 International (CC BY 4.0)\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}