06-document-term-matrices.Rmd

# Tidying and casting document-term matrices {#dtm}

```{r echo = FALSE}
library(knitr)
opts_chunk$set(message = FALSE, warning = FALSE, cache = TRUE)
options(width = 100, dplyr.width = 150)
library(ggplot2)
theme_set(theme_light())
```

In the previous chapters, we've been analyzing text arranged in the tidy text format: a table with one-token-per-document-per-row, as is constructed by the `unnest_tokens` function. This lets us use the popular suite of tidy tools such as dplyr, tidyr, and ggplot2. We've demonstrated that many text analyses can be performed using these principles.

However, most of the existing R tools for natural language processing, besides the tidytext package, aren't compatible with this format. The [CRAN Task View for Natural Language Processing](https://cran.r-project.org/web/views/NaturalLanguageProcessing.html) lists a large selection of packages that take other inputs. One of the most common formats is the [document-term matrix](https://en.wikipedia.org/wiki/Document-term_matrix), a sparse matrix with one row for each document in a collection and one column for each term or word. The value that goes into the matrix is typically the count of that term in that document, or sometimes the tf-idf (see Chapter 4). These matrices are sparse (they consist mostly of zeroes), so specialized algorithms and data structures can be used to deal with them that are efficient and fast.

The tidytext package can integrate these important packages into an analysis while still relying on the suite of tidy tools for processing and visualization. The two key verbs are:

* `tidy()`: Turns an object, such as a document-term matrix, into a tidy data frame.
* `cast_`: Turns a tidy one-term-per-row data frame into a document-term matrix. tidytext provides the functions `cast_sparse()` (a sparse matrix from the Matrix package), `cast_dtm()` (`DocumentTermMatrix` objects from tm), and `cast_dfm()` (`dfm` objects from quanteda).

In this chapter, we'll examine some examples of tidying document-term matrices, as well as converting from a tidy format into a sparse matrix.

## Tidying a document-term matrix

Many existing text mining packages provide and expect **document-term matrix**, or DTM. A DTM is a matrix where

* each row represents one document,
* each column represents one term, and
* each value (usually) contains the number of appearances of that term in that document.

DTMs are usually implemented as sparse matrices, meaning the vast majority of values are 0. These objects can be interacted with as though they were matrices, but are stored in a more efficient format.

### Tidying DocumentTermMatrix objects

One commonly used implementation of DTMs in R is the `DocumentTermMatrix` class in the tm package. Many existing text mining datasets are provided in this format. For example, consider the corpus of Associated Press newspaper articles included in the topicmodels package.

```{r AssociatedPress}
library(tm)

data("AssociatedPress", package = "topicmodels")
class(AssociatedPress)
AssociatedPress
```

We see that this dataset contains `r nrow(AssociatedPress)` documents (each of them an AP article) and `r ncol(AssociatedPress)` terms (distinct words). Notice that this DTM is 99% sparse (99% of document-word pairs are zero).

If we wanted to analyze this data with tidy tools, we would first need to turn it into a one-token-per-document-per-row data frame. The broom package [@R-broom] introduced the `tidy` verb, which takes a non-tidy object and turns it into a tidy data frame. The tidytext package implements that method for `DocumentTermMatrix` objects:

```{r ap_td, dependson = "AssociatedPress"}
library(dplyr)
library(tidytext)

ap_td <- tidy(AssociatedPress)
ap_td
```

Notice that we now have a tidy three-column `tbl_df`, with variables `document`, `term`, and `count`. This tidying operation is similar to the `melt` function from the reshape2 package [@R-reshape2] for non-sparse matrices.

As we've seen in previous chapters, this form is convenient for analysis with the dplyr, tidytext and ggplot2 packages. For example, you can perform sentiment analysis on these newspaper articles.

```{r ap_sentiments, dependson = "ap_td"}
ap_sentiments <- ap_td %>%
  inner_join(get_sentiments("bing"), by = c(term = "word"))

ap_sentiments
```

This could let us visualize which words from these AP articles most often contributed to positive or negative sentiment:

```{r dependson = "ap_sentiments", fig.height = 6, fig.width = 7}
library(ggplot2)

ap_sentiments %>%
  count(sentiment, term, wt = count) %>%
  ungroup() %>%
  filter(n >= 200) %>%
  mutate(n = ifelse(sentiment == "negative", -n, n)) %>%
  mutate(term = reorder(term, n)) %>%
  ggplot(aes(term, n, fill = sentiment)) +
  geom_bar(alpha = 0.8, stat = "identity") +
  ylab("Contribution to sentiment") +
  coord_flip()
```

### Tidying dfm objects

Other text mining packages provide alternative implementations of document-term matrices, such as the `dfm` (document-feature matrix) class from the quanteda package [@R-quanteda]. Consider the corpus of presidential inauguration speeches that comes with the quanteda package:

```{r inaug_dfm, message = FALSE}
library(methods)

data("inaugCorpus", package = "quanteda")
inaug_dfm <- quanteda::dfm(inaugCorpus)

inaug_dfm
```

The `tidy` method works on these objects as well, turning them into a one-token-per-document-per-row table:

```{r inaug_td, dependson = "inaug_dfm"}
inaug_td <- tidy(inaug_dfm)
inaug_td
```

We may be interested in finding the words most specific to each inaugural speeches, which can be done by calculating the TF-IDF of each term-speech pair using the `bind_tf_idf` function from chapter 4.

```{r presidents, dependson = "inaug_td", fig.width = 8, fig.height = 8}
inaug_tf_idf <- inaug_td %>%
  bind_tf_idf(term, document, count) %>%
  arrange(desc(tf_idf))

inaug_tf_idf
```

We could then pick four notable inaugural addresses (from Washington, Lincoln, Kennedy, and Obama), and visualize the words most specific to each speech.

```{r dependson = "presidents"}
speeches <- c("1793-Washington", "1861-Lincoln",
              "1961-Kennedy", "2009-Obama")

inaug_tf_idf %>%
  filter(document %in% speeches) %>%
  group_by(document) %>%
  top_n(10, tf_idf) %>%
  ungroup() %>%
  mutate(term = reorder(term, tf_idf)) %>%
  ggplot(aes(term, tf_idf, fill = document)) +
  geom_bar(stat = "identity", alpha = 0.8, show.legend = FALSE) +
  facet_wrap(~ document, scales = "free") +
  coord_flip() +
  labs(x = "",
       y = "TF-IDF")
```

Having a corpus in this format is also useful for visualizations. We could extract the year from each document's name, and compute the total number of words within each year.

```{r year_term_counts, dependson = "inaug_td"}
library(tidyr)

year_term_counts <- inaug_td %>%
  extract(document, "year", "(\\d+)", convert = TRUE) %>%
  complete(year, term, fill = list(count = 0)) %>%
  group_by(year) %>%
  mutate(year_total = sum(count))

year_term_counts
```

This lets us pick several words and visualize how they changed in frequency over time.

```{r year_term_counts_plot, dependson = "year_term_counts"}
year_term_counts %>%
  filter(term %in% c("god", "america", "foreign", "union")) %>%
  ggplot(aes(year, count / year_total)) +
  geom_point() +
  geom_smooth() +
  facet_wrap(~ term) +
  scale_y_continuous(labels = scales::percent_format()) +
  ylab("% frequency of word in inaugural address")
```

TODO: conclude section

## Casting tidy text data into a matrix

Just as some existing text mining packages provide document-term matrices as sample data or output, some algorithms expect these matrices as input. Therefore, tidytext provides `cast_` verbs for converting from a tidy form to these matrices.

For example, we could take the tidied AP dataset and cast it back into a document-term matrix:

```{r}
ap_td %>%
  cast_dtm(document, term, count)
```

Similarly, we could cast it into a Term-Document Matrix with `cast_tdm`, or quanteda's dfm with `cast_dfm`:

```{r }
# cast into a Term-Document Matrix
ap_td %>%
  cast_tdm(term, document, count)

# cast into quanteda's dfm
ap_td %>%
  cast_dfm(term, document, count)
```

Some tools simply require a sparse matrix:

```{r}
library(Matrix)

# cast into a Matrix object
m <- ap_td %>%
  cast_sparse(document, term, count)

class(m)
dim(m)
```

This casting process allows for easy reading, filtering, and processing to be done using dplyr and other tidy tools, after which the data can be converted into a document-term matrix for machine learning applications.

## Tidying corpus objects with metadata

You can also tidy Corpus objects from the tm package. For example, consider a Corpus containing 20 documents:

```{r reuters}
reut21578 <- system.file("texts", "crude", package = "tm")
reuters <- VCorpus(DirSource(reut21578),
                   readerControl = list(reader = readReut21578XMLasPlain))

reuters
```

The `tidy` verb creates a table with one row per document:

```{r reuters_td, dependson = "reuters"}
reuters_td <- tidy(reuters)
reuters_td
```

Another variation of a corpus object is `corpus` from the quanteda package:

```{r inaug_td2}
library(quanteda)

data("inaugCorpus")

inaugCorpus

inaug_td <- tidy(inaugCorpus)
inaug_td
```

This lets us work with tidy tools like `unnest_tokens` to analyze the text alongside the metadata.

```{r inaug_words, dependson = "inaug_td2"}
inaug_words <- inaug_td %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words)

inaug_words
```

We could then, for example, see how the appearance of a word changes over time:

```{r inaug_freq, dependson = "inaug_words"}
library(tidyr)

inaug_freq <- inaug_words %>%
  count(Year, word) %>%
  ungroup() %>%
  complete(Year, word, fill = list(n = 0)) %>%
  group_by(Year) %>%
  mutate(year_total = sum(n),
         percent = n / year_total) %>%
  ungroup()

inaug_freq %>%
  filter(word == "america")
```

For instance, we could display the top 6 terms that have changed in frequency over time.

```{r dependson = "models", fig.width=8, fig.height=6}
library(scales)

inaug_freq %>%
  filter(word %in% c("americans", "century", "foreign", "god",
                     "union", "constitution")) %>%
  ggplot(aes(Year, percent)) +
  geom_point(alpha = 0.8) +
  geom_smooth() +
  facet_wrap(~ word, scales = "free_y") +
  scale_y_continuous(labels = percent_format()) +
  ylab("Frequency of word in speech")
```