numeric-normalization.qmd

---
pagetitle: "Feature Engineering A-Z | Normalization"
---

# Normalization {#sec-numeric-normalization}

::: {style="visibility: hidden; height: 0px;"}
## Normalization
:::

Normalization is a method where we modify a variable by subtracting the mean and dividing by the standard deviation

$$X_{scaled} = \dfrac{X - \text{mean}(X)}{\text{sd}(X)}$$

Performing this transformation means that the resulting variable will have a mean of 0 and a standard deviation and variance of 1.
This method is a learned transformation.
So we use the training data to derive the right values of $\text{sd}(X)$ and $\text{mean}(X)$ and then these values are used to perform the transformations when applied to new data.
It is a common misconception that this transformation is done to make the data normally distributed.
This transformation doesn't change the distribution, it scales the values.
Below is a figure @fig-normalization-not-normal that illustrates that point

```{r}
#| label: fig-normalization-not-normal
#| echo: false
#| message: false
#| fig-cap: |
#|   Normalization doesn't make data more normal. The green curve indicates the density of the unit normal
#|   distribution. 
#| fig-alt: |
#|   2 histograms of distribution one above the other. The top distribution shows a bimodal
#|   distribution. Below is the same distribution after being normalized. Both appear clearly
#|   non-normally distributed. The green curve is overlaid lower histogram. It doesn't follow.
library(ggplot2)
library(dplyr)
library(tidyr)
set.seed(1234)

plotting_data <- tibble(Original = (rbeta(1000, 0.3, 0.5) + rnorm(1000, sd = 0.05)) * 10) |>
  mutate(Transformed = (Original - mean(Original)) / sd(Original)) |>
  pivot_longer(everything())

norm_curve <- tibble(
  x = seq(min(plotting_data$value) - 1, max(plotting_data$value), by = 0.2),
  y = dnorm(seq(min(plotting_data$value) - 1, max(plotting_data$value), by = 0.2)),
  name = "Transformed"
) |>
  mutate(y = 1000 / sum(y) * y)

plotting_data |>
  ggplot(aes(value)) +
  geom_histogram(binwidth = 0.2) +
  facet_grid(name~., scales = "free_y") +
  theme_minimal() +
  labs(x = NULL, y = NULL) +
  geom_line(aes(x, y), data = norm_curve, color = "green4")
```

In @fig-normalization-not-normal we see some distribution, before and after applying normalization to it.
Both the original and transformed distribution are bimodal, and the transformed distribution is no more normal than the original.
That is fine because the transformation did its job by moving the values close to 0 and a specific spread, which in this case is a variance of 1.

## Pros and Cons

### Pros

-   If you don't have any severe outliers then you will rarely see any downsides to applying normalization
-   Fast calculations
-   Transformation can easily be reversed, making its interpretations easier on the original scale

### Cons

-   Not all software solutions will be helpful when applying this transformation to a constant variable. You will often get a "division by `0`" error
-   Cannot be used with sparse data as it isn't preserved because of the centering that is happening. If you only scale the data you don't have a problem
-   This transformation is highly affected by outliers, as they affect the mean and standard deviation quite a lot

Below is the figure @fig-normalization-outlier is an illustration of the effect of having a single high value.
In this case, a single observation with the value `10000` moved the transformed distribution much tighter around zero.
And all but removed the variance of the non-outliers.

```{r}
#| label: fig-normalization-outlier
#| echo: false
#| message: false
#| fig-cap: |
#|   Outliers can have a big effect on the resulting distribution when applying normalization.
#| fig-alt: |
#|   4 histograms of distribution in 2 columns. The top distribution shows the same bimodal
#|   distribution. Below are the same distributions after being normalized. The right column
#|   shows the effect of having one outlier at 10000, which in this case made the transformed
#|   distribution about a fifth of the width.
library(ggplot2)
library(dplyr)
library(tidyr)
set.seed(1234)

rand_val <- (rbeta(1000, 0.3, 0.5) + rnorm(1000, sd = 0.05)) * 10

plotting_data <- 
  bind_rows(
    tibble(Original = rand_val) |>
      mutate(Transformed = (Original - mean(Original)) / sd(Original)) |>
      pivot_longer(everything()) |>
      mutate(outlier = "No outlier"),
    tibble(Original = c(rand_val, 500)) |>
      mutate(Transformed = (Original - mean(Original)) / sd(Original)) |>
      pivot_longer(everything()) |>
      mutate(outlier = "One outlier at 10000")
  ) |>
  filter(value < 15)
  
plotting_data |>
  ggplot(aes(value)) +
  geom_histogram(binwidth = 0.2) +
  facet_grid(name ~ outlier, scales = "free") +
  theme_minimal() +
  labs(x = NULL, y = NULL)
```

## R Examples

We will be using the `ames` data set for these examples.

```{r}
#| label: show-data
#| message: false
library(recipes)
library(modeldata)
data("ames")

ames |>
  select(Sale_Price, Lot_Area, Wood_Deck_SF, Mas_Vnr_Area)
```

{recipes} provides a step to perform scaling, centering, and normalization.
They are called `step_scale()`, `step_center()` and `step_normalize()` respectively.

Below is an example using `step_scale()`

```{r}
#| label: step_scale
scale_rec <- recipe(Sale_Price ~ ., data = ames) |>
  step_scale(all_numeric_predictors()) |>
  prep()

scale_rec |>
  bake(new_data = NULL, Sale_Price, Lot_Area, Wood_Deck_SF, Mas_Vnr_Area)
```

We can also pull out the value of the standard deviation for each variable that was affected using `tidy()`

```{r}
#| label: step_scale-tidy
scale_rec |>
  tidy(1)
```

We could also have used `step_center()` and `step_scale()` together in one recipe

```{r}
#| label: step_center-step_scale
center_scale_rec <- recipe(Sale_Price ~ ., data = ames) |>
  step_center(all_numeric_predictors()) |>
  step_scale(all_numeric_predictors()) |>
  prep()

center_scale_rec |>
  bake(new_data = NULL, Sale_Price, Lot_Area, Wood_Deck_SF, Mas_Vnr_Area)
```

Using `tidy()` we can see information about each step

```{r}
#| label: step_center-step_scale-tidy
center_scale_rec |>
  tidy()
```

And we can pull out the means using `tidy(1)`

```{r}
#| label: step_center-step_scale-tidy-one
center_scale_rec |>
  tidy(1)
```

and the standard deviation using `tidy(2)`

```{r}
#| label: step_center-step-scale-tidy-two
center_scale_rec |>
  tidy(2)
```

Since these steps often follow each other, we often use the `step_normalize()` as a shortcut to do both operations in one step

```{r}
#| label: step_normalize
scale_rec <- recipe(Sale_Price ~ ., data = ames) |>
  step_normalize(all_numeric_predictors()) |>
  prep()

scale_rec |>
  bake(new_data = NULL, Sale_Price, Lot_Area, Wood_Deck_SF, Mas_Vnr_Area)
```

And we can still pull out the means and standard deviations using `tidy()`

```{r}
#| label: step_normatize-tidy
scale_rec |>
  tidy(1) |>
  filter(terms %in% c("Lot_Area", "Wood_Deck_SF", "Mas_Vnr_Area"))
```

## Python Examples

```{python}
#| label: python-setup
#| echo: false
import pandas as pd
from sklearn import set_config

set_config(transform_output="pandas")
pd.set_option('display.precision', 3)
```

We are using the `ames` data set for examples.
{sklearn} provided the `StandardScaler()` method we can use.
By default we can use this method to perform both the centering and scaling.

```{python}
#| label: standardscaler
from feazdata import ames
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler

ct = ColumnTransformer(
    [('normalize', StandardScaler(), ['Sale_Price', 'Lot_Area', 'Wood_Deck_SF',  'Mas_Vnr_Area'])], 
    remainder="passthrough")

ct.fit(ames)
ct.transform(ames)
```

To only perform scaling, you set `with_mean=False`.

```{python}
#| label: standardscaler-with_means-false
ct = ColumnTransformer(
    [('scaling', StandardScaler(with_mean=False, with_std=True), ['Sale_Price', 'Lot_Area', 'Wood_Deck_SF',  'Mas_Vnr_Area'])], 
    remainder="passthrough")

ct.fit(ames)
ct.transform(ames)
```

And to only perform centering you set `with_mean=True`.

```{python}
#| label: standardscaler-with_mean-true
ct = ColumnTransformer(
    [('centering', StandardScaler(with_mean=True, with_std=False), ['Sale_Price', 'Lot_Area', 'Wood_Deck_SF',  'Mas_Vnr_Area'])], 
    remainder="passthrough")

ct.fit(ames)
ct.transform(ames)
```