data_standardization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**数据标准化（归一化）处理**是数据挖掘的一项基础工作，不同评价指标往往具有不同的量纲和量纲单位，这样的情况会影响到数据分析的结果。为了消除的指标之间的量纲影响，需要数据标准化处理，以解决数据指标间的可比性。\n",
    "\n",
    "**数据标准化**是将数据按比例缩放，使之落入一个小的特定区间。在某些比较和评价的指标处理中经常会用到，去除数据的单位限制，将其转化为无量纲的纯数值，便于不同单位或量级的指标能够进行比较和加权。\n",
    "\n",
    "目前**数据标准化方法**有多种，归结起来可以分为直线型方法(如极值法、标准差法)、折线型方法(如三折线法)、曲线型方法(如半正态性分布)。不同的标准化方法，对系统的评价结果会产生不同的影响，然而不幸的是，在数据标准化方法的选择上，还**没有通用的法则可以遵循**。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<span style=\"color:red\">1. min-max normalizatin/0-1 normalization</span>**\n",
    "> 是对原始数据的线性变换，使结果值映射到[0, 1]之间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$x^* = \\frac{x-min}{max-min}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "def normaliatin(xArr):  # xArr:numpy array\n",
    "    '''normalization by columns'''\n",
    "    return (xArr-np.min(xArr, 0)) / (np.max(xArr, 0)-np.min(xArr, 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 将数据映射到[-1, 1]\n",
    "def normalization(xArr):\n",
    "    return ((xArr-np.mean(xArr, 0)) / (np.max(xArr, 0)- np.min(xArr, 0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**缺点**：当有新数据加入时，可能导致max与min的变化，需要重新定义。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<span style=\"color:red\">2. zero-mean normalization</span>**\n",
    "> 基于原始数据的均值(mean)和标准差(standard deviation)进行数据的标准化，进过处理的数据符合标准正态分布。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$x^* = \\frac{x-mean}{std}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def z_scoreNorm(xArr):  # xArr:numpy array\n",
    "    '''normalization by columns'''\n",
    "    xArrMeans = np.mean(xArr, 0)\n",
    "    xArrStd = np.std(xArrr, 0)\n",
    "    return (xArr-xArrMeans) / xArrStd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<span style=\"color:red\">3. lg函数转换</span>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$x^* = \\frac{lg(x)}{lg(max)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def lgNorm(array_like):\n",
    "    return np.log10(array_like) / np.log10(np.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<span style=\"color:red\">4. arctan函数转换</span>**\n",
    "> 要想映射到区间[0,1]，则所有的数据应该>=0, 小于0的数据将被映射到[-1,0]上"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$x^* = \\frac{arctan(x)*2}{np.pi}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**z-score standardization的表现更好：**\n",
    "> 在分类聚类中，需要使用距离来度量相似性的时候\n",
    "\n",
    "> 或者使用PCA技术来进行降维的时候"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在不涉及距离度量、协方差计算、数据不符合正态分布的时候，可以使用第一种方法或其他归一化方法，比如图像中将RGB图像转换为灰度图像后将其值限定在[0, 255]的范围。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}