Machine Learning/三連詞(Trigram)之應用/article_modifier_自動文件修改器(作業).ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"三連詞(Trigram)之應用.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"WneQvgOG1JqD"},"source":["題目: 將某篇文章以上下文相同，比方三連詞(trigram)方式修改內容<br>\r\n","說明: 某篇文章中我們可以找出所有的三連詞(trigram)，以及在前字與後字出現時，<br>\r\n","按照出現度隨機選出一個字去換掉中間字，這是利用三連詞修改文章內容的最基本作法。<br>\r\n","一旦字典的資料結構建立，我們就以某種機率(比方20%)去置換原文，並將置換文與原文印出來<br>\r\n","<br>\r\n","延伸: 可用五連詞或七連詞去取代中間字，可利用三連詞之前兩字去更換第三字，<br>\r\n","可增加詞性的相同性(Parts Of Speech)提高可讀性，甚至使用Word2Vec, Glove，或者RNN的??<br>\r\n","<br>\r\n","範例程式檔名: article_modifier_自動文件修改器.py<br>\r\n","模組: sklearn, random, numpy, nltk, bs4<br>\r\n","輸入檔：positive(作業數據).review<br>\r\n","成績: 被置換文的合理性與可讀性<br>"]},{"cell_type":"markdown","metadata":{"id":"dMuSrydr1-kg"},"source":["使用三連詞 trigrams 練習簡易文件產生器"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"c_0YtKU_2Acd","executionInfo":{"status":"ok","timestamp":1608899849510,"user_tz":-480,"elapsed":1524,"user":{"displayName":"王俊煒","photoUrl":"","userId":"10865038187423164500"}},"outputId":"c64ab1a0-ae15-4612-df64-88c0271934eb"},"source":["import nltk\r\n","nltk.download('punkt')\r\n","\r\n","import random\r\n","import numpy as np\r\n","from bs4 import BeautifulSoup"],"execution_count":1,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading package punkt to /root/nltk_data...\n","[nltk_data]   Package punkt is already up-to-date!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"sQ---xoi2GR3"},"source":["load the reviews"]},{"cell_type":"code","metadata":{"id":"5xSQDi7W2IKf","executionInfo":{"status":"ok","timestamp":1608899849510,"user_tz":-480,"elapsed":1518,"user":{"displayName":"王俊煒","photoUrl":"","userId":"10865038187423164500"}}},"source":["positive_reviews = BeautifulSoup(open('positive(作業數據).review', encoding='ISO-8859-1').read(), \"lxml\")\r\n","positive_reviews = positive_reviews.findAll('review_text')"],"execution_count":2,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yOuvyn-O2MlN"},"source":["提出**三連詞**並置入字典<br>\r\n","(w1, w2)當作key,[w3]當作值"]},{"cell_type":"code","metadata":{"id":"ZiUR65nq2OPn","executionInfo":{"status":"ok","timestamp":1608899851089,"user_tz":-480,"elapsed":3092,"user":{"displayName":"王俊煒","photoUrl":"","userId":"10865038187423164500"}}},"source":["trigrams = {}\r\n","for review in positive_reviews:\r\n","    s = review.text.lower()\r\n","    tokens = nltk.tokenize.word_tokenize(s)\r\n","    for i in range(len(tokens) - 2):\r\n","        k = (tokens[i], tokens[i + 1])\r\n","        if k not in trigrams:\r\n","            trigrams[k] = []\r\n","        trigrams[k].append(tokens[i + 2])"],"execution_count":3,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ygX7aCG22okp"},"source":["將最後字矩陣變成似然機率(likelihood)向量"]},{"cell_type":"code","metadata":{"id":"79JA8bLC2vPa","executionInfo":{"status":"ok","timestamp":1608899851090,"user_tz":-480,"elapsed":3089,"user":{"displayName":"王俊煒","photoUrl":"","userId":"10865038187423164500"}}},"source":["trigram_probabilities = {}\r\n","for k, words in trigrams.items():\r\n","    # 產生一個 word -> count 字典\r\n","    if len(set(words)) > 1:\r\n","        # 如果最後字(last word)不只有一個機率 \r\n","        d = {}\r\n","        n = 0\r\n","        for w in words:\r\n","            if w not in d:\r\n","                d[w] = 0\r\n","            d[w] += 1\r\n","            n += 1\r\n","        for w, c in d.items():\r\n","            d[w] = float(c) / n\r\n","        trigram_probabilities[k] = d"],"execution_count":4,"outputs":[]},{"cell_type":"code","metadata":{"id":"hVvWrnd14JCg","executionInfo":{"status":"ok","timestamp":1608899851090,"user_tz":-480,"elapsed":3085,"user":{"displayName":"王俊煒","photoUrl":"","userId":"10865038187423164500"}}},"source":["def random_sample(d):\r\n","    # 從字典隨機選出一個帶機率值的樣本，回傳累積機率值最大的字\r\n","    r = random.random()\r\n","    cumulative = 0\r\n","    for w, p in d.items():\r\n","        cumulative += p\r\n","        if r < cumulative:\r\n","            return w"],"execution_count":5,"outputs":[]},{"cell_type":"code","metadata":{"id":"aeBb6H934SYq","executionInfo":{"status":"ok","timestamp":1608899851091,"user_tz":-480,"elapsed":3081,"user":{"displayName":"王俊煒","photoUrl":"","userId":"10865038187423164500"}}},"source":["def test_spinner():\r\n","    review = random.choice(positive_reviews)\r\n","    s = review.text.lower()\r\n","    print('Original:', s)\r\n","    tokens = nltk.tokenize.word_tokenize(s)\r\n","    for i in range(len(tokens) - 2):\r\n","        if random.random() < 0.2:   # 20% chance of replacement\r\n","            k = (tokens[i], tokens[i + 1])\r\n","            if k in trigram_probabilities:\r\n","                w = random_sample(trigram_probabilities[k])\r\n","                tokens[i + 1] = w\r\n","    print('Spun:')\r\n","    print(' '.join(tokens).replace(' .', '.').replace(' \\'', '\\'').replace(' ,', ',').replace('$ ', '$').replace(' !', '!'))"],"execution_count":6,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Ovs8-cbb4wD0","executionInfo":{"status":"ok","timestamp":1608899851091,"user_tz":-480,"elapsed":3077,"user":{"displayName":"王俊煒","photoUrl":"","userId":"10865038187423164500"}},"outputId":"94894fcb-6762-4d2f-f5f1-e072fab85c0b"},"source":["test_spinner()"],"execution_count":7,"outputs":[{"output_type":"stream","text":["Original: \n","i had no problem with the product.  the directions were pretty simple.  \n","\n","problem:  i never received my rebate information!!!\n","\n","\n","Spun:\n","i had no problem with the description. the directions were pretty simple. problem : i never received my rebate information!!!\n"],"name":"stdout"}]}]}