This repository explains how to analyze Japanese texts by J-LIWC2015. J-LIWC2015 is a Japanese-translated version of the LIWC2015 dictionary (Pennebaker, Booth, Jordan, & Blackburn, 2015). Psychometric properties of the J-LIWC2015 dictionary are introduced in Igarashi, Okuda, & Sasahara (2022).
Unlike English, there is no word boundary in a sentence in Japanese. It is mandatory for natural language processing to segment a Japanese text document into words before the main analysis. This repository provides basic instructions for using J-LIWC2015 and sample scripts for preprocessing texts for word segmentation.
Make sure you have installed/downloaded all of the following prerequisites in your development environment.
- MeCab/IPADIC (instructions in Japanese)
- morphological analysis (word segmentation) and part of speech analysis (POS tagging)
Japanese_Dictionary.dic
- distributed at http://www.liwc.net/dictionaries/ for non-commercial use only (you need to have a valid LIWC2015 serial number to log in to the site)
user_dict.dic
(in this repository)- user dictionary for MeCab/IPADIC
- Preprocessing: Analyze Japanese text file(s) by MeCab/IPADIC with the user dictionary (
user_dict.dic
) for word segmentation and POS tagging - Main analysis: Use the LIWC2015 software or other natural language processing libraries for category-by-category word frequency analysis
- Postprocessing (optional): Add POS tagging information to the output of the LIWC2015 software
Sample scripts are available at Google Colab.
- Python (Colab): Preprocessing and postprocessing (by @soramame0518 and @okudashi)
- Preprocessing: Analyze a Japanese text (e.g.,
sample.txt
) by MeCab/IPADIC withuser_dict.dic
in Python. In the example in Colab, two output files are generated:wakachi.txt
: word segmentation outputpos_rate.txt
: POS tagging output
- Main analysis: Run the LIWC2015 software, click
Dictionary
→Load New Dictionary
, and chooseJ-LIWC2015.dic
. Then analyzewakachi.txt
(word segmentation output). An output file is:LIWC2015 Results (wakachi).txt
: word frequency analysis output
- Postprocessing (optional): Combine
LIWC2015 Results (wakachi).txt
(word frequency analysis output) withpos_rate.txt
(POS tagging output) so that users can obtain more detailed information about POS of the text (see the example in Colab). A merged output file is:result.txt
: word frequency and POS tagging output
- Python (Colab): Preprocessing, main analysis, and postprocessing (by @soramame0518 and @okudashi)
- R: Analyzing Japanese texts with R and J-LIWC2015 (in Japanese)
Non-commercial users of the LIWC2015 software can use J-LIWC2015.dic in other programming languages (Python, R, etc.) with MeCab/IPADIC. This means that the users can seamlessly integrate the findings of preprocessing, main analysis, and postprocessing (optional) (see the example in Colab). Note that the developer does not formally support the use of the dictionary outside of the LIWC2015 software (please do it at your own risk). The compatibility of the outputs generated by and outside of the LIWC2015 software is also not guaranteed.
Any request for the distribution of the dictionary file (J-LIWC2015.dic) is not accepted. Questions about the commercial use of J-LIWC2015 should be directed to Receptiviti.
Igarashi, T., Okuda, S., & Sasahara, K. (2022). Development of the Japanese Version of the Linguistic Inquiry and Word Count Dictionary 2015. Frontiers in Psychology, 13:841534. https://doi.org/10.3389/fpsyg.2022.841534
MIT