Japanese Learning Environment

Browse lists of Japanese texts, including simplified articles and news. Upload your own content to generate PDFs featuring Japanese radicals, kanji, vocabulary, and sentences for learning. Create or explore lists with articles, radicals, kanji, and vocabulary items, track your progress, and chat with other learners.

Built using Laravel for Server API and React for the client side generated with create-react-app.

Japanese data comes from Electronic Dictionary Research and Development Group, and are used in conformance with the Group's licence. This site uses the JMdict, Kanjidic2, JMnedict, and Radkfile dictionary files. JLPT data comes from Jonathan Waller's JLPT Resources page.

Table of Contents

• Features
  • Backend
  • Frontend
• Setup
  • Docker Setup
  • Test API
  • Local Setup
    • Laravel API Setup
    • MySQL Database Setup
    • React App Setup
  • To Do List
• Ongoing Development

Features

• Laravel CRUD REST API endpoints for Articles, Lists, Roles, Users and Posts(forum).
• Like, hashtag, and comment functionalities for articles, lists, and posts.
• Full CRUD and search functionality for lists of kanji, radicals, words, and sentences (sentences sourced and linked from the Tatoeba community).
• Japanese language data extracted from XML to CSV using plain PHP and MySQL script, formatted and imported to Laravel via Laravel migrations CLI. Kanjis and words were matched against JLPT levels assigned from Jonathan Waller's JLPT Resources.
• Authentication with Laravel/passport and Laravel's Eloquent ORM.
• Text scanning algorithm to find kanjis and words used in user provided article.
• PDFs generation with english meanings for Kanjis and Words based on saved Lists or Article content.
• State management via Redux and custom styling with Bootstrap and CSS.

Backend

• laravel-snappy - for PDF generating. PDFs generated using laravel's blade templates structure. wkhtmltopdf wkhtmltopdf is required in order for the laravel-snappy library to work.
• passport
• laravel helpers - more comfortable customized version of the original laravel helpers
• laravel/ui - scaffolding for the frontend of laravel API landing page. In this project React was used.

Frontend

• react-moment of moment for date formatting.
• axios for HTTP requesting.
• react-bootstrap for components bootstrap for overall styling.
• react-router-bootstrap for react-router and react-bootstrap integration.
• redux, redux-thunk and react-redux for data access.

Setup

Docker Setup

Laravel API served with PHP-FPM for php scripts execution, NGINX handles browser requests, static data as CSS, JS, images and routes PHP requests to PHP-FPM.

Now, in /processor-api repository root run:

chmod +x .docker/entrypoint.sh

Run docker containers in detached mode:

docker-compose up -d --build

Test if Mysql initialized properly, by entering container via bash or docker desktop:

docker-compose exec db bash
mysql -u <DB_USERNAME> -p
prompted password: <DB_PASSWORD>

List tables, there should be all the common ones:

SHOW databases;
USE <DB_NAME>;
SHOW TABLES;

If tables are there, proceed with Japanese data migration. Data volume is big, so it may take over a minute and be cautions when rebuilding containers to avoid duplication.

Enter container via bash or docker desktop into laravel app:

# Enter container
docker-compose exec laravel-app bash
# Create japanese material tables
php artisan migrate --path=database/migrations/japanese-data

# Create new tables, generate encryption keys for Passport
php artisan passport:install

Necessary to fill-up the categories for the entities objects in "objecttemplates" table. Creates common and admin users Example custom lists and articles

php artisan db:seed
# seed single table
php aritsan db:seed --class=<ClassNameSeeder>

If by chance something seems cached or configs not updated try reset:

composer dump-autoload
php artisan config:clear
php artisan cache:clear

Take containers down and empty volumes for clean containers rebuild

docker-compose down -v
docker-compose up -d --build

Remember after clean rebuild to run migrations and seed once again.

Test API

Basic requests for testing purposes. Enter laravel api container and register/login:

curl -X POST http://nginx_webserver/api/login \
  -H "Content-Type: application/json" \
  -d '{"email": "test@test.me", "password": "test123"}'

# or save as a variable:
TOKEN=$(curl -s -X POST http://nginx_webserver/api/login \
  -H "Content-Type: application/json" \
  -d '{"email": "test@test.me", "password": "test123"}' \
  | jq -r '.accessToken')

Use variable:

curl -X GET http://nginx_webserver/api/articles \
  -H "Authorization: Bearer $TOKEN" \
  | jq

# or single article:
curl -X GET http://nginx_webserver/api/article/5 \
  -H "Authorization: Bearer $TOKEN" \
  | jq
# or single list
curl -X GET http://nginx_webserver/api/list/3 \
  -H "Authorization: Bearer $TOKEN" \
  | jq
# or regular response
curl -X GET http://nginx_webserver/api/list/3/words-pdf \
  -H "Authorization: Bearer $TOKEN"
# or generate pdf and move localy tp test result
curl -X GET http://nginx_webserver/api/list/3/words-pdf \
  -H "Authorization: Bearer $TOKEN" \
  -o words-test.pdf
docker cp laravel_app:/var/www/html/words-test.pdf C:\Users\USER-NAME\Downloads\

Test PDF file on a local machine:

curl -X GET http://nginx_webserver/api/article/5/kanjis-pdf \
  -H "Authorization: Bearer $TOKEN" \
  -o kanjis-test.pdf
docker cp laravel_app:/var/www/html/kanjis-test.pdf C:\Users\USER-NAME\Downloads\

Local Setup

Required

• Fonts supporting Japanese language, for example: fonts-ipafont-gothic.
• Composer - PHP package manager
• Node. - Node js runtime for frontend assets of laravel.
• For PDF generating laravel-snappy to work requires wkhtmltopdf. Install it as a composer dependency or manually here into your machine.

Also, possibly paths for wkhtmltopdf will not work, uncomment variables in .env to use needed ones or modify them according needs.

Optional choices for faster setup:

• Xampp - web server Apache, PHP and MariaDB(MySQL).
• Laravel Herd - dev environment with all you need for laravel development.

Laravel API

In processor-api directory Install composer packages:

composer install

Change .env file settings for your database.

cp .env.example .env

Generate unique app key

php artisan key:generate

Database MySQL

Requirements:

• Make sure your DB Collation is utf8mb4_general_ci for Japanese characters support.
• Must assure that MySQL config in my.ini has packets size adjusted max_allowed_packet=128M in order for migrations to enable importing bigger sized japanese files into the database.

If mysql config is correct, proceed steps below

# Create common tables
php artisan migrate --path=database/migrations/now

# Create japanese material tables
php artisan migrate --path=database/migrations/japanese-data

Add Passport clients

# Create new tables, generate encryption keys for Passport
php artisan passport:install

Seed initial data

# Necessary to fill-up the categories for the resources objects in "objecttemplates" table.
# Creates common and admin users
# Example custom lists
# Example articles
php artisan db:seed

API Documentation page is served on laravel side, to have it work install packages.

npm install
# to watch changes
npm run watch

React App

In client directory Install node modules:

npm install

Start react app

npm start

To Do List

• Make demo gifs to showcase main features.
• Add swagger for Laravel API, add models for swagger to use swagger UI for quick API usage
• Refactor single component to functional component using latest react hooks in a composition way to have example component.
• Strengthen authorization with more persistant implementation..
• Implement react-query for query-based approach of managing server-data facing for frontend and fixing issues like cache, refetch, cancel requests after unmount.
• Create nhk easy news scrapper to get each days news.
• re-vamp styling, especially for small UI elements like links, buttons with icons, screen spacings.
• Design a way to offload scanning algorithm from client to server.
• Create service worker to build queues for scanning algorithm of user texts to find kanjis and words.
• Create Github Actions for frontend.
• Create Github Actions for backend.
• Write E2E tests for frontend.
• [?] Client side PDF customization. Generate on Backend, customize on frontend.
• [?] Cache or Store pre-generated materials like PDFs.

Ongoing

• couple /search endpoints doesnt work