Preview (Product review) project

Read this in other languages: 한국어

Product review crawler and preprocessing with OCR and etc.
Combine with honeycomb (Flask server + SQLAlchemy) and MySQL DB.

Composition

Driver

Can use proxy IP if you have.
Can changes the user-agent if you want.
Failover process when crawler stops like IP banned Case.

Route handler

Error handling with retry pattern when fetch to the Webserver
Define fetch url to Web server

Image processing

Use PaddleOCR to analyize product explanation by seller.
Due to large size of the image, there is a image cutting logic without ignoring any characters.
TODO: multiprocessing would be added in this process.

GUI

Use GUI with tkinter to develop tagging tools.
TWO TYPES
1. OCR tagging
  - OCR tagging tools.
2. Review tagging
  - Review tagging tools.

Crawler work at

product_link_crawler.py
- Crawling at naver shopping price comparsion tabs.
- Crawl data and fetch to the server.
review_crawler.py
- Crawling at the product-unit price comparison page.
- Get OCR data with PaddleOCR/
- Get review data by json payloads.
- Get spec data provided by naver.
- Fetch to the server

Dockerfile (with only testing product_link_crawler.py)

Sequence Diagram

product_link_crawler.py

review_crawler.py