Skip to content

Latest commit

 

History

History
14 lines (12 loc) · 651 Bytes

README.md

File metadata and controls

14 lines (12 loc) · 651 Bytes

Royal road mining

Webscraper for RoyalRoad.com, custom-built with BeautifulSoup4. Current specific functions:

  • Search novel
  • Scrape novel data
  • Scrape names and url's of multiple novels in a category

In-development:

  • Storage and retrieval with MongoDB
  • Extension of scraping pipeline to collect incomes on patreon if given
  • Data Analysis pipeline to conduct exploratory data analysis
  • NLP pipeline to summarise large amount of reviews and extract main critiques and praises

Possible improvements:

  • Speed up storage and retrieval times. Make memory usage more efficient (explore async, caching requests, and PostgreSQL and MongoDB)