simple_links_scraper

Is a simple Ruby on Rail web application that attempts to extract all links in any given web application (except web apps that require javascript). The application only extracts links of the given url, this means it will not follow identified links that it finds.

Getting Started

These instructions will get you a copy of the project up and running on your local machine

Prerequisites

What things you need to install the software and how to install them

• Ruby-v2.5.1p57
• Rails-v5.2.1
• Postgresql

Installing

A step by step series of examples that tell you how to get a development env running

1. Clone this repo
2. cd to my-scraper directory
3. Install dependencies bundle install
4. Setup figaro figaro install
5. Setup some environment variables in config/application.yml file

   DOMAIN: [your-domain]
   SMTP_USER_NAME: [your gmail address]
   SMTP_PASSWORD: [your gmail password]

   NB: replace everything that is inside []
6. Prepare database rake db:setup
7. Migrate database rake db:migrate
8. Start sidekiq bundle exec sidekiq -q default -q mailers
9. Start development server rails s 😎

Docker Folks

1. Install docker-compose
2. Run docker-compose build
3. Create Database docker-compose run app bundle rake db:create
4. Run migrations docker-compose run app bundle exec rake db:migrate
5. Start your app docker-compose up
6. Visit http://localhost:3000

Running the tests

bundle exec rspec

Built With

• Ruby - The programming language
• Rails - Web framework
• Postgres - Database
• Sidekiq - For background jobs
• Faraday - For external HTTP requests
• Redis - Fast data store

Authors

• Gideon Kimutai

License

This project is licensed under the MIT License - see the LICENSE.md file for details