Go-Scrapper

Go-Scrapper is a HTTP Restful API for crawling web application.

Installation

Clone from Github repository

git clone git@github.com:anacondaf/go-scrapper.git

Setup

# install all dependencies
go mod tidy

• Install swaggo

go install github.com/swaggo/swag/cmd/swag@latest

• To generate swagger document for http service

swag init -g ./src/core/application/http/server.go

Usage

To run and use this repository, follow these steps:

• Running Postgres server
• Create app.env file and copy all contents from .env.example
• Run by:

# Using golang air
air

# Using golang run
go run main.go

Run App

Use docker compose to start all services

docker compose up -d

Azure Devops

https://dev.azure.com/goscrapper-org/GoScrapper

Accounts

Portainer

1. User: admin | Password: P@$$word!

Features

• Written in Golang
• HTTP Framework - Go Fiber
• ORM - Gorm
• Supports PSQL, MongoDB
• Crawler Framework - Go Colly
• OpenAPI - Supports Client Service Generation

Click to See More!

• Response Caching - Go Redis
• Scheduler Library - Cdule
• Cloud storage - AWS S3
• API Versioning