Mandai Backend

A Django-based web scraping application for commodity prices with vendor and market management.

Features

Automated web scraping of commodity prices using Selenium
Scheduled scraping tasks with Celery
CRUD operations for vendors and markets
Data persistence with Django ORM
API endpoints for data access
Redis integration for task queue management

Tech Stack

Django
Celery
Redis
Selenium
Docker

Prerequisites

Python 3.8+
Redis Instance

Setup & Installation

Clone the repository

git clone https://github.com/yogesh-bhandare/mandai-backend-api.git
cd mandai-backend-api

Create virtual environment For MacOS/Linux/Windows::

python3 -m venv venv
source venv/bin/activate

python -m venv venv
.\venv\Scripts\activate

Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

Redis Setup Using Docker: This will start Redis on redis://localhost:6170

docker compose -f compose.yaml up -d

Environment Configuration Create .env in project root:

CELERY_BROKER_REDIS_URL="redis://localhost:6170"
DEBUG=True

Database Setup

python manage.py migrate

Start Services In separate terminals:

# Terminal 1: Django Server
python manage.py runserver

# Terminal 2: Celery Worker & Beat
celery -A mandai worker --beat