Enhacing Video Retrival with Robust CLIP-Based Multimodal System

Authors

• Tuan Quang
• Huy Van Dinh
• Minh Dung Le Quynh
• Anh Tuan Nguyen
• Huy Tien Nguyen
• Minh-Hung An

Sytem pipeline

Setup Instruction

• Data Setup

  Download files below and move those files to backend folder

  • index_1_36.bin
  • sketch_index.bin
  • Pre-trained models for sketch
  • person_detection.npy

• Install MongoDB and Anaconda

  MongoDB is for keyframes storage. Anaconda is for python environment. Please use the link below for installing instruction.

  • Install MongoDB Driver
  • Install MongoDB Compass for User Interface
  • Install Anaconda

• Backend

  After cloning the repository, navigate to the backend folder. After installing anaconda, use the command below for installing packages for the environment.

  conda create -n AIC_ENV
  conda activate AIC_ENV
  pip install git+https://github.com/openai/CLIP.git
  pip install -r requirements.txt
  

• Sketch Pre-trained models

  After downloading pre-trained modeels for sketch, move file tsbir_model_final.pt to path VN_Video_Search_with_Sketch/backend/SketchModel/model

• Frontend

  Install Node.js for installing dependencies for frontend. After installing Node.js, navigate to frontend folder and run the command below for isntalling the dependencies.

  • Install Node.js

  npm install
  

Usage

• Frames Extraction

  After activating conda environment, navigate to backend/Notebook/Transnet.ipynb for frame extraction. Run the notebook with AIC_ENV environment in Backend Instruction.

• Creating binary files

  Please refer to backend/Notebook for creating binary files for CLIP and Sketch Model.

• MongoDB Database

  For database, make sure to turn on the driver for mongo. Please take a look at the Install MongoDB and Anaconda for references. Navigate to backend folder and run the command below for pushing data into database.

  python setup_database.py
  

• Backend

  Navigate to backend folder and run the command

  python app.py
  

• Frontend

  Navigate to frontend folder and run the command

  npm run start
  

  A http://localhost:3000/home should be launched after executing the command.