Welcome to DocuThinker! This is a full-stack (FERN-Stack) application that integrates an AI-powered document processing backend with a React-based frontend. The app allows users to upload documents for summarization, generate key insights, and chat with an AI based on the document's content.
- π Overview
- π Live Deployments
- β¨ Features
- βοΈ Technologies
- πΌοΈ User Interfaces
- π Complete File Structure
- π οΈ Getting Started
- π API Endpoints
- π± Mobile App
- π¦ Containerization
- π Deployment
- βοΈ Load Balancing & Caching
- π Jenkins Integration
- π’ Kubernetes Integration
- π§ Contributing
- π License
- π Additional Documentation
- π¨βπ» Author
The DocuThinker app is designed to provide users with a simple, AI-powered document management tool. Users can upload PDFs or Word documents and receive summaries, key insights, and discussion points. Additionally, users can chat with an AI using the document's content for further clarification.
DocuThinker is created using the FERN-Stack architecture, which stands for Firebase, Express, React, and Node.js. The backend is built with Node.js and Express, integrating Firebase for user authentication and MongoDB for data storage. The frontend is built with React and Material-UI, providing a responsive and user-friendly interface.
It is currently deployed live on Vercel and Render. You can access the live app here.
We have deployed the entire app on Vercel and Render. You can access the live app here.
- Frontend: Deployed on Vercel. Access the live frontend here.
- Backend: Deployed on Render. You can access the live backend here.
- Backup: We have a backup of the app on Netlify. You can access the backup app here.
Note: The backend server may take a few seconds to wake up if it has been inactive for a while. The first API call may take a bit longer to respond. Subsequent calls should be faster as the server warms up.
Additionally, the app is currently on the Free Tier of Render, so it may take longer to process your request since we are only allocated 512MB and 0.1 CPU.
DocuThinker offers a wide range of features to help users manage and analyze their documents effectively. Here are some of the key features of the app:
- Document Upload & Summarization: Upload PDFs or Word documents for AI-generated summaries.
- Key Insights & Discussion Points: Generate important ideas and topics for discussion from your documents.
- AI Chat Integration: Chat with an AI using your documentβs original context.
- Voice Chat with AI: Chat with an AI using voice commands for a more interactive experience.
- Sentiment Analysis: Analyze the sentiment of your document text for emotional insights.
- Multiple Language Support: Summarize documents in different languages for global users.
- Content Rewriting: Rewrite or rephrase document text based on a specific style or tone.
- Actionable Recommendations: Get actionable recommendations based on your document content.
- Bullet Point Summaries: Generate bullet point summaries for quick insights and understanding.
- Document Categorization: Categorize documents based on their content for easy organization.
- Profile Management: Update your profile information, social media links, and theme settings.
- User Authentication: Secure registration, login, and password reset functionality.
- Document History: View all uploaded documents and their details.
- Mobile App Integration: React Native mobile app for on-the-go document management.
- API Documentation: Swagger (OpenAPI) documentation for all API endpoints.
- Authentication Middleware: Secure routes with JWT and Firebase authentication middleware.
- Containerization: Dockerized the app with Docker & K8s for easy deployment and scaling.
- Continuous Integration: Automated testing and deployment with GitHub Actions & Jenkins.
- Frontend:
- React: JavaScript library for building user interfaces.
- Material-UI: React components for faster and easier web development.
- Axios: Promise-based HTTP client for making API requests.
- React Router: Declarative routing for React applications.
- Context API: State management for React applications.
- Tailwind CSS: Utility-first CSS framework for styling.
- Craco: Create React App Configuration Override for customizing Webpack.
- Webpack: Module bundler for JavaScript applications.
- Backend:
- Express: Web application framework for Node.js.
- Redis: In-memory data structure store for caching.
- Firebase Admin SDK: Firebase services for server-side applications.
- Node.js: JavaScript runtime for building scalable network applications.
- Firebase Authentication: Secure user authentication with Firebase.
- Firebase Auth JWT: Generate custom tokens for Firebase authentication.
- Middlewares: Firebase authentication middleware for securing routes and JWT middleware for token verification.
- AI/ML Services:
- Google Cloud Natural Language API: Machine learning models for text analysis.
- Google Gemini API: AI-powered chatbot for enhanced user interaction.
- Google AI Studio: Tools for building and deploying machine learning models.
- NLP: Natural Language Processing for customized chat/text analysis and summarization models.
- NER: Named Entity Recognition for identifying entities in text.
- POS Tagging: Part-of-Speech Tagging for analyzing word types in text.
- Database:
- MongoDB: NoSQL database for storing user data and documents.
- Firestore: Cloud Firestore for storing user data and documents.
- Redis: In-memory data structure store for caching.
- Mobile App:
- React Native: JavaScript framework for building mobile applications.
- Expo: Framework and platform for universal React applications.
- Firebase SDK: Firebase services for mobile applications.
- React Navigation: Routing and navigation for React Native apps.
- API Documentation:
- Swagger: OpenAPI documentation for all API endpoints.
- OpenAPI: Specification for building APIs with RESTful architecture.
- Containerization:
- Docker: Containerization platform for building, shipping, and running applications.
- Kubernetes: Container orchestration for automating deployment, scaling, and management.
- Load Balancing & Caching:
- NGINX: Web server for load balancing, reverse proxying, and caching.
- CI/CD & Deployment:
- GitHub Actions: Automated workflows for testing and deployment.
- Jenkins: Automation server for continuous integration and deployment.
- Render: Cloud platform for hosting and scaling web applications. (Used to deploy the backend)
- Vercel: Cloud platform for hosting and deploying web applications. (Used to deploy the frontend)
- Netlify: Cloud platform for hosting and deploying web applications. (Used as a backup)
DocuThinker features a clean and intuitive user interface designed to provide a seamless experience for users. The app supports both light and dark themes, responsive design, and easy navigation. Here are some screenshots of the app:
The DocuThinker app is organized into separate subdirectories for the frontend, backend, and mobile app. Each directory contains the necessary files and folders for the respective components of the app. Here is the complete file structure of the app:
DocuThinker-AI-App/
βββ backend/
β βββ ai_ml/
β β βββ perform_ner_pos.py # Named Entity Recognition and Part-of-Speech Tagging
β β βββ sen_analysis.py # Sentiment analysis for document text
β β βββ chat.js # Chatbot integration for AI chat functionality
β β βββ analyzer.js # Document analyzer for generating key ideas and discussion points
β β βββ textStatistics.js # Text statistics for analyzing document content
β β βββ documentClassifier.js # Document classifier for categorizing documents
β β βββ summarizer.js # Document summarizer for generating summaries
β β βββ (and many more files...) # Additional AI/ML services
β βββ middleware/
β β βββ jwt.js # Authentication middleware with JWT for the app's backend
β βββ controllers/
β β βββ controllers.js # Controls the flow of data and logic
β βββ models/
β β βββ models.js # Data models for interacting with the database
β βββ services/
β β βββ services.js # Models for interacting with database and AI/ML services
β βββ views/
β β βββ views.js # Output formatting for success and error responses
β βββ redis/
β β βββ redisClient.js # Redis client for caching data in-memory
β βββ swagger/
β β βββ swagger.js # Swagger documentation for API endpoints
β βββ .env # Environment variables (git-ignored)
β βββ firebase-admin-sdk.json # Firebase Admin SDK credentials (git-ignored)
β βββ index.js # Main entry point for the server
β βββ Dockerfile # Docker configuration file
β βββ manage_server.sh # Shell script to manage and start the backend server
β βββ README.md # Backend README file
β
βββ frontend/
β βββ public/
β β βββ index.html # Main HTML template
β β βββ manifest.json # Manifest for PWA settings
β βββ src/
β β βββ assets/ # Static assets like images and fonts
β β β βββ logo.png # App logo or images
β β βββ components/
β β β βββ ChatModal.js # Chat modal component
β β β βββ Spinner.js # Loading spinner component
β β β βββ UploadModal.js # Document upload modal component
β β β βββ Navbar.js # Navigation bar component
β β β βββ Footer.js # Footer component
β β β βββ GoogleAnalytics.js # Google Analytics integration component
β β βββ pages/
β β β βββ Home.js # Home page where documents are uploaded
β β β βββ LandingPage.js # Welcome and information page
β β β βββ Login.js # Login page
β β β βββ Register.js # Registration page
β β β βββ ForgotPassword.js # Forgot password page
β β β βββ HowToUse.js # Page explaining how to use the app
β β βββ App.js # Main App component
β β βββ index.js # Entry point for the React app
β β βββ App.css # Global CSS 1
β β βββ index.css # Global CSS 2
β β βββ reportWebVitals.js # Web Vitals reporting
β β βββ styles.css # Custom styles for different components
β β βββ config.js # Configuration file for environment variables
β βββ .env # Environment variables file (e.g., REACT_APP_BACKEND_URL)
β βββ package.json # Project dependencies and scripts
β βββ craco.config.js # Craco configuration file
β βββ Dockerfile # Docker configuration file
β βββ manage_frontend.sh # Shell script for managing and starting the frontend
β βββ README.md # Frontend README file
β βββ package.lock # Lock file for dependencies
β
βββ mobile-app/ # Mobile app directory
β βββ app/ # React Native app directory
β βββ .env # Environment variables file for the mobile app
β βββ app.json # Expo configuration file
β βββ components/ # Reusable components for the mobile app
β βββ assets/ # Static assets for the mobile app
β βββ constants/ # Constants for the mobile app
β βββ hooks/ # Custom hooks for the mobile app
β βββ scripts/ # Scripts for the mobile app
β βββ babel.config.js # Babel configuration file
β βββ package.json # Project dependencies and scripts
β βββ tsconfig.json # TypeScript configuration file
β
βββ kubernetes/ # Kubernetes configuration files
β βββ manifests/ # Kubernetes manifests for deployment, service, and ingress
β βββ backend-deployment.yaml # Deployment configuration for the backend
β βββ backend-service.yaml # Service configuration for the backend
β βββ frontend-deployment.yaml # Deployment configuration for the frontend
β βββ frontend-service.yaml # Service configuration for the frontend
β βββ firebase-deployment.yaml # Deployment configuration for Firebase
β βββ firebase-service.yaml # Service configuration for Firebase
β βββ configmap.yaml # ConfigMap configuration for environment variables
β
βββ nginx/
β βββ nginx.conf # NGINX configuration file for load balancing and caching
β βββ Dockerfile # Docker configuration file for NGINX
β
βββ images/ # Images for the README
βββ .env # Environment variables file for the whole app
βββ docker-compose.yml # Docker Compose file for containerization
βββ jsconfig.json # JavaScript configuration file
βββ package.json # Project dependencies and scripts
βββ package-lock.json # Lock file for dependencies
βββ postcss.config.js # PostCSS configuration file
βββ tailwind.config.js # Tailwind CSS configuration file
βββ render.yaml # Render configuration file
βββ vercel.json # Vercel configuration file
βββ openapi.yaml # OpenAPI specification for API documentation
βββ manage_docuthinker.sh # Shell script for managing and starting the app (both frontend & backend)
βββ jenkins_cicd.sh # Shell script for managing the Jenkins CI/CD pipeline
βββ .gitignore # Git ignore file
βββ LICENSE.md # License file for the project
βββ README.md # Comprehensive README for the whole app
βββ (and many more files...) # Additional files and directories not listed here
Ensure you have the following tools installed:
- Node.js (between v14 and v20)
- npm or yarn
- Firebase Admin SDK credentials
- Redis for caching
- MongoDB for data storage
- RabbitMQ for handling asynchronous tasks
- Docker for containerization (optional)
- Postman for API testing (optional)
- Expo CLI for running the mobile app
- Jenkins for CI/CD (optional)
- Kubernetes for container orchestration (optional)
- React Native CLI for building the mobile app
- Firebase SDK for mobile app integration
- Firebase API Keys and Secrets for authentication
- Expo Go app for testing the mobile app on a physical device
- Tailwind CSS for styling the frontend
- .env file with necessary API keys (You can contact me to get the
.env
file - but you should obtain your own API keys for production).
Additionally, basic fullstack development knowledge and AI/ML concepts are recommended to understand the app's architecture and functionalities.
-
Clone the repository:
git clone https://github.com/hoangsonww/DocuThinker-AI-App.git cd DocuThinker-AI-App/backend
-
Navigate to the frontend directory:
cd frontend
-
Install dependencies:
npm install
-
Start the Frontend React app:
npm start
-
Build the Frontend React app (for production):
npm run build
-
Alternatively, you can use
yarn
to install dependencies and run the app:yarn install yarn start
-
Or, for your convenience, if you have already installed the dependencies, you can directly run the app in the root directory using:
npm run frontend
This way, you don't have to navigate to the
frontend
directory every time you want to run the app. -
The app will run on
http://localhost:3000
. You can access it in your browser.
Note that this is optional since we are deploying the backend on Render. However, you can (and should) run the backend locally for development purposes.
-
Navigate to the root (not
backend
) directory:cd backend
-
Install dependencies:
npm install
-
Start the backend server:
npm run server
-
The backend code is in the
backend
directory. Feel free to explore the API endpoints and controllers.
Note: Be sure to use Node v.20 or earlier to avoid compatibility issues with Firebase Admin SDK.
- Navigate to the mobile app directory:
cd mobile-app
- Install dependencies:
npm install
- Start the Expo server:
npx expo start
- Run the app on an emulator or physical device: Follow the instructions in the terminal to run the app on an emulator or physical device.
The backend of DocuThinker provides several API endpoints for user authentication, document management, and AI-powered insights. These endpoints are used by the frontend to interact with the backend server:
Method | Endpoint | Description |
---|---|---|
POST | /register |
Register a new user in Firebase Authentication and Firestore, saving their email and creation date. |
POST | /login |
Log in a user and return a custom token along with the user ID. |
POST | /upload |
Upload a document for summarization. If the user is logged in, the document is saved in Firestore. |
POST | /generate-key-ideas |
Generate key ideas from the document text. |
POST | /generate-discussion-points |
Generate discussion points from the document text. |
POST | /chat |
Chat with AI using the original document text as context. |
POST | /forgot-password |
Reset a user's password in Firebase Authentication. |
POST | /verify-email |
Verify if a user's email exists in Firestore. |
GET | /documents/{userId} |
Retrieve all documents associated with the given userId . |
GET | /documents/{userId}/{docId} |
Retrieve a specific document by userId and docId . |
GET | /document-details/{userId}/{docId} |
Retrieve document details (title, original text, summary) by userId and docId . |
DELETE | /delete-document/{userId}/{docId} |
Delete a specific document by userId and docId . |
DELETE | /delete-all-documents/{userId} |
Delete all documents associated with the given userId . |
POST | /update-email |
Update a user's email in both Firebase Authentication and Firestore. |
POST | /update-password |
Update a user's password in Firebase Authentication. |
GET | /days-since-joined/{userId} |
Get the number of days since the user associated with userId joined the service. |
GET | /document-count/{userId} |
Retrieve the number of documents associated with the given userId . |
GET | /user-email/{userId} |
Retrieve the email of a user associated with userId . |
POST | /update-document-title |
Update the title of a document in Firestore. |
PUT | /update-theme |
Update the theme of the app. |
GET | /user-joined-date/{userId} |
Get date when the user associated with userId joined the service. |
GET | /social-media/{userId} |
Get the social media links of the user associated with userId . |
POST | /update-social-media |
Update the social media links of the user associated with userId . |
POST | /update-profile |
Update the user's profile information. |
POST | /update-document/{userId}/{docId} |
Update the document details in Firestore. |
POST | /update-document-summary |
Update the summary of a document in Firestore. |
POST | /sentiment-analysis |
Analyzes the sentiment of the provided document text |
POST | /bullet-summary |
Generates a summary of the document text in bullet points |
POST | /summary-in-language |
Generates a summary in the specified language |
POST | /content-rewriting |
Rewrites or rephrases the provided document text based on a style |
POST | /actionable-recommendations |
Generates actionable recommendations based on the document text |
More API endpoints will be added in the future to enhance the functionality of the app. Feel free to explore the existing endpoints and test them using Postman or Insomnia.
- Swagger Documentation: You can access the Swagger documentation for all API endpoints by running the backend server and navigating to
http://localhost:5000/api-docs
. - Redoc Documentation: You can access the Redoc documentation for all API endpoints by running the backend server and navigating to
http://localhost:5000/api-docs/redoc
.
For example, our API endpoints documentation looks like this:
Additionally, we also offer API file generation using OpenAPI. You can generate API files using the OpenAPI specification. Here is how:
npx openapi-generator-cli generate -i http://localhost:5000/api-docs -g typescript-fetch -o ./api
This will generate TypeScript files for the API endpoints in the api
directory. Feel free to replace or modify the command as needed.
- View the API Documentation
- Open Swagger Editor.
- Upload the
openapi.yaml
file or paste its content. - Visualize and interact with the API documentation.
- Test the API
- Import
openapi.yaml
into Postman:- Open Postman β Import β Select
openapi.yaml
. - Test the API endpoints directly from Postman.
- Open Postman β Import β Select
- Or use Swagger UI:
- Provide the file URL or upload it to view and test endpoints.
- Generate Client Libraries
- Install OpenAPI Generator:
npm install @openapitools/openapi-generator-cli -g
- Generate a client library:
openapi-generator-cli generate -i openapi.yaml -g <language> -o ./client
- Replace
<language>
with the desired programming language.
- Generate Server Stubs
- Generate a server stub:
openapi-generator-cli generate -i openapi.yaml -g <framework> -o ./server
- Replace
<framework>
with the desired framework.
- Run a Mock Server
- Install Prism:
npm install -g @stoplight/prism-cli
- Start the mock server:
prism mock openapi.yaml
- Validate the OpenAPI File
- Use Swagger Validator:
- Upload
openapi.yaml
or paste its content to check for errors.
- Upload
This guide enables you to view, test, and utilize the API.
- We use Node.js and Express to build the backend server for DocuThinker.
- The backend API is structured using Express and Firebase Admin SDK for user authentication and data storage.
- We use the MVC (Model-View-Controller) pattern to separate concerns and improve code organization.
- Models: Schema definitions for interacting with the database.
- Controllers: Handle the business logic and interact with the models.
- Views: Format the output and responses for the API endpoints.
- Services: Interact with the database and AI/ML services for document analysis and summarization.
- Middlewares: Secure routes with Firebase authentication and JWT middleware.
- The API endpoints are designed to be RESTful and follow best practices for error handling and response formatting.
- The Microservices Architecture is also used to handle asynchronous tasks and improve scalability.
- The API routes are secured using Firebase authentication middleware to ensure that only authenticated users can access the endpoints.
- The API controllers handle the business logic for each route, interacting with the data models and formatting the responses.
-
You can test the API endpoints using Postman or Insomnia. Simply make a POST request to the desired endpoint with the required parameters.
-
For example, you can test the
/upload
endpoint by sending a POST request with the document file as a form-data parameter. -
Feel free to test all the API endpoints and explore the functionalities of the app.
curl --location --request POST 'http://localhost:3000/register' \
--header 'Content-Type: application/json' \
--data-raw '{
"email": "test@example.com",
"password": "password123"
}'
curl --location --request POST 'http://localhost:3000/upload' \
--header 'Authorization: Bearer <your-token>' \
--form 'File=@"/path/to/your/file.pdf"'
The backend APIs uses centralized error handling to capture and log errors. Responses for failed requests are returned with a proper status code and an error message:
{
"error": "An internal error occurred",
"details": "Error details go here"
}
The DocuThinker mobile app is built using React Native and Expo. It provides a mobile-friendly interface for users to upload documents, generate summaries, and chat with an AI. The mobile app integrates with the backend API to provide a seamless experience across devices.
Currently, it is in development and will be released soon on both the App Store and Google Play Store.
Stay tuned for the release of the DocuThinker mobile app!
Below is a screenshot of the mobile app (in development):
The DocuThinker app can be containerized using Docker for easy deployment and scaling. Follow these steps to containerize the app:
- Run the following command to build the Docker image:
docker compose up --build
- The app will be containerized and ready to run on port 3000.
You can also view the image in the Docker Hub repository here.
-
Install the Vercel CLI:
npm install -g vercel
-
Deploy the frontend:
vercel
-
Follow the instructions in your terminal to complete the deployment.
-
The backend can be deployed on platforms like Heroku, Render, or Vercel.
-
Currently, we are using Render to host the backend. You can access the live backend here.
-
Please note that we are currently on the Free Tier of Render. This means that the backend server may take a few seconds to wake up if it has been inactive for a while.
-
Therefore, the first API call may take a bit longer to respond. Subsequent calls should be faster as the server warms up. It is completely normal to take up to 2 minutes for the first API call to respond.
-
Also, the Free Tier of Render only allocates 512MB and 0.1 CPU. This may result in slower response times for API calls and document processing.
-
Additionally, during high traffic periods, the server may take longer to respond, although we have employed NGINX for load balancing and caching.
- We are using NGINX for load balancing and caching to improve the performance and scalability of the app.
- The NGINX configuration file is included in the repository for easy deployment. You can find the file in the
nginx
directory. - Feel free to explore the NGINX configuration file and deploy it on your own server for load balancing and caching.
- NGINX can also be used for SSL termination, reverse proxying, and serving static files. More advanced configurations can be added to enhance the performance of the app.
- You can also use Cloudflare or AWS CloudFront for content delivery and caching to improve the speed and reliability of the app, but we are currently using NGINX for load balancing and caching due to costs and simplicity.
- For more information, refer to the NGINX Directory.
- The NGINX configuration file is included in the repository for easy deployment. You can find the file in the
- We are also using Docker with NGINX to deploy the NGINX configuration file and run the server in a containerized environment. The server is deployed and hosted on Render.
- Additionally, we are using Redis for in-memory caching to store frequently accessed data and improve the performance of the app.
- Redis can be used for caching user sessions, API responses, and other data to reduce the load on the database and improve response times.
- You can set up your own Redis server or use a managed service like Redis Labs or AWS ElastiCache for caching.
- We are using Jenkins for continuous integration and deployment. The Jenkins pipeline is set up to automatically test and deploy the app whenever changes are pushed to the main branch.
- The pipeline runs the tests, builds the app, and deploys it to Vercel and Render. Feel free to visit the pipeline at
Jenkinsfile
. - The pipeline is triggered automatically whenever a new commit is pushed to the main branch.
- You can set up your own Jenkins pipeline to automate testing and deployment for your projects by following these commands and steps:
-
Install Jenkins:
brew install jenkins
-
Start Jenkins:
brew services start jenkins
-
Access Jenkins: Open your browser and go to
http://localhost:8080
to access the Jenkins dashboard. -
Follow the instructions to set up Jenkins and create a new pipeline.
If successful, you should see the Jenkins pipeline running and deploying the app automatically whenever changes are pushed to the main branch. Here is an example:
- We are using Kubernetes for container orchestration and scaling. The app can be deployed on a Kubernetes cluster for high availability and scalability.
- The Kubernetes configuration files are included in the repository for easy deployment. You can find the files in the
kubernetes
directory. - Feel free to explore the Kubernetes configuration files and deploy the app on your own Kubernetes cluster.
- You can also use Google Kubernetes Engine (GKE), Amazon EKS, or Azure AKS to deploy the app on a managed Kubernetes cluster.
We welcome contributions from the community! Follow these steps to contribute:
-
Fork the repository.
-
Create a new branch:
git checkout -b feature/your-feature
-
Commit your changes:
git commit -m "Add your feature"
-
Push the changes:
git push origin feature/your-feature
-
Submit a pull request: Please submit a pull request from your forked repository to the main repository. I will review your changes and merge them into the main branch shortly.
Thank you for contributing to DocuThinker! π
This project is licensed under the Creative Commons Attribution-NonCommercial License. See the LICENSE file for details.
The DocuThinker open-source project is for educational purposes only and should not be used for commercial applications. Feel free to use it for learning and personal projects!
For more information on the DocuThinker app, please refer to the following resources:
Here are some information about me:
- Son Nguyen - An aspiring Software Developer & Data Scientist
- Feel free to connect with me on LinkedIn.
- If you have any questions or feedback, please feel free to reach out to me at hoangson091104@gmail.com.
- Also, check out my portfolio for more projects and articles.
- If you find this project helpful, or if you have learned something from the source code, consider giving it a star βοΈ. I would greatly appreciate it! π
Happy Coding and Analyzing! π
Created with β€οΈ by Son Nguyen in 2024.