Skip to content

Latest commit

 

History

History
66 lines (42 loc) · 2.3 KB

README.md

File metadata and controls

66 lines (42 loc) · 2.3 KB
Raw

Threaded URL Requests and Data Analysis - Python-threading

This Python script and documentation provide a threaded approach for making HTTP requests to a list of URLs, analyzing response data, finding attribute occurrences, and calculating rarity based on a collection size.

Usage

Installation

  1. Make sure you have Python 3.x installed.

  2. Install the required Python packages using pip:

pip install requests psutil
  1. Clone or download this repository to your local machine.

Running the Script

You can use the provided Python script threaded_url_requests.py for multi-threaded URL requests and data analysis. The script includes a class ThreadRequests and three main functions for different tasks:

  1. get_data(urls_list: list): Fetches data from a list of URLs.
  2. get_occurrence_data(urls_list: list): Fetches data and finds attribute occurrences.
  3. get_rarity_data(urls_list: list, collection_size): Fetches data, finds attribute occurrences, and calculates rarity.

Here's how to run these functions:

# Example usage
import threaded_url_requests

# List of URLs to fetch data from
urls_list = [...]
collection_size = 20  # Set your collection size

# Fetch data from URLs
attribute_list = threaded_url_requests.get_data(urls_list)

# Find attribute occurrences
occurrence_data = threaded_url_requests.get_occurrence_data(urls_list)

# Calculate rarity
rarity_data = threaded_url_requests.get_rarity_data(urls_list, collection_size)

Documentation

Detailed documentation for the provided Python script threaded_url_requests.py:

  • Class ThreadRequests: A class for handling multi-threaded URL requests and data analysis.
  • Methods: run, check_status, worker_get, find_occurrence, get_occurrence, calculate_rarity, and more.
  • Properties: responses.
  • Function get_data(urls_list): Fetches data from a list of URLs using multi-threaded requests.
  • Function get_occurrence_data(urls_list): Fetches data and finds attribute occurrences from a list of URLs using multi-threaded requests.
  • Function get_rarity_data(urls_list, collection_size): Fetches data, finds attribute occurrences, and calculates rarity from a list of URLs using multi-threaded requests.

License

This project is licensed under the MIT License

Author

Yasir Ali

LinkedIn:\yasirali179