feature/implement-unit-testing

[abeltavares]

This pull request introduces unit tests for the market data processing system. Key components covered in the tests include:

1. StockApiClient Tests:

   • Test the initialization of StockApiClient with valid API keys.
   • Test the get_stocks method to ensure it retrieves stock symbols correctly.
   • Test scenarios with empty API responses and invalid data formats.

2. CryptoApiClient Tests:

   • Test the initialization of CryptoApiClient with a valid API key.
   • Test the get_data method to ensure it retrieves cryptocurrency data accurately.
   • Handle scenarios with empty API responses and invalid data formats.

3. Storage Tests:

   • Test the connect method to ensure successful database connections.
   • Validate the close method for proper closure of database connections.
   • Verify the store_data method for correct insertion of data into the database.
   • Test scenarios with both valid and invalid data types.

4. MarketDataEngine Tests:

   • Test the process_stock_data method to ensure correct orchestration of stock data processing.
   • Validate the process_crypto_data method for proper orchestration of cryptocurrency data processing.

5. Mocking and Exception Handling:

   • Utilize the unittest.mock library to simulate API responses and log messages.
   • Test scenarios where API responses contain invalid or missing data.
   • Validate exception handling in case of errors during data retrieval and storage.

6. Test Coverage and Execution:

   • Ensure that the test suite covers a significant portion of the codebase.
   • Verify that the tests run successfully without any failures.

These unit tests aim to improve the reliability and maintainability of the market data processing system by validating key functionalities and handling edge cases. The PR provides a foundation for ongoing development and refactoring efforts.

Additionally, this PR includes tests for the DAGs that validate the existence, loading, scheduling intervals and functions called by each task, ensuring their proper configuration and execution.

This PR introduces several tools to improve code quality and consistency. In addition to pre-commit hooks for running unit tests, the .pre-commit-config.yaml file also includes a hook for the Black code formatter. Black is an automatic code formatter that makes your Python code more readable and compliant with PEP 8, the Python style guide.

Developers can set up these pre-commit hooks by following the instructions provided in the README.md file. These hooks ensure that all committed code is properly formatted and that all unit tests pass, preventing code changes that may break existing functionality.

Furthermore, this PR also introduces a GitHub Actions workflow that runs unit tests and the Black formatter every time a pull request is made. This ensures that all proposed changes are properly tested and formatted before they can be merged into the main codebase.